Re: [Xen-devel] Struggling with PCI-Passthrough

[Gordan Bobic <gordan@xxxxxxxxxx>]

On 03/25/2014 08:33 PM, Alexander Brychcy wrote:
> Am 25.03.2014 11:46, schrieb Gordan Bobic:
> > On 2014-03-24 19:18, Alexander Brychcy wrote:
> >
> > > As ever there are only problems with domU #2. The first one, that i
> > > could not passthrough an USB controller resolved. No matter at which
> > > port you plug an USB device on the mainboard, it is always assigned to
> > > the XHCI controller.
> > I found that on Intel ICH USB controllers the PCI USB device that the
> > USB device is attached to changes dependant on what is connected. There
> > is some odd routing going on. This is especially the case when a USB
> > hub is connected to a port. If you just want to pass through, say, a
> > keyboard, it is stable, but IIRC the device has to be connected to the
> > port before you re-assign the PCI device that is the port to xen-pcistub.
> >
> > My motherboard also comes with a NEC USB 3.0 adapter on-board which
> > doesn't suffer from this problem, so for VMs that require multiple
> > USB devices I have a USB hub on that controller and I pass the
> > whole controller to the VM.
> >
> > > My workaround is to add another PCIe USB controller
> > > for the vm, so i'm still able to have usb connection in dom0 for backup
> > > or other purposes.
> > Sounds similar to what I do. Note that you can also pass specific USB
> > devices to the domU, rather than using PCI passthrough for the USB
> > controllers. Unfortunately this comes with ~5% CPU idling overhead
> > in domU due to USB polling.
> >
> > > Tried to passthrough a NIC to verify if it is a bios related bug. This
> > > works without any problems and nothing freezes at all.
> > I suspect you are just seeing the Intel ICH USB weirdness I described
> > above.
> I think that should be the same weirdness. I declare the problem solved
> :) vm will get their own dedicated controller when everythin else is
> working.
If you have a spare slot for a USB card, that is probably the easiest 
option. The NEC USB 3.0 controller works very well on my system. Having 
said that, I did manage to make passing through of my ICH USB devices to 
work as well, it's just fiddly until you get it right.
> > > So last but not least the VGA Passthrough. Issue is still present. HVM
> > > boots and GPU is assigned and working. But after some time respectively
> > > an application tries to access some kind of function the whole system
> > > freezes.
> > How much RAM are you passing to the VM? It is possible your motherboard
> > suffers from the same problem as mine and IOMEM DMA is bypassing the
> > root PCIe hub. Do the following:
> >
> > lspci -v | grep "Memory at" | sed -e 's/.*Memory at //' -e 's/ .*//' | sort
> >
> > Convert the first address that comes back from hex, and set the amount
> > of memory you are passing to your VM to a few MB below that.
> >
> > Then see if the crash still happens after a while. If it doesn't, then
> > there is a good chance that what is happening is that your VM's
> > memory is trampling over the IOMEM region of the host because it
> > isn't being translated by the IOMMU, probably because DMA is
> > bypassing it. In my case this is caused by a bug/feature of NF200
> > PCIe switches, but it could also be caused by a buggy motherboard
> > chipset or firmware.
> Edited the config to 768MB - way below the address 2051MB. First i
> though it works, but only because the 5 Minutes felt like hours. Win7 is
> a pain in the ass with such scanty amount of memory.
Your motherboard maps IOMEM as low as 768MB? Seriously?

Anyway, you are saying it crashed anyway, so it may not be the same 
problem. Then again, it could be - can you check in device manager where 
the IOMEM for the GPU was mapped in domU? Then cross-check that against 
the IOMEM ranges given by lspci -vvv and see if there is an overlap. If 
there isn't, then it's definitely not the same problem. If there is, it 
could plausibly still be the same problem (only much harder to solve in 
your case).
> > > Nearly forgot the RAM problem. The maximum amount of RAM i can assign to
> > > the HVM when i passthrough the GPU is 3GB. Haven't tried it yet but may
> > > this patch still be usable with 4.4.0?
> > > http://www.davidgis.fr/blog/index.php?2013/04/05/937-xen-43-unstable-vga-passthrough-hd-7970-windows-7-64-bits-with-more-than-3gb-for-ram
> Tried to apply the patch, but there were to many changes since 4.2.0 and
> currently i don't have to time to find all the changes in the current files.
> > I think this is for a different bug, but probably worth a try
> > depending on what results you get with domU memory size reduced to
> > ensure it cannot trample IOMEM regions.
> I think it has definetly something to do with the memory mapping. I want
> to assign 8GB of RAM to the hvm. And the dedicated memory of the gpu to
> passthrough itself is 4GB.
That can't be right. No standard GPU maps a 4GB aperture. Newer Nvidia 
cards map 128MB+32MB of IOMEM. ATI, IIRC, map 128MB or 256MB. Certainly 
not 4GB. You can modify the Nvidia cards' BIOS to enlarge the direct 
mapped IOMEM area to cover the whole of the GPU RAM (e.g. if you want to 
use it as a fast RAM disk for swapping when you are not otherwise using 
the GPU). There may be a similar hack for ATI cards. But either way, no 
way is all of the VRAM mapped into IOMEM on an unmodified card.
> The thread "[Xen-devel] [PATCH v3 0/4] Add
> max-ram-below-4g (was Add pci_hole_min_size machine option)" aroused my
> attention.
Yes, I've been keeping an eye on it, too, but it may not solve your 
problem if you find that the domU IOMEM overlaps dom0 IOMEM. I lucked 
out and it doesn't on my machine, but the proper fix would be a generic, 
adaptive vBAR=pBAR patch. I started working on a patch in that direction 
(what my half finished working bodge set out to achieve initially, but 
I've not had much time recently to work on it. The domU memory map is 
defined in multiple places and I hadn't managed to get them all to agree.
Gordan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel