Re: [Xen-users] VGA passthrough with Xen 4.3 and xl toolstack - performance degradation resolved?

[Gordan Bobic <gordan@xxxxxxxxxx>]

On 03/14/2014 07:37 PM, H. Sieger wrote:
> The first thing I check before attempting passthrough is
> pci-list-assignable-devices (or equivalent in xl lingo). Since I use 2
> AMD cards, I need the driver for the dom0 card.
>
> xen-pciback is loaded as a module from initramfs using:
> |echo "xen-pciback passthrough=1 hide=(02:00.0)(02:00.1)" >>
> /etc/initramfs-tools/modules|
> I have not tried to build it into the kernel and use a grub command to
> grab the graphics card, but pciback seems to seize the devices just fine.
Are you 100% certain that xen-pciback grabs the device BEFORE the radeon 
FB driver is loaded? There are several ways to do this:
1) On RHEL/Fedora based distros the module init stuff is in 
/etc/sysconfig/modules and module config is in /etc/modprobe.d/ - it may 
be different on your distro. Blacklist the radeon (or fglrx if that is 
what you are using) driver in /etc/modprobe.d/, and create a module init 
script (normal shell script, make sure it's executable) in 
/etc/sysconfig/modules that modprobes xen-pciback (make sure config for 
it including the list of devices to seize is in /etc/modprobe.d/) to 
sieze the secondary device, THEN modprobes radeon/fglrx after the device 
you want to pass through has bee seized by xen-pciback.
2) Use a pre-install option for your radeon/fglrx driver that insmods 
xen-pciback before it loads the GPU driver.
Both approaches will work, but you have to make sure that your initrd 
doesn't contain the radeon or fglrx driver and that those aren't getting 
loaded there.
> Regarding error 22 I guess I would file a bug report but since doing the
> tests I have changed everything back to my regular hardware and it won't
> be easy for me switching back and forth. I also hope that some users on
> the Linux Mint forum will report their experience. By the way, I
> followed my own how-to here:
> http://forums.linuxmint.com/viewtopic.php?f=42&t=112013 and
> http://forums.linuxmint.com/viewtopic.php?f=42&t=112013&start=360#p818716 (this
> 2nd for Linux Mint 13).
>
>
> "You still haven't clarified whether you are able to reboot the domU with
> the ATI card and still have it working without performance degradation,
> BSODs, the card not showing up at all, or crashing the host."
>
> I thought I did - yes, I can reboot the domU with the ATI card and still
> have it working without any performance degradation, BSODs, or other
> issues. I did that multiple times (at least 10 times), in several cases
> running WEI or Unigine for simple benchmarks. Each time I get the same
> performance in domU as the previous time, and the host (dom0) works just
> fine. Just as I would expect.I also switched between domU and dom0 to
> perform various tasks - no issue (I use a USB KVM switch for
> keyboard/mouse).
That surprises the living daylights out of me. What driver version are 
you using in domU? I is vaguely possible that the very latest driver has 
finally been fixed to do a bus reset before trying to initialize the 
card. Or are you using primary passthrough and re-POST the card in domU 
using it's BIOS to get it back into a clean state?
> I do see your point in checking for radeon driver issues, but I still
> wonder why in one case it works fine (Xen 4.1.2 with xm) and in the
> other case (Xen 4.3 with xl or xm) doesn't.
As far as I can tell you also changed the kernel and initrd in the 
process, so it is easily possible the combo that doesn't work uses an 
initrd with the radeon or fglrx driver getting pre-loaded to give you 
the high-res console before the pciback driver can seize it.
I have also seen other deeply bizzare issues with ATI hardware, 
specifically when mixing cards of very different generations, e.g. a 
4850 primary with 7970 and sometimes with 7450 secondary for 
passthrough. It looks vaguely like the primary card's BIOS spots the 
other ATI cards and tries to initialize them, and messes them up. I have 
also seen it happen the other way around where the card in domU 
interferes with the card in dom0 (I am running with PCIe ACS disabled 
because it is broken on my system - on yours it should be working fine, 
which may well save you from all sorts of weird side effects).
Just out of interest, have you tried your Nvidia card with Xen 4.3.x? 
Does that work?
Gordan

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users