Xen project Mailing List

"Are you 100% certain that xen-pciback grabs the device BEFORE the radeon FB driver is loaded?"

I'm not a 100% certain. However, I've been using the same (initramfs) method across different Linux Mint/Xen releases and with different hardware (my regular Nvidia GPU for domU, as well as the AMD 7770 tested here). Since the PCI IDs of the GPU are listed by xm pci-list-assignable-devices (or the xl counterpart) I assume pciback took control.

So if I understand you correctly, you believe that the fglrx or radeon driver may have initialized the graphics card before that pciback module was able to grab it. So when the pciback module takes control of the GPU, it does so in a different (initialized) state, compared to when pciback grabs the GPU before fglrx or radeon kicks in.

Just for information, I did not install the fglrx driver in my tests but used the radeon driver. However, on my regular hardware I use the fglrx driver for AMD 7770 used by dom0, and pass through the Nvidia card to domU - the nouveau (Nvidia) driver is blacklisted on the kernel command ("nouveau.blacklist=1") and doesn't show in lsmod.

Checking dmesg on my regular hardware, pciback kicks in before the fglrx driver is loaded:
[    9.329564] pciback 0000:02:00.0: seizing device
[    9.329570] pciback 0000:02:00.1: seizing device
[    9.329730] xen: registering gsi 44 triggering 0 polarity 1
[    9.329744] xen: --> pirq=44 -> irq=44 (gsi=44)
[    9.329878] pciback 0000:02:00.0: enabling device (0000 -> 0003)
[    9.329891] xen: registering gsi 40 triggering 0 polarity 1
[    9.329892] Already setup the GSI :40
[    9.330050] xen_pciback: backend is passthrough
...
...
[   11.494115] fglrx: module license 'Proprietary. (C) 2002 - ATI Technologies, Starnberg, GERMANY' taints kernel.

I didn't check dmesg when I did the tests, so I can't be sure that the Radeon driver also kicks in after pciback, but it's likely the case.

I believe Fedora has pciback compiled into the kernel? This should make it easier to attach it to the GPU.

One more test I ran today is this: I added the following line into /etc/default/grub, followed by update-grub:
GRUB_CMDLINE_LINUX_XEN_REPLACE_DEFAULT="xen-pciback.hide=(02:00.0)(02:00.1) nouveau.blacklist=1 quiet nomodeset"

I can see it's being used in the dmesg output, but I don't see any real difference. It looks like the initramfs method is good enough to load pciback and assign the GPU before a graphics driver gets loaded.

"That surprises the living daylights out of me. What driver version are
you using in domU? I is vaguely possible that the very latest driver has
finally been fixed to do a bus reset before trying to initialize the
card. Or are you using primary passthrough and re-POST the card in domU
using it's BIOS to get it back into a clean state?"

I just downloaded the latest non-beta 64 bit driver (Catalyst Software Suite) from the AMD website which is 13.12 - see here. Prior to installing the Catalyst suite I installed the .net 4.5 stuff.
I'm doing secondary passthrough, for some reason I never managed to make primary passthrough work (even not with the Nvidia card). I did nothing else but install these two packages (.net and AMD Catalyst).

How do I check if ACS works on my X79 platform? I haven't got a clue.

"Just out of interest, have you tried your Nvidia card with Xen 4.3.x?
Does that work?"

I'm writing this on Linux Mint 16 running a 3.11.0-18-generic kernel on Xen 4.3.0 (4.3.0-1ubuntu1.3 to be exact), with my AMD 7770 for dom0 and the Nvidia Quadro 2000 for domU (Windows 7 Pro). I use the xl toolstack for this and it works nicely, which is why the AMD tests are quite surprising to me. Before that I used xm with Xen 4.3.0 and it also worked just fine with the Nvidia card.

The only issue I saw with the Nvidia card was the error 22 problem with xm that appeared long ago in Xen 4.1.3.

Heiko

On Friday, March 14, 2014 10:21 PM, Gordan Bobic <gordan@xxxxxxxxxx> wrote:

On 03/14/2014 07:37 PM, H. Sieger wrote:
> The first thing I check before attempting passthrough is
> pci-list-assignable-devices (or equivalent in xl lingo). Since I use 2
> AMD cards, I need the driver for the dom0 card.
>
> xen-pciback is loaded as a module from initramfs using:
> |echo "xen-pciback passthrough=1 hide=(02:00.0)(02:00.1)" >>
> /etc/initramfs-tools/modules|
> I have not tried to build it into the kernel and use a grub command to
> grab the graphics card, but pciback seems to seize the devices just fine.

Are you 100% certain that xen-pciback grabs the device BEFORE the radeon
FB driver is loaded? There are several ways to do this:

1) On RHEL/Fedora based distros the module init stuff is in
/etc/sysconfig/modules and module config is in /etc/modprobe.d/ - it may
be different on your distro. Blacklist the radeon (or fglrx if that is
what you are using) driver in /etc/modprobe.d/, and create a module init
script (normal shell script, make sure it's executable) in
/etc/sysconfig/modules that modprobes xen-pciback (make sure config for
it including the list of devices to seize is in /etc/modprobe.d/) to
sieze the secondary device, THEN modprobes radeon/fglrx after the device
you want to pass through has bee seized by xen-pciback.

2) Use a pre-install option for your radeon/fglrx driver that insmods
xen-pciback before it loads the GPU driver.

Both approaches will work, but you have to make sure that your initrd
doesn't contain the radeon or fglrx driver and that those aren't getting
loaded there.

> Regarding error 22 I guess I would file a bug report but since doing the
> tests I have changed everything back to my regular hardware and it won't
> be easy for me switching back and forth. I also hope that some users on
> the Linux Mint forum will report their experience. By the way, I
> followed my own how-to here:
> http://forums.linuxmint.com/viewtopic.php?f=42&t=112013 and
> http://forums.linuxmint.com/viewtopic.php?f=42&t=112013&start=360#p818716 (this
> 2nd for Linux Mint 13).
>
>
> "You still haven't clarified whether you are able to reboot the domU with
> the ATI card and still have it working without performance degradation,
> BSODs, the card not showing up at all, or crashing the host."
>
> I thought I did - yes, I can reboot the domU with the ATI card and still
> have it working without any performance degradation, BSODs, or other
> issues. I did that multiple times (at least 10 times), in several cases
> running WEI or Unigine for simple benchmarks. Each time I get the same
> performance in domU as the previous time, and the host (dom0) works just
> fine. Just as I would expect.I also switched between domU and dom0 to
> perform various tasks - no issue (I use a USB KVM switch for
> keyboard/mouse).

That surprises the living daylights out of me. What driver version are
you using in domU? I is vaguely possible that the very latest driver has
finally been fixed to do a bus reset before trying to initialize the
card. Or are you using primary passthrough and re-POST the card in domU
using it's BIOS to get it back into a clean state?

> I do see your point in checking for radeon driver issues, but I still
> wonder why in one case it works fine (Xen 4.1.2 with xm) and in the
> other case (Xen 4.3 with xl or xm) doesn't.

As far as I can tell you also changed the kernel and initrd in the
process, so it is easily possible the combo that doesn't work uses an
initrd with the radeon or fglrx driver getting pre-loaded to give you
the high-res console before the pciback driver can seize it.

I have also seen other deeply bizzare issues with ATI hardware,
specifically when mixing cards of very different generations, e.g. a
4850 primary with 7970 and sometimes with 7450 secondary for
passthrough. It looks vaguely like the primary card's BIOS spots the
other ATI cards and tries to initialize them, and messes them up. I have
also seen it happen the other way around where the card in domU
interferes with the card in dom0 (I am running with PCIe ACS disabled
because it is broken on my system - on yours it should be working fine,
which may well save you from all sorts of weird side effects).

Just out of interest, have you tried your Nvidia card with Xen 4.3.x?
Does that work?

Gordan

Re: [Xen-users] VGA passthrough with Xen 4.3 and xl toolstack - performance degradation resolved?