Re: [Xen-users] VGA Passthrough / Xen 4.2 / Linux 3.9.2

[Gordan Bobic <gordan@xxxxxxxxxx>]

On 05/15/2013 02:33 AM, Alex Karaoui wrote:
> Aloha!
>
> Gordan, I've been following this thread reading your very detailed and
> thorough analysis.  Like you I too am using an AMD Radeon (I have a 5870
> and a 5750).  I'm also using a dual socket Xeon (5440 on Supermicro
> X7DWA-N).  I haven't gotten as far as you have though, not even close.
>   I never video to work, not even once.  I had to stop working on
> getting Xen to work because I'm involved in several development projects
> at the moment and just didn't have the time, but I hope to get back to
> it soon.
>
> Kindly, would it be possible for you to answer _some_ the following:
>   (it's a lot, so pick and choose, but I would like best if I could
> reproduce your environment).
>
> 1) Could you post your grub.cfg, kernel configuration (from /boot),
My machine is diskless using NFS root and booting via PXE, but my pxe 
boot config file entry is here:
label xen
        kernel mboot.c32
        append xen.gz noreboot dom0_vcpus_pin --- 
vmlinuz-3.9.2-1.el6xen.x86_64 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM 
root=nfs:10.11.12.13:/nfsroot/normandy,rw,proto=tcp,noatime,nolock,nocto,actimeo=300 
ip=eth0:dhcp selinux=0 intel_iommu=on elevator=deadline iomem=relaxed 
noirqbalance --- initramfs-3.9.2-1.el6xen.x86_64.img
(sorry about the line wraps, I'm sure you can figure it out)

You may or may not require noirqbalance - I found that without it the 
domU would reliably BSOD every time.
> and your xm's domU configuration file?
It is in xenstore, so I cannot easily get to it. You should be able to 
reverse engineer most of it from the attached xend.log.
I created the VM using virt-manager.

> 2) What distro and kernel version are your running?
EL6 (Scientific Linux) with the kernel and xen packages from here:
http://xen.crc.id.au/support/guides/install/

Note: /boot/xen.gz I use is from xen-hypervisor-4.2.1-6 package.
The rest of the xen stack (using xm) is 4.2.2. This is because of this 
bug that is not yet resolved:
http://xen.crc.id.au/bugs/view.php?id=5

Versions of /boot/xen.gz newer than 4.2.1-6 don't work for me, and the 
problem has been tracked down to XSA46.
> 3) Did you compile the Xen kernel or the dom0 kernel with any patches or
> are they stock?
Unmodified packages from the source mentioned above.

> 4) Did you compile qemu with any patches?
All packages are as pulled down from EL6 repositories or from the Xen 
EL6 kernels site.
> 5) What version of Xen are you using?  Still using the xm toolchain, yes?
Yes, xm.

> 6) Are you using 'radeon' OSS driver with Kernel-Mode-Swtiching or the
> AMD proprietary driver?
My dom0 GPU is nvidia, using the Nvidia binary driver (modified 295.75 
due to this:
http://www.altechnative.net/2013/04/14/wquxga-a-k-a-omgwtf-ibm-t221-3840x2400-204dpi-monitor-part-6-regressing-drivers-and-xen/

I have not yet had a chance to write a LD_PRELOAD library that fakes out 
RandR geometry the same way the fakexinerama library does to enable 
sensible use of my T221.
> 7) Are you using pciback compiled into the kernel or as a module?  How
> are you invoking it?
It's a module.
# cat /etc/modprobe.d/xen-pciback.conf
options xen-pciback permissive=1 
hide=(00:1a.0)(00:1b.0)(00:1d.2)(02:00.0)(07:00.0)(07:00.1)
02:00.0 is a Marvell NIC (one of the two on-board ones)
07:00.? is the ATI card
the rest is USB conrollers.

The radeon driver is blacklisted.

Before I start the VM, I detach the devices from the host:

====
# cat usr/local/sbin/detach.sh
#!/bin/bash

modprobe xen-pciback

virsh nodedev-detach pci_0000_00_1a_0
virsh nodedev-detach pci_0000_00_1b_0
virsh nodedev-detach pci_0000_00_1d_2
virsh nodedev-detach pci_0000_02_00_0
virsh nodedev-detach pci_0000_07_00_0
virsh nodedev-detach pci_0000_07_00_1
====

If you are using radeon cards for both dom0 and domU you will have to do 
some additional magic because you can't just blacklist the radeon driver.
(Note: Modified from my nvidia based config when I was trying Nvidia 
8800GT for dom0 and Quadro 2000 for domU, both using the same binary driver)
# cat etc/modprobe.d/radeon.conf
install radeon /usr/local/sbin/detach-radeon.sh; insmod 
/lib/modules/$(/bin/uname -r)/kernel/drivers/gpu/drm/radeon/radeon.ko
====
# cat usr/local/sbin/detach-radeon.sh
#!/bin/bash

modprobe xen-pciback

virsh nodedev-detach pci_0000_07_00_0
virsh nodedev-detach pci_0000_07_00_1
====

In this example, the passed through card is 07:00:?, adjust the IDs 
accordingly for your setup. What this does is when the radeon driver 
gets probed, it will first invoke the script above, which will detach 
the passthrough device from the host and bind it to xen-pciback driver. 
This will make the radeon driver unable to bind the device, and it will 
thus only bind to the dom0 primary device, which will leave the 
secondary free for passthrough.
You can omit the duplicated lines in detach.sh since there is no point 
in detaching the GPU twice (although I don't think there's any harm in 
doing so).
> 8) What version of Windows and what version of Catalyst are you using?
>   Are you also using CCC?
Windows 7 in this test. Latest 13.4 ATI drivers. No CCC, just the bare 
driver.
> 9) Is the dom0 running Xorg?  Is Xorg running on your primary (BIOS) or
> secondary card?
Yes, running on the BIOS primary GPU (Nvidia 8800GT).

> 10) Which card did you pass through the primary or secondary?
Secondary, Radeon 6450.

> What I would ultimately like to do is create a LiveCD for anyone
> attempting this in the future that's configured to some extent to run
> AMD cards using PCI passthrough.
I don't think this is possible without at least some manual 
configuration. Specifically, the device PCI IDs to be detached and 
passed through will differ on every system. Thankfully, when swapping 
cards around, the PCI ID is based on the slot the card is plugged into 
so when replacing GPUs, as long as you put it in the same slot, you 
don't have to change any IDs in your configs.
> But having followed your posts, I'm just amazed that you got it working,
> and now, seemingly flawlessly.  I just wish I could understand how.
Me too - I am more frustrated by the fact that it is working now than I 
was by the fact that it wasn't working before.
The only definitive thing I can narrow it down to at the moment is 
ejecting the card to reset it just prior to starting a game. Then again, 
I half expect that tonight when I try it again, it will just BSOD on me 
all over the place, even though the configuration can't have changed 
since powering the machine off last night.
Gordan

Attachment: xend.log
Description: Text document

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users