Re: [Xen-users] HVM VGA Passthrough to Win7 64 with Xen 4.3

[Gordan Bobic <gordan@xxxxxxxxxx>]

I have no prior experience with that specific motherboard, so cannot be 
100% sure (BIOS/firmware/hardware bugs are disappointingly common these 
days). The rest sounds just fine.
Things to check/try in no particular order:

1) Make sure that the GPU is never touched during dom0 initialization. 
Blacklist the radeon, fglrx or any other drivers that might bind to the 
ATI card(s). If your xen-pciback module is built into the kernel, you 
can ensure this using something like the following in your kernel boot 
options:
xen-pciback.hide=(02:00.0)(02.00.1)

If the dom0 driver has ever bound to the device, it will have 
initialized it, and this is known to wreak havoc afterwards since the 
card is no longer in the state the domU driver expects to find it in, it 
gets double-initialized and either runs cripplingly slowly or not at 
all. The ATI BIOS and drivers aren't quite clever enough when it comes 
to this.
If your xen-pciback is a module, you'll have to blacklist the 
radeon/fglrx/whatever driver. Xorg will still end up loading it 
explicitly when it starts in dom0, so you may need to use an "install" 
option on the module (don't know how Ubuntu configs are laid out, on EL 
you'd put it in /etc/modprobe.d/$modulename.conf and possible add stuff 
to /etc/sysconfig/modules/$modulename.modules. The point of this is that 
you pre-load xen-pciback before the radeon driver is loaded. You need to 
tell xen-pciback (in a similar config) which devices to bind to. The net 
result is that xen-pciback loads before any other drivers for the 
device, thus ensuring other drivers don't taint the device before it is 
made available to a domU.
2) Depending on BIOS bugs, check where your PCI memory hole begins, 
using something like the following in dom0:
# lspci -vvv | grep "Memory at " | sed -e 's/.*Memory at //' | sort

Until you get it working, set your domU RAM to less than the lowest 
value the above command returns.
3) Disable the generic VGA device in domU (not in the xen config, in 
domU device manager). I had a similar problem where my quasi-Quadro 
wouldn't initialize in domU with the same error, because the driver was 
trying to map legacy VGA ports which were already claimed by the 
emulated VGA device.
4) Try to figure out (lspci -t helps) your layout WRT PCIe bridges and 
slots. If you can, put your secondary (passthrough) card on a PCIe 
bridge of it's own. This can help with some buggy hardware.
Also, you probably don't want to hear this, but I've had slightly better 
luck (although by no means perfect) with quadrified GeForce cards than 
ATI cards, despite AMD's recent efforts to make this work. Then again, I 
have what is probably the buggiest and most problematic motherboard ever 
made commercially available (EVGA SR-2).
Gordan

On 08/31/2013 10:43 PM, Dan Schmitt wrote:
> Hardware:
> 16gb Ram
> AMD FX-8320
> MSI 970A-G43
> Primary GPU Radeon HD6570 01:00.0,01:00.1
> Secondary GPU Radeon HD7950 02:00.0,02:00.1
>
> Dom0 Ubuntu 12.04
> Kernel 3.5.0-39
> xen 4.3
> xl toolset
> DomU Win7 64bit
>
> It took some shuffling to hide the HD7950 from
> Ubuntu, it's boot process, and X, but pciback seems
> to own it now (xl pci-assingable-list shows the 02:00.0
> and 02:00.1 when the Win7 DomU isn't running.)
>
> Win7 runs/installs with the emulated video card, and
> networking works.
> With the drivers installed Win7 seems to see the 7950,
> the hardware manager shows it with a yellow warning
> triangle/excalmation point. I'm seeing
>
> This device cannot find enough free resources that it can
> use (Code 12)
>
> If you want to use this device, you will need to disable
> one of the other devices on this system.
>
> In the /var/log/xen/xen-hotplug.log I'm seeing lots of
>
> RTNETLINK answers: Operation not supported
> RTNETLINK answers: Operation not supported
>
> but I think they are harmless.
> In /var/log/xen/qemu-dm-Win7.hvm.log I'm seeing:
>
> xen_ram_alloc: do not alloc 10f800000 bytes of ram at 0 when runstate
> is INMIGRATE
> xen_ram_alloc: do not alloc 800000 bytes of ram at 10f800000 when
> runstate is INMIGRATE
> xen_ram_alloc: do not alloc 10000 bytes of ram at 110000000 when
> runstate is INMIGRATE
> xen_ram_alloc: do not alloc 10000 bytes of ram at 110010000 when
> runstate is INMIGRATE
>
> in snippets of dmesg from boot that may be of interest:
>
> Grub command line (I think I have all the magic to hide 02:00.0):
>
> [    0.000000] Command line: placeholder
> root=UUID=14de2da2-d40b-48b8-a0ce-a43f71ddd612 ro apparmor=0 quiet
> splash blacklist.nouveau=1 xen-pciback.permissive
> xen-pciback.hide=(02:00.0)(02:00.2)
> pci=resource_alignment=02:00.0,02.00.2
>
> We can see it on the PCI bus:
>
> [    3.793517] pci 0000:02:00.0: [1002:679a] type 00 class 0x030000
> [    3.793539] pci 0000:02:00.0: reg 10: [mem 0xc0000000-0xcfffffff 64bit pref]
> [    3.793557] pci 0000:02:00.0: reg 18: [mem 0xfe900000-0xfe93ffff 64bit]
> [    3.793569] pci 0000:02:00.0: reg 20: [io  0xd000-0xd0ff]
> [    3.793592] pci 0000:02:00.0: reg 30: [mem 0xfe940000-0xfe95ffff pref]
> [    3.793598] pci 0000:02:00.0: Disabling memory decoding and
> releasing memory resources.
> [    3.793668] pci 0000:02:00.0: supports D1 D2
> [    3.793670] pci 0000:02:00.0: PME# supported from D1 D2 D3hot
> [    3.793718] pci 0000:02:00.1: [1002:aaa0] type 00 class 0x040300
> [    3.793740] pci 0000:02:00.1: reg 10: [mem 0xfe9
>
> I don't know why vgaarb grabs it (or if this is ok):
>
> [    3.835681] xen-balloon: Initialising balloon driver.
> [    3.835716] xen/balloon: Xen selfballooning driver disabled for domain0.
> [    3.835820] vgaarb: device added:
> PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
> [    3.835826] vgaarb: device added:
> PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none
> [    3.835830] vgaarb: loaded
> [    3.835831] vgaarb: bridge control possible 0000:02:00.0
> [    3.835832] vgaarb: bridge control possible 0000:01:00.0
>
> [    3.884755] pci 0000:02:00.0: BAR 0: assigned [mem
> 0xc0000000-0xcfffffff 64bit pref]
> [    3.884764] pci 0000:02:00.0: BAR 2: assigned [mem
> 0xfe900000-0xfe93ffff 64bit]
> [    3.884772] pci 0000:02:00.0: BAR 6: assigned [mem
> 0xfe940000-0xfe95ffff pref]
>
> The kernel grabs the 6570:
> The kernel grabs the 6570:
>
> [    4.680418] [drm] initializing kernel modesetting (TURKS
> 0x1002:0x6759 0x1462:0x2502).
> [    4.680483] [drm] register mmio base: 0xFEA20000
> [    4.680485] [drm] register mmio size: 131072
> [    4.680692] ATOM BIOS: 113
> [    4.680727] radeon 0000:01:00.0: VRAM: 1024M 0x0000000000000000 -
> 0x000000003FFFFFFF (1024M used)
> [    4.680730] radeon 0000:01:00.0: GTT: 512M 0x0000000040000000 -
> 0x000000005FFFFFFF
> [    4.680732] [drm] Detected VRAM RAM=1024M, BAR=256M
> [    4.680734] [drm] RAM width 128bits DDR
> [    4.681001] [TTM] Zone  kernel: Available graphics memory: 7982182 kiB
> [    4.681007] [TTM] Zone   dma32: Available graphics memory: 2097152 kiB
> [    4.681009] [TTM] Initializing pool allocator
> [    4.681018] [TTM] Initializing DMA pool allocator
> [    4.681044] [drm] radeon: 1024M of VRAM memory ready
> [    4.681046] [drm] radeon: 512M of GTT memory ready.
> [    4.681077] [drm] GART: num cpu pages 131072, num gpu pages 131072
> [    4.681670] [drm] Loading TURKS Microcode
>
> I don't know why it goes after the 7950 here (or how to stop it):
>
> [    5.034399] Console: switching to colour frame buffer device 240x67
> [    5.051586] fb0: radeondrmfb frame buffer device
> [    5.051587] drm: registered panic notifier
> [    5.051592] [drm] Initialized radeon 2.18.0 20080528 for
> 0000:01:00.0 on minor 0
> [    5.051622] radeon 0000:02:00.0: enabling device (0000 -> 0003)
> [    5.300587] sd 2:0:0:0: [sdb] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [    5.317630] radeon 0000:02:00.0: VRAM: 3072M 0x0000000000000000 -
> 0x00000000BFFFFFFF (3072M used)
> [    5.317633] radeon 0000:02:00.0: GTT: 512M 0x00000000C0000000 -
> 0x00000000DFFFFFFF
> [    5.317635] [drm] Detected VRAM RAM=3072M, BAR=256M
> [    5.317637] [drm] RAM width 384bits DDR
> [    5.317645] [drm] radeon: 3072M of VRAM memory ready
> [    5.317647] [drm] radeon: 512M of GTT memory ready.
> [    5.317661] [drm] GART: num cpu pages 131072, num gpu pages 131072
> [    5.318173] [drm] Loading TAHITI Microcode
> [    5.696909] radeon 0000:02:00.0: WB enabled
> [    5.696912] radeon 0000:02:00.0: fence driver on ring 0 use gpu
> addr 0x00000000c0000c00 and cpu addr 0xffff880407f99c00
> [    5.696915] radeon 0000:02:00.0: fence driver on ring 1 use gpu
> addr 0x00000000c0000c04 and cpu addr 0xffff880407f99c04
> [    5.696917] radeon 0000:02:00.0: fence driver on ring 2 use gpu
> addr 0x00000000c0000c08 and cpu addr 0xffff880407f99c08
> [    5.696919] [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
> [    5.696920] [drm] Driver supports precise vblank timestamp query.
> [    5.697011] radeon 0000:02:00.0: radeon: using MSI.
> [    5.697081] [drm] radeon: irq initialized.
> [    5.716856] [drm] ring test on 0 succeeded in 1 usecs
> [    5.716861] [drm] ring test on 1 succeeded in 1 usecs
> [    5.716866] [drm] ring test on 2 succeeded in 1 usecs
> [    5.717408] [drm] ib test on ring 0 succeeded in 0 usecs
> [    5.717443] [drm] ib test on ring 1 succeeded in 0 usecs
> [    5.717472] [drm] ib test on ring 2 succeeded in 0 usecs
> [    5.770027] [drm] Radeon Display Connectors
>
> [    5.837219] fb1: radeondrmfb frame buffer device
> [    5.837224] [drm] Initialized radeon 2.18.0 20080528 for
> 0000:02:00.0 on minor 1
>
> It seems like the drm stuff lets go of it and picback gets
> it:
>
> [   12.475932] [drm] radeon: finishing device.
> [   12.481586] radeon 0000:02:00.0: ffff880409aa7800 unpin not necessary
> [   12.481858] [drm] radeon: ttm finalized
> [   12.482371] pciback 0000:02:00.0: seizing device
> [   12.584854] pciback 0000:02:00.1: seizing device
>
> I don't know if this is good:
>
> [   12.603755] vgaarb: device changed decodes:
> PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
> [   12.603758] vgaarb: transferring owner from PCI:0000:01:00.0 to
> PCI:0000:02:00.0
>
> This seems like what I want to happen when the Win7 hvm starts up:
>
> [   40.821498] xen-pciback: vpci: 0000:02:00.0: assign to virtual slot 0
> [   40.822125] xen-pciback: vpci: 0000:02:00.1: assign to virtual slot 0 func 1
>
> My xorg is set up to only look at the 6750
> with BusId "PCI:1:0:0"
>
> So, qeustions.
>
> Do I have hardware that will work for hvm
> Win7 (HD6570 for Dom0, and HD7950 for DomU,
> 970 bridge chip Mobo and VT-d enabled?)
>
> Do I have the right software to "work out
> of the box"?  (4.3 xen release and 12.04
> ubuntu)
>
> Am I doint something silly or are there more things
> I should look for?
>
> My next explorations will be trying to pass the 7950 to
> an ubuntu HVM to see what happens, and see if booting
> in text mode (and keeping the kernel/frame buffer out
> of it) does anything.
>
> Thanks for your time.
>
>           Dan S
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxx
> http://lists.xen.org/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users