[Xen-devel] [Xen-unstable][Qemu-xen] HVM Guest reading of Expansion ROM from passthroughed PCI device returns data from emulated VGA rom

[Sander Eikelenboom <linux@xxxxxxxxxxxxxx>]

Hi,

I'm trying to get secondary vga-passthrough on a HVM guest to work with a AMD 
HD6570 and the native kernel radeon driver and kernel modesetting.
So the guest still gets the emulated stdvga or cirrus device(used in my case 
here) as primary/boot vga adapter.

- When i don't passthrough the radeon card, the linux native radeon driver 
loads fine.
- When i do passtrough the device to a HVM with the same kernel:
  The driver in the guest tries to read the pci expansion rom from the 
passthroughed device to get the vbios.
  The driver reports a successful read, but fails because it can't find the 
right string at the right offset.

So I inspected the rom by using sysfs with:
echo 1 > /sys/bus/pci/BDF/rom
cat /sys/bus/pci/BDF/rom

- When i use this in dom0 (so without passthrough) i the contents of the ROM 
are valid (as expected since the driver loads fine)
- When i use this in the Guest (passthrouhed), the contents of the ROM i get 
are not from the passedthrough adapter, but from the emulated cirrus card.
  (it's the same as the "tools/firmware/vgabios/VGABIOS-lgpl-latest.cirrus.bin")

So i tried both qemu-xen and qemu-xen-traditional, but both with the same 
result.
To see if some addresses get mangled i enabled the xen passthrough debug 
printk's for both qemu's and added a printk to the radeon driver to see where 
it tries to read the rom.

from dom0 lspci:
07:00.0 VGA compatible controller [0300]: Advanced Micro Devices [AMD] nee ATI 
Turks [Radeon HD 6570] [1002:6759] (prog-if 00 [VGA controller])
        Subsystem: PC Partner Limited Device [174b:e193]
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
<MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 32
        Region 0: Memory at a0000000 (64-bit, prefetchable) [disabled] 
[size=256M]
        Region 2: Memory at f99c0000 (64-bit, non-prefetchable) [disabled] 
[size=128K]
        Region 4: I/O ports at a000 [disabled] [size=256]
        Expansion ROM at f99a0000 [disabled] [size=128K]

from qemu-dm-log:
vgabios-cirrus.bin: ROM id 101300b8 / PCI id 101300b8
pxe-e1000.rom: ROM id 8086100e / PCI id 8086100e
xen_platform: changed ro/rw state of ROM memory area. now is rw state.
xen: I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
xen: I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
[00:05.0] xen_pt_initfn: Assigning real physical device 07:00.0 to devfn 0x28
[00:05.0] xen_pt_register_regions: IO region 0 registered (size=0x10000000lx 
base_addr=0xa0000000lx type: 0x4)
[00:05.0] xen_pt_register_regions: IO region 2 registered (size=0x20000lx 
base_addr=0xf99c0000lx type: 0x4)
[00:05.0] xen_pt_register_regions: IO region 4 registered (size=0x100lx 
base_addr=0xa000lx type: 0x1)
[00:05.0] xen_pt_register_regions: Expansion ROM registered (size=0x00020000 
base_addr=0xf99a0000)

That seems to match ...

from guest lspci:
00:05.0 VGA compatible controller [0300]: Advanced Micro Devices [AMD] nee ATI 
Turks [Radeon HD 6570] [1002:6759] (prog-if 00 [VGA controller])
        Subsystem: PC Partner Limited Device [174b:e193]
        Physical Slot: 5
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
<MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 36
        Region 0: Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Region 2: Memory at f3040000 (64-bit, non-prefetchable) [size=128K]
        Region 4: I/O ports at c100 [size=256]
        Expansion ROM at f3060000 [disabled] [size=128K]

from the radeon driver in the guest:

[    1.283333] pci 0000:00:01.1: BAR 0: reserving [io  0x01f0-0x01f7 flags 
0x110] (d=0, p=0)
[    1.283333] pci 0000:00:01.1: BAR 1: reserving [io  0x03f6 flags 0x110] 
(d=0, p=0)
[    1.283333] pci 0000:00:01.1: BAR 2: reserving [io  0x0170-0x0177 flags 
0x110] (d=0, p=0)
[    1.283333] pci 0000:00:01.1: BAR 3: reserving [io  0x0376 flags 0x110] 
(d=0, p=0)
[    1.283333] pci 0000:00:01.1: BAR 4: reserving [io  0xc260-0xc26f flags 
0x40101] (d=0, p=0)
[    1.283333] pci 0000:00:01.2: BAR 4: reserving [io  0xc240-0xc25f flags 
0x40101] (d=0, p=0)
[    1.283333] pci 0000:00:02.0: BAR 0: reserving [mem 0xf0000000-0xf1ffffff 
flags 0x42208] (d=0, p=0)
[    1.283333] pci 0000:00:02.0: BAR 1: reserving [mem 0xf3094000-0xf3094fff 
flags 0x40200] (d=0, p=0)
[    1.283493] pci 0000:00:03.0: BAR 0: reserving [io  0xc000-0xc0ff flags 
0x40101] (d=0, p=0)
[    1.285517] pci 0000:00:03.0: BAR 1: reserving [mem 0xf2000000-0xf2ffffff 
flags 0x42208] (d=0, p=0)
[    1.286666] pci 0000:00:05.0: BAR 0: reserving [mem 0xe0000000-0xefffffff 
flags 0x14220c] (d=0, p=0)
[    1.286666] pci 0000:00:05.0: BAR 2: reserving [mem 0xf3040000-0xf305ffff 
flags 0x140204] (d=0, p=0)
[    1.286666] pci 0000:00:05.0: BAR 4: reserving [io  0xc100-0xc1ff flags 
0x40101] (d=0, p=0)
[    1.286666] pci 0000:00:06.0: BAR 0: reserving [mem 0xf3090000-0xf3093fff 
flags 0x140204] (d=0, p=0)

<snip>

[    1.744843] [drm] Initialized drm 1.1.0 20060810
[    1.746503] [drm] radeon kernel modesetting enabled.
[    1.747964] [drm:drm_pci_init], 
[    1.749911] [drm:drm_get_pci_dev], 
[    1.752812] xen: --> pirq=32 -> irq=36 (gsi=36)
[    1.755203] [drm:drm_get_minor], 
[    1.757902] [drm:drm_get_minor], new minor assigned 64
[    1.760337] [drm:drm_get_minor], 
[    1.762547] [drm:drm_get_minor], new minor assigned 0
[    1.765509] [drm] initializing kernel modesetting (TURKS 0x1002:0x6759 
0x174B:0xE193).
[    1.767705] [drm] register mmio base: 0xF3040000
[    1.769135] [drm] register mmio size: 131072
[    1.770551] [drm] radeon_atrm_get_bios: failed 
[    1.771878] [drm] radeon_acpi_vfct_bios: failed 
[    1.773193] [drm] igp_read_bios_from_vram: failed 
[    1.774744] [drm:radeon_read_bios], BIOS radeon_read_bios: size: 0x8c00 
biosiomemstart: 0xf3060000
[    1.777717] [drm] radeon_read_bios: succes 
[    1.779057] [drm:radeon_get_bios], BIOS headerstart: 0x2077 bios length: 6|8 
headerstring 0xc6 0x7c 0x0 0x0 
[    1.784874] [drm:radeon_get_bios], COMBIOS detected
[    1.786811] radeon 0000:00:05.0: Expecting atombios for evergreen GPU
[    1.788255] radeon 0000:00:05.0: Fatal error during GPU init
[    1.789919] [drm] radeon: finishing device.

So the address that the radeon_read_bios function uses, also matches.

So i'm pretty clueless how it ends up with the data from the cirrus rom when 
it's reading the right address ...


I'm using:
Xen:               Xen-unstable git:c5e9596
Linux dom0 + domU: 3.12-mergewindow + konrad's jumplabel patch (i also tested 
with 3.9.0 kernel)

I have attached the qemu-xen logs, but i have the qemu-xen-traditional logs as 
well.

Attached:
- xl-dmesg.txt      output of "xl dmesg" with start and shutdown of the guest
- xl-create.txt     output of "xl -vvvvv create" of the guest
- lspci-dom0.txt    output of "lspci -vvvknn" for dom0
- lspci-guest.txt   output of "lspci -vvvknn" for guest
- qemu-dm-guest.log
- dmesg-guest.txt

--

Sander

Attachment: dmesg-guest.txt
Description: Text document

Attachment: lspci-dom0.txt
Description: Text document

Attachment: lspci-guest.txt
Description: Text document

Attachment: qemu-dm-guest.log
Description: Binary data

Attachment: xl-create.txt
Description: Text document

Attachment: xl-dmesg.txt
Description: Text document

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel