[Xen-users] GPU passthrough on Xen 4.4.0, FLReset-

[Joshua Purcell <joshua.purcell@xxxxxxxxx>]

I've made alot of progress on getting a Windows VM to have control
over a 290x using Xen, but it's not yet working properly. I have a ton
of details I can share on what I've done, but for now I have some
specific questions for those who have gone through these steps
successfuly on similar hardware. Let me know if any other details
would be helpful.

None of my PCI devices (including the 290x) shows FLR (function level
reset) enabled. Is this a requirement for all GPUs that will be passed
to a VM or is it just for swapping the card between dom0 and domU?
Would I still need FLR enabled if I ensured that dom0 only sees loads
xen-pciback module for my 290x? 'sudo lspci -vv' reports FLReset- for
all devices (not just my 290x). First I need to ensure FLR is critical
for a successful PCI (or GPU) passthrouth. If so, then this makes me
think I need to either 1) enable something in the BIOS or 2) change a
kernel or module parameter. I've heard that some motherboards (for
instance the Intel DH55HC) have an option to enable FLR in the chipset
section of the BIOS. My motherboard (ASRock Z97 Extreme6) has VT-d
support, but doesn't have any FLR option in the BIOS.

Does anyone with experience regarding GPU passthrough have FLReset+
next to their GPUs?

Here is the output of of 'xl dmesg':

 Xen 4.4.0
(XEN) Xen version 4.4.0 (joshua@xxxxxxxxxxxx) (gcc (GCC) 4.9.0
20140521 (prerelease)) debug=n Sat May 31 15:52:14 PDT 2014
(XEN) Latest ChangeSet:
(XEN) Bootloader: GRUB 2.02~beta2
(XEN) Command line: /xen-4.4.0.gz iommu=1 iommu_inclusive_mapping=1
dom0_mem=16G xsave=1
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 0 MBR signatures
(XEN)  Found 0 EDD information structures
(XEN) Multiboot-e820 RAM map:
(XEN)  0000000000000000 - 0000000000058000 (usable)
(XEN)  0000000000058000 - 0000000000059000 (reserved)
(XEN)  0000000000059000 - 000000000009f000 (usable)
(XEN)  000000000009f000 - 00000000000a0000 (reserved)
(XEN)  0000000000100000 - 0000000069e4c000 (usable)
(XEN)  0000000069e4c000 - 0000000069e53000 (ACPI NVS)
(XEN)  0000000069e53000 - 000000006ab54000 (usable)
(XEN)  000000006ab54000 - 000000006b085000 (reserved)
(XEN)  000000006b085000 - 000000007d185000 (usable)
(XEN)  000000007d185000 - 000000007d216000 (reserved)
(XEN)  000000007d216000 - 000000007d25d000 (usable)
(XEN)  000000007d25d000 - 000000007d398000 (ACPI NVS)
(XEN)  000000007d398000 - 000000007df7d000 (reserved)
(XEN)  000000007df7d000 - 000000007dfff000 type 20
(XEN)  000000007dfff000 - 000000007e000000 (usable)
(XEN)  000000007f000000 - 00000000bf200000 (reserved)
(XEN)  00000000f0000000 - 00000000f8000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fed00000 - 00000000fed04000 (reserved)
(XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ff000000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 000000083fe00000 (usable)
(XEN) ACPI: RSDP 000F0010, 0024 (r2 ALASKA)
(XEN) ACPI: XSDT 7D365088, 0094 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FACP 7D376090, 010C (r5 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: DSDT 7D3651B8, 10ED6 (r2 ALASKA    A M I       35 INTL 20120711)
(XEN) ACPI: FACS 7D397F80, 0040
(XEN) ACPI: APIC 7D3761A0, 0092 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FPDT 7D376238, 0044 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: SSDT 7D376280, 0539 (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
(XEN) ACPI: SSDT 7D3767C0, 0B74 (r1 CpuRef  CpuSsdt     3000 INTL 20051117)
(XEN) ACPI: SSDT 7D377338, 01C7 (r1  PmRef LakeTiny     3000 INTL 20051117)
(XEN) ACPI: MCFG 7D377500, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) ACPI: HPET 7D377540, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
(XEN) ACPI: SSDT 7D377578, 036D (r1 SataRe SataTabl     1000 INTL 20120711)
(XEN) ACPI: SSDT 7D3778E8, 5B5E (r1 SaSsdt  SaSsdt      3000 INTL 20120711)
(XEN) ACPI: AAFT 7D37D448, 049F (r1 ALASKA OEMAAFT   1072009 MSFT       97)
(XEN) ACPI: BGRT 7D37D8E8, 0038 (r0 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: DMAR 7D37D920, 00B8 (r1 INTEL      BDW         1 INTL        1)
(XEN) ACPI: SSDT 7D37D9D8, 0579 (r1 Intel_ IsctTabl     1000 INTL 20120711)
(XEN) System RAM: 31690MB (32450764kB)
(XEN) Domain heap initialised
(XEN) ACPI: 32/64X FACS address mismatch in FADT -
7d397f80/0000000000000000, using 32
(XEN) Processor #0 7:12 APIC version 21
(XEN) Processor #2 7:12 APIC version 21
(XEN) Processor #4 7:12 APIC version 21
(XEN) Processor #6 7:12 APIC version 21
(XEN) Processor #1 7:12 APIC version 21
(XEN) Processor #3 7:12 APIC version 21
(XEN) Processor #5 7:12 APIC version 21
(XEN) Processor #7 7:12 APIC version 21
(XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) Failed to enable Interrupt Remapping: Will not enable x2APIC.
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3598.990 MHz processor.
(XEN) Initing memory sharing.
(XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) spurious 8259A interrupt: IRQ7.
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 16 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN)  - VMCS shadowing
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Brought up 8 CPUs
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1f5b000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000820000000->0000000828000000 (4158762
pages to be allocated)
(XEN)  Init. ramdisk: 000000083f32a000->000000083fdff200
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff81f5b000
(XEN)  Init. ramdisk: ffffffff81f5b000->ffffffff82a30200
(XEN)  Phys-Mach map: ffffffff82a31000->ffffffff84a31000
(XEN)  Start info:    ffffffff84a31000->ffffffff84a314b4
(XEN)  Page tables:   ffffffff84a32000->ffffffff84a5b000
(XEN)  Boot stack:    ffffffff84a5b000->ffffffff84a5c000
(XEN)  TOTAL:         ffffffff80000000->ffffffff84c00000
(XEN)  ENTRY ADDRESS: ffffffff818ee1f0
(XEN) Dom0 has maximum 8 VCPUs
(XEN) Scrubbing Free RAM:
.....................................................................................................................................................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch
input to Xen)
(XEN) Freed 260kB init memory.

Here is my /etc/xen/vwin.hvm (vm config):

name = 'vwin'
builder = 'hvm'
memory = 8192
vcpus = 2
disk = [ '/home/joshua/vms/vwin.img,raw,xvda,w',
'file:/home/joshua/vms/win81.iso,hdc:cdrom,r' ]
vif = [ 'bridge=xenbr0,ip=192.168.1.115' ]
vnclisten = '0.0.0.0'
vnc = 1
pci = [ '01:00.0', '01:00.1' ]
pci_permissive = 1

When I successfully start the VM, here is the output:

Parsing config from /etc/xen/vwin.hvm
libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel
doesn't support reset from sysfs for PCI device 0000:01:00.0
libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel
doesn't support reset from sysfs for PCI device 0000:01:00.1

The above is what first made me think there is an issue (which led me
to find out more regarding FLR). When Windows boots I see the 290x,
and when I run the AMD test to see what hardware I have it finds it as
being a 290x. I can install the drivers for the card and no errors are
reported. I reboot and look at Device Manager and I see the following
message associated with the problem 290x device:

This device cannot find enough free resources that it can use. (Code 12)

I've looked around for other devices which may be conflicting with the
290x, and there is one other device which has no drivers currently.
Other than that there is another Microsoft display device in the same
section of the list next to the 290x. Disabling one or both of these
devices and rebooting has no effect.

Here's the relevant portion of my grub config:

multiboot /xen-4.4.0.gz /xen-4.4.0.gz iommu=1
iommu_inclusive_mapping=1 dom0_mem=16G xsave=1
module /vmlinuz-linux /vmlinuz-linux
root=UUID=917153ed-f386-4987-8fd0-9e293f8d969a rw
modprobe.blacklist=radeon i915 quiet rootflags=subvol=rootvol
console=tty0 xen-pciback.passthrough=1
xen-pciback.hide=(01:00.0)(01:00.1)

I think 'iommu=1' is related to previous versions of Xen, and I'm not
sure what 'iommu_inclusive_mapping=1' even does. I haven't verified
that I get the same current functionality without these lines yet.


Please request any other details if you think it would be helpful.
This is an up-to-date archlinux, Xen 4.4.0-4 system with the following
specs:

motherboard: ASRock Z97 Extreme6 (BIOS v1.30)
cpu: Intel i7-4790
memory: G.SKILL Ripjaws Z Series F3-2133C11Q-32GZL
domU OS: archlinux
domU a/v: Asus R9290X DirectCU II OC
dom0 OS: Windows8.1 64-bit
dom0 a/v: Intel HD Graphics 4600
ssd: 2x Samsung EVO 840 250GB
optical: LG Blu-ray UH12NS30
psu: Enermax Maxrevo EMR1500EWT

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users