[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] GPU passthrough on Xen 4.4.0, FLReset-

  • To: xen-users@xxxxxxxxxxxxx
  • From: Gordan Bobic <gordan@xxxxxxxxxx>
  • Date: Wed, 04 Jun 2014 08:59:55 +0100
  • Delivery-date: Wed, 04 Jun 2014 08:01:18 +0000
  • List-id: Xen user discussion <xen-users.lists.xen.org>

On 2014-06-03 02:12, Joshua Purcell wrote:
I've made alot of progress on getting a Windows VM to have control
over a 290x using Xen, but it's not yet working properly.

Let me guess - it works fine on a fresh host boot, but doesn't
work any more once you rebooot the domU? If so, it's a known

None of my PCI devices (including the 290x) shows FLR (function level
reset) enabled.

There are no GPUs known to support FLR.

Is this a requirement for all GPUs that will be passed
to a VM or is it just for swapping the card between dom0 and domU?

It is not an absolute requirement, no.

Would I still need FLR enabled if I ensured that dom0 only sees loads
xen-pciback module for my 290x? 'sudo lspci -vv' reports FLReset- for
all devices (not just my 290x). First I need to ensure FLR is critical
for a successful PCI (or GPU) passthrouth. If so, then this makes me
think I need to either 1) enable something in the BIOS or 2) change a
kernel or module parameter. I've heard that some motherboards (for
instance the Intel DH55HC) have an option to enable FLR in the chipset
section of the BIOS. My motherboard (ASRock Z97 Extreme6) has VT-d
support, but doesn't have any FLR option in the BIOS.

FLR is not a BIOS or motherboard level feature, it is a PCI device
feature. If the card doesn't come with FLR there is nothing the
motherboard or the BIOS can magically do to help.

Does anyone with experience regarding GPU passthrough have FLReset+
next to their GPUs?

As I said above - there is no such thing as a GPU with FLR.


Parsing config from /etc/xen/vwin.hvm
libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel
doesn't support reset from sysfs for PCI device 0000:01:00.0
libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel
doesn't support reset from sysfs for PCI device 0000:01:00.1

The above is what first made me think there is an issue (which led me
to find out more regarding FLR). When Windows boots I see the 290x,
and when I run the AMD test to see what hardware I have it finds it as
being a 290x. I can install the drivers for the card and no errors are
reported. I reboot and look at Device Manager and I see the following
message associated with the problem 290x device:

This device cannot find enough free resources that it can use. (Code 12)

Yup - that's the well known ATI reset problem.

Essentially it boils down to the fact that ATI drivers don't know
what to do with a GPU in a pre-initialized state and can't re-initialize
it. That's why you have to reboot the whole machine once you reboot
the domU.

IIRC _very_ recent kernels implement a PCI bus reset method of resetting
the device which should be good enough, but I am not sure what version
of Xen you have to run with that for both to be aware of the feature and
let you use it.

In the meantime, I can vouch for the Nvidia based solution - I have a
triple seat gaming/workstation system based on Nvidia cards and it works
lovely, and I can reboot domUs as many times as I want with no ill effects
because the Nvidia driver running in domU knows how to reinitialize a
card even if it is in a questionable state.

Technically only Quadro/Tesla/Grid cards are supported, but GTX480,
GTX680 and GTX780Ti cards are relatively trivial to modify into
equivalent Quadro/Tesla cards for purposes of getting GPU passthrough
to work. Nvidia driver has a device ID white-list and it won't initialize
the card if the device ID isn't in the white-list.

GTX480 -> Quadro 6000: BIOS mod
GTX680 -> Tesla K10: Remove 1 resistor off the back of the PCB
GTX780Ti -> Quadro K6000: Add 18K type 1206 resistor across two
pins on the EEPROM

I have a pair of 780Ti cards in my system (each passed to a different
VM) and it works just fine. I never managed to get any ATI cards to
work properly in the same setup. Probably best to stay away from
those if you want something that "just works".


Xen-users mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.