Re: [Xen-users] GPU passthrough on Xen 4.4.0, FLReset-

[Gordan Bobic <gordan@xxxxxxxxxx>]

On 2014-06-03 02:12, Joshua Purcell wrote:
> I've made alot of progress on getting a Windows VM to have control
> over a 290x using Xen, but it's not yet working properly.
Let me guess - it works fine on a fresh host boot, but doesn't
work any more once you rebooot the domU? If so, it's a known
problem.

> None of my PCI devices (including the 290x) shows FLR (function level
> reset) enabled.
There are no GPUs known to support FLR.

> Is this a requirement for all GPUs that will be passed
> to a VM or is it just for swapping the card between dom0 and domU?
It is not an absolute requirement, no.

> Would I still need FLR enabled if I ensured that dom0 only sees loads
> xen-pciback module for my 290x? 'sudo lspci -vv' reports FLReset- for
> all devices (not just my 290x). First I need to ensure FLR is critical
> for a successful PCI (or GPU) passthrouth. If so, then this makes me
> think I need to either 1) enable something in the BIOS or 2) change a
> kernel or module parameter. I've heard that some motherboards (for
> instance the Intel DH55HC) have an option to enable FLR in the chipset
> section of the BIOS. My motherboard (ASRock Z97 Extreme6) has VT-d
> support, but doesn't have any FLR option in the BIOS.
FLR is not a BIOS or motherboard level feature, it is a PCI device
feature. If the card doesn't come with FLR there is nothing the
motherboard or the BIOS can magically do to help.

> Does anyone with experience regarding GPU passthrough have FLReset+
> next to their GPUs?
As I said above - there is no such thing as a GPU with FLR.

[...]

> Parsing config from /etc/xen/vwin.hvm
> libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel
> doesn't support reset from sysfs for PCI device 0000:01:00.0
> libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel
> doesn't support reset from sysfs for PCI device 0000:01:00.1
>
> The above is what first made me think there is an issue (which led me
> to find out more regarding FLR). When Windows boots I see the 290x,
> and when I run the AMD test to see what hardware I have it finds it as
> being a 290x. I can install the drivers for the card and no errors are
> reported. I reboot and look at Device Manager and I see the following
> message associated with the problem 290x device:
>
> This device cannot find enough free resources that it can use. (Code 
> 12)
Yup - that's the well known ATI reset problem.

Essentially it boils down to the fact that ATI drivers don't know
what to do with a GPU in a pre-initialized state and can't re-initialize
it. That's why you have to reboot the whole machine once you reboot
the domU.

IIRC _very_ recent kernels implement a PCI bus reset method of resetting
the device which should be good enough, but I am not sure what version
of Xen you have to run with that for both to be aware of the feature and
let you use it.

In the meantime, I can vouch for the Nvidia based solution - I have a
triple seat gaming/workstation system based on Nvidia cards and it works
lovely, and I can reboot domUs as many times as I want with no ill 
effects
because the Nvidia driver running in domU knows how to reinitialize a
card even if it is in a questionable state.

Technically only Quadro/Tesla/Grid cards are supported, but GTX480,
GTX680 and GTX780Ti cards are relatively trivial to modify into
equivalent Quadro/Tesla cards for purposes of getting GPU passthrough
to work. Nvidia driver has a device ID white-list and it won't 
initialize
the card if the device ID isn't in the white-list.

GTX480 -> Quadro 6000: BIOS mod
GTX680 -> Tesla K10: Remove 1 resistor off the back of the PCB
GTX780Ti -> Quadro K6000: Add 18K type 1206 resistor across two
pins on the EEPROM

I have a pair of 780Ti cards in my system (each passed to a different
VM) and it works just fine. I never managed to get any ATI cards to
work properly in the same setup. Probably best to stay away from
those if you want something that "just works".

Gordan

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users