Re: [Xen-users] Xen 4.3 Passthrough Problems & Documentation

[Gordan Bobic <gordan@xxxxxxxxxx>]

On Mon, 15 Jul 2013 11:33:32 -0400, Casey DeLorme <cdelorme@xxxxxxxxx> 
wrote:
>   - Upstream qemu fails to load virtual machines with VGA passthrough
>  and a large amount of memory (3600MB+ in my case breaks the DomU).
>
>  I may be wrong, but my understanding is that the PCI passthrough
>  related BAR memory patch was for qemu-traditional, not upstream.
>
> The patch points to files in the qemu-xen-dir not
> qemu-xen-traditional-dir, so I am pretty sure it is for upstream. ÂIf
> it wasn't then it should have had no effect when added, instead it
> broke my HVM using upstream-qemu.
Don't quote me on this (because I don't have time right now to
search through the xen-devel archives), but IIRC no changes
were made to qemu-upstream because it was deemed to not suffer
from the PCI memory overrun issue. But either way, if it
broke something, that's bad. You should probably fire off a report
on the -devel mailing list about it.

> SSD's are fast, so between fast and faster the line gets blurry I
> guess. ÂIf I was using an HDD it would probably be a different story.
It's the other way around - you'll notice _more_ difference on
an SSD because the overhead will be much greater compared to
what the disk is capable of. Without PV disk drivers my
bottleneck was qemu-dm running out of CPU in dom0.

>  Testing sysfs reset:
>
>  Modern linux kernel sysfs comes with reset files that can be used to
>  reset (some) PCI devices manually:
>
>  - [Kernel Docs
>
>
>
> (https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci)](https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci
> [3]
>   [3])
>
>  I decided to give this a try to see if it would allow me to reset 
> the
>  adapter from within Linux, where I could then tie a script to
> automate
>  the reset process when a DomU is rebooted.
>
>  The planned scenario:
>
>  - Windows boots and initializes the graphics card
>  - I shut down windows and the card remains initialized
>  - I reset the graphics card state by:
>  ÂÂ Â - Unbinding from pciback
>  Â Â - issuing a reset
>  Â Â - rebinding it
>  - Booting windows should initialize a fresh card
>
>  I think you'll find this process is entirely at the mercy of
>  what the driver does in domU. Quadro drivers seem to handle
>  this very gracefully.
>
> This is good news, because I am hoping Linux handles things the same
> way.
That would be good news if we knew exactly what the Nvidia
driver does in domU. But we don't, and are unlikely to
ever find out.

As I mentioned before, I gave up on ATI cards for more
than just this reason, and am happily using Quadrified
(and soon Gridified) GeForce cards for my requirements.

> Â Primary passthrough might work better because it re-executes
>  the BIOS which may well get the card to a clean state, but
>  I am purely guessing since I have given up on ATI cards
>  some time ago for number of reasons.
>
> That is possible, I never had luck getting primary passthrough 
> working
> before, maybe I will try again. ÂHowever then I have to use
> traditional qemu again, so again ideally I'd rather use upstream and
> work around secondary passthrough.Â
I hate to say this, but my experience is that if you are after
an easy option - using ATI cards isn't it.

Don't get me wrong - I still use an ATI 4850 card for dom0
on my system; but that is only because:

1) I cannot get anything but ATI cards to work in slot 1
on my EVGA SR-2 (bizzare as that may sound, tried 7 different
Nvidia cards, 1 sound card, 2 disk controllers and 2 ATI
cards - and of all those only the 2 ATI cards worked in
that slot)

2) 4850 because Radeon 4xxx is the last series that had
two dual-link DVI ports (all later cards only have single
link wired into the 2nd DVI port)

3) It means I can just blacklist nvidia in dom0 and not
have to selectively work around the Nvidia GPU I use
for dom0.

4) It is the best available within the above constraints
in single-slot form factor.

Using the right tool for the job helps. :)

>   - I am to believe that Windows ejection is probably working because
> it
>  is using AMD drivers.
>
>  Ejecting a Quadro card on Win7 "worked" for me, but I never
>  actually saw any benefit from doing so with Quadro cards
>  since they work fine after a domU reboot anyway.
>
> If I could achieve that with an AMD I would be happy, but I haven't
> found any good instructions on how to actually mod the GTX to Quadro
> that doesn't involve hardware modifications.
I'm in the process of writing up a detailed article on this at the
moment, but 5xx and earlier series cards only require soft-modding.

The only 5xx (still soft-moddable) series card worth considering
is the 580 which is modifiable into a Quadro 7000 - but it's
pointless because 7000 isn't "MultiOS" capable, so the Quadro
driver doesn't do it's magic to make it work in domU.

6xx series cards do require a small hardware modification in addition
to BIOS modification, and to do the BIOS modification properly
you have to strip out all of the UEFI crap they have put in there
(also doable - not tested my method yet, but will do so when
time permits).

FYI, I'm currently running modified GTS450 (Q2000), GTX470 (Q5000)
and GTX480 (Q6000) cards.

>   - The reset in Linux fails when it has no drivers so the reset
>  probably triggers a driver operation
>
>  You have a reset option under /sys/ when the driver is loaded?
>  I've never seen that. I thought it was specifically related
>  to FLreset PCI level functionality.
>
> Supposedly the reset files were an alternative addition to the
> `do_flr`? ÂI did read a little bit about it, but not much by way of
> documentation around it yet.
So presumably that depends on the driver implementing the reset.

>   - The driver operation probably fails because it is not tied to an
> AMD
>  driver
>
>  And you have definitely confirmed that it does something (or even
>  exists) when the fglrx driver is claiming the device?
>
> I have not, but if it's anything like Windows then this is exactly
> what should be happening right? ÂI am basing this off of that thought
> and the fact that if no driver is attached the reset throws an error.
> ÂIt's all speculation right now, I was hoping someone with knowledge
> about pciback or sysfs could confirm it.
I really wouldn't make any assumptions about what the GPU driver
might or might not do, especially since my experience, especially
in the case of ATI, is that their Windows and Linux drivers are
quite fundamentally different.

> If that is the case there is a strong possibility attaching it to say
> the radeon or fglrx driver would handle a reset properly.
>
> I did test resetting emulated graphics in a virtual machine
> successfully, so I can say that the reset appears to do "something".Â
You are probably better of going the primary passthrough route and
getting the BIOS to re-post the card.

The only reason why I haven't tried that with my Quadrified cards
is because, well, they work as it is. I can live with the loading
screen only being available in VNC.

>   Another option I have not yet tested would be loading the radon
> driver
>  to bind and unbind it before adding it back to pciback, which may
>  cause the proper reset chain to occur. ÂI didn't see it in the
>  drivers list though and wouldn't know where to begin loading it
>  without causing problems.
>
>  Well, you can modprobe fglrx and see if/what it breaks. :)
>
> Good idea, I will have to install fglrx first, but hopefully that 
> will
> load the driver into `/sys/bus/pci/drivers`.
Good luck. :)

>   If anyone knows how to cause a D0>D3>D0 power change to a device
>  through sysfs let me know because I would like to try that next.
>
>  Hmm... Abusing power management - I like the idea. :)
>  It is not likely to work if the card takes auxiliary power
>  input, though. :(
>
> Hmm good point, it does take two auxiliary power inputs. ÂI thought
> D0/D3 operations were for device hibernation, does auxiliary power
> prevent that from being possible?
No idea. On the SR-2 I tried toggling PCIe slot enable jumpers to try
to reset the card at runtime, but it didn't do anything of the sort.
As far as I can tell, it only tells the BIOS to hide the device at
POST time.

Gordan

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users