[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] Xen 4.3 Passthrough Problems & Documentation

On Mon, 15 Jul 2013 11:33:32 -0400, Casey DeLorme <cdelorme@xxxxxxxxx> wrote:
  - Upstream qemu fails to load virtual machines with VGA passthrough
 and a large amount of memory (3600MB+ in my case breaks the DomU).

 I may be wrong, but my understanding is that the PCI passthrough
 related BAR memory patch was for qemu-traditional, not upstream.

The patch points to files in the qemu-xen-dir not
qemu-xen-traditional-dir, so I am pretty sure it is for upstream. ÂIf
it wasn't then it should have had no effect when added, instead it
broke my HVM using upstream-qemu.

Don't quote me on this (because I don't have time right now to
search through the xen-devel archives), but IIRC no changes
were made to qemu-upstream because it was deemed to not suffer
from the PCI memory overrun issue. But either way, if it
broke something, that's bad. You should probably fire off a report
on the -devel mailing list about it.

SSD's are fast, so between fast and faster the line gets blurry I
guess. ÂIf I was using an HDD it would probably be a different story.

It's the other way around - you'll notice _more_ difference on
an SSD because the overhead will be much greater compared to
what the disk is capable of. Without PV disk drivers my
bottleneck was qemu-dm running out of CPU in dom0.

 Testing sysfs reset:

 Modern linux kernel sysfs comes with reset files that can be used to
 reset (some) PCI devices manually:

 - [Kernel Docs


I decided to give this a try to see if it would allow me to reset the
 adapter from within Linux, where I could then tie a script to
 the reset process when a DomU is rebooted.

 The planned scenario:

 - Windows boots and initializes the graphics card
 - I shut down windows and the card remains initialized
 - I reset the graphics card state by:
 ÂÂ Â - Unbinding from pciback
 Â Â - issuing a reset
 Â Â - rebinding it
 - Booting windows should initialize a fresh card

 I think you'll find this process is entirely at the mercy of
 what the driver does in domU. Quadro drivers seem to handle
 this very gracefully.

This is good news, because I am hoping Linux handles things the same

That would be good news if we knew exactly what the Nvidia
driver does in domU. But we don't, and are unlikely to
ever find out.

As I mentioned before, I gave up on ATI cards for more
than just this reason, and am happily using Quadrified
(and soon Gridified) GeForce cards for my requirements.

 Primary passthrough might work better because it re-executes
 the BIOS which may well get the card to a clean state, but
 I am purely guessing since I have given up on ATI cards
 some time ago for number of reasons.

That is possible, I never had luck getting primary passthrough working
before, maybe I will try again. ÂHowever then I have to use
traditional qemu again, so again ideally I'd rather use upstream and
work around secondary passthrough.Â

I hate to say this, but my experience is that if you are after
an easy option - using ATI cards isn't it.

Don't get me wrong - I still use an ATI 4850 card for dom0
on my system; but that is only because:

1) I cannot get anything but ATI cards to work in slot 1
on my EVGA SR-2 (bizzare as that may sound, tried 7 different
Nvidia cards, 1 sound card, 2 disk controllers and 2 ATI
cards - and of all those only the 2 ATI cards worked in
that slot)

2) 4850 because Radeon 4xxx is the last series that had
two dual-link DVI ports (all later cards only have single
link wired into the 2nd DVI port)

3) It means I can just blacklist nvidia in dom0 and not
have to selectively work around the Nvidia GPU I use
for dom0.

4) It is the best available within the above constraints
in single-slot form factor.

Using the right tool for the job helps. :)

  - I am to believe that Windows ejection is probably working because
 is using AMD drivers.

 Ejecting a Quadro card on Win7 "worked" for me, but I never
 actually saw any benefit from doing so with Quadro cards
 since they work fine after a domU reboot anyway.

If I could achieve that with an AMD I would be happy, but I haven't
found any good instructions on how to actually mod the GTX to Quadro
that doesn't involve hardware modifications.

I'm in the process of writing up a detailed article on this at the
moment, but 5xx and earlier series cards only require soft-modding.

The only 5xx (still soft-moddable) series card worth considering
is the 580 which is modifiable into a Quadro 7000 - but it's
pointless because 7000 isn't "MultiOS" capable, so the Quadro
driver doesn't do it's magic to make it work in domU.

6xx series cards do require a small hardware modification in addition
to BIOS modification, and to do the BIOS modification properly
you have to strip out all of the UEFI crap they have put in there
(also doable - not tested my method yet, but will do so when
time permits).

FYI, I'm currently running modified GTS450 (Q2000), GTX470 (Q5000)
and GTX480 (Q6000) cards.

  - The reset in Linux fails when it has no drivers so the reset
 probably triggers a driver operation

 You have a reset option under /sys/ when the driver is loaded?
 I've never seen that. I thought it was specifically related
 to FLreset PCI level functionality.

Supposedly the reset files were an alternative addition to the
`do_flr`? ÂI did read a little bit about it, but not much by way of
documentation around it yet.

So presumably that depends on the driver implementing the reset.

  - The driver operation probably fails because it is not tied to an

 And you have definitely confirmed that it does something (or even
 exists) when the fglrx driver is claiming the device?

I have not, but if it's anything like Windows then this is exactly
what should be happening right? ÂI am basing this off of that thought
and the fact that if no driver is attached the reset throws an error.
ÂIt's all speculation right now, I was hoping someone with knowledge
about pciback or sysfs could confirm it.

I really wouldn't make any assumptions about what the GPU driver
might or might not do, especially since my experience, especially
in the case of ATI, is that their Windows and Linux drivers are
quite fundamentally different.

If that is the case there is a strong possibility attaching it to say
the radeon or fglrx driver would handle a reset properly.

I did test resetting emulated graphics in a virtual machine
successfully, so I can say that the reset appears to do "something".Â

You are probably better of going the primary passthrough route and
getting the BIOS to re-post the card.

The only reason why I haven't tried that with my Quadrified cards
is because, well, they work as it is. I can live with the loading
screen only being available in VNC.

  Another option I have not yet tested would be loading the radon
 to bind and unbind it before adding it back to pciback, which may
 cause the proper reset chain to occur. ÂI didn't see it in the
 drivers list though and wouldn't know where to begin loading it
 without causing problems.

 Well, you can modprobe fglrx and see if/what it breaks. :)

Good idea, I will have to install fglrx first, but hopefully that will
load the driver into `/sys/bus/pci/drivers`.

Good luck. :)

  If anyone knows how to cause a D0>D3>D0 power change to a device
 through sysfs let me know because I would like to try that next.

 Hmm... Abusing power management - I like the idea. :)
 It is not likely to work if the card takes auxiliary power
 input, though. :(

Hmm good point, it does take two auxiliary power inputs. ÂI thought
D0/D3 operations were for device hibernation, does auxiliary power
prevent that from being possible?

No idea. On the SR-2 I tried toggling PCIe slot enable jumpers to try
to reset the card at runtime, but it didn't do anything of the sort.
As far as I can tell, it only tells the BIOS to hide the device at
POST time.


Xen-users mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.