Re: [Xen-users] Reporting success with VGA Passthrough, and some other question/issues, mainly with Audio

[Gordan Bobic <gordan@xxxxxxxxxx>]

On 01/04/2014 08:38 PM, Zir Blazer wrote:
> 1) VGA Passthrough
>
> At first, I used xl pci-assignable-add to manually add everytime I
> rebooted Dom0 the GPU and the HDMI device, but decided to add these to
> the syslinux.cfg file to skip that step. Either way, I didn't had any
> issues making the Radeon itself free to pass it to the VM (As I was
> using my Xeon Haswell integrated GPU as main Video Card, and didn't
> installed the Radeon Drivers on Dom0) when I used xl create and had the
> pci = option in the VM's CFG file. However, it either BSODed, or Windows
> was unable to use it as it appeared with a yellow exclamation mark while
> on Device Manager.
All xl pci-assignable-add does is detach the device from its current 
driver and assign it to the xen-pciback driver. The problem is that once 
the driver for the device loads, it initializes the device, which may 
leave it in a state that the driver in domU cannot deal with gracefully. 
This is particularly the case with ATI GPUs.
I'm not sure how your distro does it, but on EL6 I have a multi-pronged 
approach:
/etc/modprobe.d/xen-pciback.conf:
options xen-pciback permissive=1 
hide=(00:1a.1)(07:00.0)(07:00.1)(0d:00.0)(0d:00.1)(09:00.0)(0f:00.0)
Two GPUs, each with HDMI audio, two USB controllers and a sound card, if 
you're wondering
/etc/sysconfig/modules/xen-pciback.modules
#!/bin/sh

modprobe xen-pciback
modprobe nvidia

This ensures that the Nvidia devices to be passed through are grabbed by 
xen-pciback _BEFORE_ the Nvidia driver loads. I don't know if this is 
actually necessary with Nvidia - I don't think it is, but I still have 
it as a hangover from my futile attempts to get ATI cards to work 
properly for my setup before I finally gave up on them (both Linux and 
Windows drivers are just too fundamentally crippled and broken for my 
use-case).
> Googling around I found that the latest version of QEMU broke VGA
> Passthrough, and that using qemu-xen-traditional fixed it, which I
> through I was using. However, there was a problem with that. Using
> device_model = qemu-xen-traditional, as told by most Xen VGA Passthrough
> guides available currently, I got this error:
>
> WARNING: ignoring device_model directive.
> WARNING: Use "device_model_override" instead if you really want a
> non-default device_model
>
> I ignored that because the VM was successfully created, and besides,
> when I replaced device_model = qemu-dm with device_model_override =
> qemu-xen-traditional, it throwed another error which made it to not even
> create the VM. However, I recently discovered that I instead had to use
> device_model_version = qemu-xen-traditional. It worked pretty much
> flawlessly with that. Basically, there are a lot of guides, and even the
> Xen wiki, that are severely outdated in this area. I spend weeks trying
> to figure out what I was doing wrong due bad documentation, maybe
> because I didn't digged deep enough earlier, but still, most of the
> easily accessible data and google results are for older versions, and
> some critical options like device_model have changed.
>
> http://wiki.xen.org/wiki/Xen_Configuration_File_Options - old parameters
> which I was using
> http://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html - what I should
> have used on first place
>
> This is really worth writing about because I'm sure that someone else
> will sooner or later stumble here (Saw several people with this issue on
> google), as some guides assumes you're using an specific Linux
> distribution with an older Xen version instead of something bleeding edge.
Have you changed the incorrect info on the wiki? If you haven't, please 
do so - it is a wiki, after all.
> After finally being able to see Windows Desktop on the Monitor plugged
> to the Radeon 5770, I installed the Radeon 5770 Drivers from Device
> Manager with an INF file instead of the full Catalyst Control Center, as
> I hear than that gives more possible BSOD issues. Additionally, after
> around one week of playing around with the GPU on the VM (Even leaving a
> game open during all the night), I don't seem to notice issues, and the
> games I tried (Path of Exile, League of Legends) worked flawlessly with
> it. I only had a single GPU crash, with lost of Monitor signal and the
> VM destroying itself, but that may not be necesarrily attributable to Xen.
ATI drivers are buggy in all sorts of ways. Issues I have had include:

1) GPU-Z causes a crash (the host may survive it with PCI ACL support, 
my hardware lacks it so it crashes the whole host)
2) Automatic power management is broken, at least on my 7970 - the fan 
doesn't spin up according to the driver's power curve, possibly sensor 
access being broken in a VM (potentially related to why GPU-Z crashes). 
The net result is that the fan sits at 20% whatever you do until the GPU 
hits 90C. At that point the card's on-board power management kicks in 
and cranks the fan up to 100%. On a standard reference design card, fan 
when running at over 80% produces huge noise and vibration - enough to 
make the disks in the machine start to generate hundreds of pending 
sectors every day. The only way to wake the card up and get it back 
below 100% fan speed is to manually force the fan speed using the CCC 
(which is difficult if you don't have it installed for fear of it 
BSOD-ing the VM).
That's the issues with drivers being broken, I'm not going to go into 
the issues of the driver needlessly crippling the capabilities right now 
because they are probably less relevant to your use case.
> Also, I didn't notice any issues with the so-called "Soft Reset" or FLR,
> but maybe that was because I builded Xen with the Radeon patch included.
Does that issue a bus reset to reset the card?

> For example, I used xl destroy to abruptly shut down the VM. The Monitor
> shows a freezed image of the last image before I did that, yet I am able
> to create it again with the Video Card passing with no issues. I didn't
> notice either any performance degradation, albeit I'm not sure if that
> is under reboot scenarios or also applies when you shut down and create
> again the VM/DomU without restarting Dom0.
Applies to most scenarios. Weirdly, I found it doesn't happen on all 
cards. For example on a 7450 it is possible to reboot the VM without 
performance degradation and video corruption. On a 7970 I never managed 
to reboot the domU without it breaking. I hypothesized that this could 
be due to the 7450 hving no auxiliary power input which makes it more 
susceptible to actually getting reset via secondary means (e.g. via PCIe 
power saving to cut off power to the slot, whereas the 7970's auxiliary 
power inputs keep it alive when the power to the slot is switched off - 
which is arguably a hardware bug with the ATI cards' power management.
> Overally, it seems very functional and quite reliable. The only actual
> issue I find, is that when I create the VM, I need to have the VM window
> (Check attached file) on the main monitor as the active window,
> otherwise it seems that after Windows XP splash screen when it changes
> video resolution it usually either BSODs, or doesn't initialize the
> Video Card properly, and instead the Monitor stays in Standby while the
> VM window displays the Windows Desktop as if there was no VGA
> Passthrough being done.
Not sure what you are describing here. If you set the domU output to VNC 
this should matter. I only ever check VNC output when troubleshooting 
e.g. to see if there's a crash. I don't think I've checked it since I 
switched to Nvidia cards.
> 2) VNC vs SDL, Keyboard and Mouse focus
>
> I have tried with both VNC and SDL and I prefer the later. When I use
> SDL, the VM window automatically pops up, through that is rather
> unneeded as it later black screens as the Radeon takes control of the
> video output in the other Monitor. With VNC I have to manually launch
> vncviewer to be able to control the VM. Most important is that with SDL,
> when I click on the VM black window, control of the Keyboard and Mouse
> goes to the VM, while on VNC I didn't ever managed to get control of the
> Keyboard. The Mouse pointer works without having to make the VM window
> the active one as if I was using a freemouse tool on a windowed game,
> however, the VM black window surface on Dom0 doesn't allow me to travel
> the entire Desktop surface of the VM, so is rather useless.
VNC works fine for me when I use it, but most of the time I use a 
separate mouse/keyboard passed to the VM. My main setup is 3 GPUs, 
monitors, sound cards, keyboards and mice all running off of one 
physical machine.
> The only thing that annoys me from using SDL is that I have the black
> screen always open while the VM is working, and having to click it
> everytime I want to switch control to the VM is rather annoying. Isn't
> there any way to change Keyboard and Mouse focus from Dom0 to a DomU and
> viceversa like if they were consoles? For example, I may want to use
> Ctrl + Alt + F1 to get control of Dom0, then use Ctrl + Alt + F2 to
> switch control to the VM. This would increase usability.
There is no way to do that. As I said, I use separate 
mouse/keyboard/monitor for each VGA passthrough VM I use.
> 3) Disk Images
>
> As can be seen in the CFG file I copypasted, I'm using file: for my IMG
> Disk Images. However, on some other documentation like that of blktap2
> on the Xen wiki, it mentions that I can use tap:tapdisk:aio:, however,
> is there any reason why I should pick one over another? Do they also
> have specific format support or anything I should be careful with?
>
> Besides, is there any easy way to mount the IMG Disk Image files on
> Linux (Including NTFS partitions) so I can retrieve or add files when
> the VM is not running? I still didn't learned to setup Networks with
> Linux and need a workable way to move data from and to the VM Disk Images.
You need to install ntfs-3g package and do something like:

losetup /dev/loop0 /path/to/file.img
kpartx -a /dev/loop0
mount /dev/mapper/loop0p1 /path/to/mountpoint

Make sure this is never mounted while using the VM or you will very 
thoroughly destroy the FS.
> I know about LVM partitions, but find them a bit harder to manage than
> Disk Images. Plus performance is currently adequate this way.
Personally I use ZFS for anything nowdays. Try it and you will never 
look back.
> 4) Audio emulation
>
> While the Windows XP VM works nicely with games, I have the issue that
> there is no audio coming from it, which seems to be the most important
> missing thing before I can call my VM "production-ready" for playing
> games as if it were my old computer. I didn't tried to pass the
> integrated Realtek Sound Card, but that would be rather stupid as I need
> sound in both the current Linux Dom0, which I'm using for simple task
> like browsing, and the Windows XP VM for games. This means I have to
> rely on emulated Audio devices, which as far that I know are common.
> soundhw = 'ac97' seems to work, Windows XP recognizes the Sound Card and
> install the Drivers for it with no issues. It also has the HDMI device
> passed to it along the GPU. However, after googling a lot, I didn't find
> any easy to get audio from a DomU to Dom0 to get mixed so I can rely on
> emulated Sound Cards instead of needing one per VM like you need Video
> Cards.
The problem is that support for Intel HD Audio has only been added very 
recently in qemu, and traditional qemu doesn't have it. The other half 
of that problem is that XP and later don't have drivers for other 
emulated devices that traditional qemu supports (e.g. the once 
ubiquitous SB16).
> As far that I know, there are other VMMs like VirtualBox where you can
> usually get easily sound from the VMs, and it also uses QEMU-emulated
> Sound Cards as Xen does. Is there any reason why getting audio doesn't
> seem to be easy to do on Xen?
If you use the default qemu (a.k.a. upstream) rather than traditional, 
this may have the Intel HDA support and you can use Intel HDA driver in 
domU with it. There is no known driver that works properly with Windows 
XP or later for any of the old emulated devices traditional qemu can 
provide.
> I know that many people considers USB
> Sound Card cheap to pass to the VM, but I don't have easy access to
> those, so I would need a Software way to get sound from multiple VMs
> using a single Headphone connected to the integrated Realtek Sound Card.
USB audio devices are _cheap_ and easily available. I use these and they 
work great:
http://amzn.to/1gzLVAX

And with buy-one-get-one-free you get two for less than the price of a beer.

Note - you may find it preferable to pass through the USB controller via 
PCI passthrough, rather than USB device via USB passthrough (USB 
passthrough seems to chew through about 5% of a CPU core per device).


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users