Xen project Mailing List

On Sun, Jan 5, 2014 at 11:59 AM, Gordan Bobic <gordan@xxxxxxxxxx> wrote:

On 01/04/2014 08:38 PM, Zir Blazer wrote:

1) VGA Passthrough

At first, I used xl pci-assignable-add to manually add everytime I
rebooted Dom0 the GPU and the HDMI device, but decided to add these to
the syslinux.cfg file to skip that step. Either way, I didn't had any
issues making the Radeon itself free to pass it to the VM (As I was
using my Xeon Haswell integrated GPU as main Video Card, and didn't
installed the Radeon Drivers on Dom0) when I used xl create and had the
pci = option in the VM's CFG file. However, it either BSODed, or Windows
was unable to use it as it appeared with a yellow exclamation mark while
on Device Manager.

All xl pci-assignable-add does is detach the device from its current driver and assign it to the xen-pciback driver. The problem is that once the driver for the device loads, it initializes the device, which may leave it in a state that the driver in domU cannot deal with gracefully. This is particularly the case with ATI GPUs.

I'm not sure how your distro does it, but on EL6 I have a multi-pronged approach:

/etc/modprobe.d/xen-pciback.conf:
options xen-pciback permissive=1 hide=(00:1a.1)(07:00.0)(07:00.1)(0d:00.0)(0d:00.1)(09:00.0)(0f:00.0)

Two GPUs, each with HDMI audio, two USB controllers and a sound card, if you're wondering

/etc/sysconfig/modules/xen-pciback.modules
#!/bin/sh

modprobe xen-pciback
modprobe nvidia

This ensures that the Nvidia devices to be passed through are grabbed by xen-pciback _BEFORE_ the Nvidia driver loads. I don't know if this is actually necessary with Nvidia - I don't think it is, but I still have it as a hangover from my futile attempts to get ATI cards to work properly for my setup before I finally gave up on them (both Linux and Windows drivers are just too fundamentally crippled and broken for my use-case).

Googling around I found that the latest version of QEMU broke VGA
Passthrough, and that using qemu-xen-traditional fixed it, which I
through I was using. However, there was a problem with that. Using
device_model = qemu-xen-traditional, as told by most Xen VGA Passthrough
guides available currently, I got this error:

WARNING: ignoring device_model directive.
WARNING: Use "device_model_override" instead if you really want a
non-default device_model

I ignored that because the VM was successfully created, and besides,
when I replaced device_model = qemu-dm with device_model_override =
qemu-xen-traditional, it throwed another error which made it to not even
create the VM. However, I recently discovered that I instead had to use
device_model_version = qemu-xen-traditional. It worked pretty much
flawlessly with that. Basically, there are a lot of guides, and even the
Xen wiki, that are severely outdated in this area. I spend weeks trying
to figure out what I was doing wrong due bad documentation, maybe
because I didn't digged deep enough earlier, but still, most of the
easily accessible data and google results are for older versions, and
some critical options like device_model have changed.

http://wiki.xen.org/wiki/Xen_Configuration_File_Options - old parameters
which I was using
http://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html - what I should
have used on first place

This is really worth writing about because I'm sure that someone else
will sooner or later stumble here (Saw several people with this issue on
google), as some guides assumes you're using an specific Linux
distribution with an older Xen version instead of something bleeding edge.

Have you changed the incorrect info on the wiki? If you haven't, please do so - it is a wiki, after all.

After finally being able to see Windows Desktop on the Monitor plugged
to the Radeon 5770, I installed the Radeon 5770 Drivers from Device
Manager with an INF file instead of the full Catalyst Control Center, as
I hear than that gives more possible BSOD issues. Additionally, after
around one week of playing around with the GPU on the VM (Even leaving a
game open during all the night), I don't seem to notice issues, and the
games I tried (Path of Exile, League of Legends) worked flawlessly with
it. I only had a single GPU crash, with lost of Monitor signal and the
VM destroying itself, but that may not be necesarrily attributable to Xen.

ATI drivers are buggy in all sorts of ways. Issues I have had include:

1) GPU-Z causes a crash (the host may survive it with PCI ACL support, my hardware lacks it so it crashes the whole host)

2) Automatic power management is broken, at least on my 7970 - the fan doesn't spin up according to the driver's power curve, possibly sensor access being broken in a VM (potentially related to why GPU-Z crashes). The net result is that the fan sits at 20% whatever you do until the GPU hits 90C. At that point the card's on-board power management kicks in and cranks the fan up to 100%. On a standard reference design card, fan when running at over 80% produces huge noise and vibration - enough to make the disks in the machine start to generate hundreds of pending sectors every day. The only way to wake the card up and get it back below 100% fan speed is to manually force the fan speed using the CCC (which is difficult if you don't have it installed for fear of it BSOD-ing the VM).

That's the issues with drivers being broken, I'm not going to go into the issues of the driver needlessly crippling the capabilities right now because they are probably less relevant to your use case.

Also, I didn't notice any issues with the so-called "Soft Reset" or FLR,
but maybe that was because I builded Xen with the Radeon patch included.

Does that issue a bus reset to reset the card?

For example, I used xl destroy to abruptly shut down the VM. The Monitor
shows a freezed image of the last image before I did that, yet I am able
to create it again with the Video Card passing with no issues. I didn't
notice either any performance degradation, albeit I'm not sure if that
is under reboot scenarios or also applies when you shut down and create
again the VM/DomU without restarting Dom0.

Applies to most scenarios. Weirdly, I found it doesn't happen on all cards. For example on a 7450 it is possible to reboot the VM without performance degradation and video corruption. On a 7970 I never managed to reboot the domU without it breaking. I hypothesized that this could be due to the 7450 hving no auxiliary power input which makes it more susceptible to actually getting reset via secondary means (e.g. via PCIe power saving to cut off power to the slot, whereas the 7970's auxiliary power inputs keep it alive when the power to the slot is switched off - which is arguably a hardware bug with the ATI cards' power management.

Overally, it seems very functional and quite reliable. The only actual
issue I find, is that when I create the VM, I need to have the VM window
(Check attached file) on the main monitor as the active window,
otherwise it seems that after Windows XP splash screen when it changes
video resolution it usually either BSODs, or doesn't initialize the
Video Card properly, and instead the Monitor stays in Standby while the
VM window displays the Windows Desktop as if there was no VGA
Passthrough being done.

Not sure what you are describing here. If you set the domU output to VNC this should matter. I only ever check VNC output when troubleshooting e.g. to see if there's a crash. I don't think I've checked it since I switched to Nvidia cards.

2) VNC vs SDL, Keyboard and Mouse focus

I have tried with both VNC and SDL and I prefer the later. When I use
SDL, the VM window automatically pops up, through that is rather
unneeded as it later black screens as the Radeon takes control of the
video output in the other Monitor. With VNC I have to manually launch
vncviewer to be able to control the VM. Most important is that with SDL,
when I click on the VM black window, control of the Keyboard and Mouse
goes to the VM, while on VNC I didn't ever managed to get control of the
Keyboard. The Mouse pointer works without having to make the VM window
the active one as if I was using a freemouse tool on a windowed game,
however, the VM black window surface on Dom0 doesn't allow me to travel
the entire Desktop surface of the VM, so is rather useless.

VNC works fine for me when I use it, but most of the time I use a separate mouse/keyboard passed to the VM. My main setup is 3 GPUs, monitors, sound cards, keyboards and mice all running off of one physical machine.

The only thing that annoys me from using SDL is that I have the black
screen always open while the VM is working, and having to click it
everytime I want to switch control to the VM is rather annoying. Isn't
there any way to change Keyboard and Mouse focus from Dom0 to a DomU and
viceversa like if they were consoles? For example, I may want to use
Ctrl + Alt + F1 to get control of Dom0, then use Ctrl + Alt + F2 to
switch control to the VM. This would increase usability.

There is no way to do that. As I said, I use separate mouse/keyboard/monitor for each VGA passthrough VM I use.

3) Disk Images

As can be seen in the CFG file I copypasted, I'm using file: for my IMG
Disk Images. However, on some other documentation like that of blktap2
on the Xen wiki, it mentions that I can use tap:tapdisk:aio:, however,
is there any reason why I should pick one over another? Do they also
have specific format support or anything I should be careful with?

Besides, is there any easy way to mount the IMG Disk Image files on
Linux (Including NTFS partitions) so I can retrieve or add files when
the VM is not running? I still didn't learned to setup Networks with
Linux and need a workable way to move data from and to the VM Disk Images.

You need to install ntfs-3g package and do something like:

losetup /dev/loop0 /path/to/file.img
kpartx -a /dev/loop0
mount /dev/mapper/loop0p1 /path/to/mountpoint

Make sure this is never mounted while using the VM or you will very thoroughly destroy the FS.

I know about LVM partitions, but find them a bit harder to manage than
Disk Images. Plus performance is currently adequate this way.

Personally I use ZFS for anything nowdays. Try it and you will never look back.

4) Audio emulation

While the Windows XP VM works nicely with games, I have the issue that
there is no audio coming from it, which seems to be the most important
missing thing before I can call my VM "production-ready" for playing
games as if it were my old computer. I didn't tried to pass the
integrated Realtek Sound Card, but that would be rather stupid as I need
sound in both the current Linux Dom0, which I'm using for simple task
like browsing, and the Windows XP VM for games. This means I have to
rely on emulated Audio devices, which as far that I know are common.
soundhw = 'ac97' seems to work, Windows XP recognizes the Sound Card and
install the Drivers for it with no issues. It also has the HDMI device
passed to it along the GPU. However, after googling a lot, I didn't find
any easy to get audio from a DomU to Dom0 to get mixed so I can rely on
emulated Sound Cards instead of needing one per VM like you need Video
Cards.

The problem is that support for Intel HD Audio has only been added very recently in qemu, and traditional qemu doesn't have it. The other half of that problem is that XP and later don't have drivers for other emulated devices that traditional qemu supports (e.g. the once ubiquitous SB16).

As far that I know, there are other VMMs like VirtualBox where you can
usually get easily sound from the VMs, and it also uses QEMU-emulated
Sound Cards as Xen does. Is there any reason why getting audio doesn't
seem to be easy to do on Xen?

If you use the default qemu (a.k.a. upstream) rather than traditional, this may have the Intel HDA support and you can use Intel HDA driver in domU with it. There is no known driver that works properly with Windows XP or later for any of the old emulated devices traditional qemu can provide.

I know that many people considers USB
Sound Card cheap to pass to the VM, but I don't have easy access to
those, so I would need a Software way to get sound from multiple VMs
using a single Headphone connected to the integrated Realtek Sound Card.

USB audio devices are _cheap_ and easily available. I use these and they work great:

http://amzn.to/1gzLVAX

And with buy-one-get-one-free you get two for less than the price of a beer.

Note - you may find it preferable to pass through the USB controller via PCI passthrough, rather than USB device via USB passthrough (USB passthrough seems to chew through about 5% of a CPU core per device).

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users

Re: [Xen-users] Reporting success with VGA Passthrough, and some other question/issues, mainly with Audio