Re: [Xen-users] Recommendations for Virtulization Hardware

[Robin Axelsson <gu99roax@xxxxxxxxxxxxxxxxxxx>]

If you want to be able to use PCI and VGA passthrough you basically need 
to make sure that your hardware supports either AMD-Vi (formerly known 
as AMD-IOMMU) or Intel VT-d extensions. In the Intel case it limits your 
choice of Motherboard (it must be supported in the BIOS) and CPU. In the 
AMD case it limits only your choice of motherboard. A good start is to 
check out one of these pages:
http://wiki.xensource.com/xenwiki/VTdHowTo
http://wiki.xen.org/wiki/VTd_HowTo

A word of warning here is that parts of the documentation is somewhat 
dated. You can also communicate with e.g. Gigabyte, Asus or ASRock 
customer support and ask them if a particular motherboard supports these 
extensions. Most motherboards also have downloadable user manuals, if 
the BIOS settings in those pages shows options to enable/disable VT-d or 
AMD-Vi/IOMMU extensions then you will be ok with that motherboard.
The other thing is choice of GPU for VGA passthrough and it is 
preferable that the GPU supports FLR or Function Level Reset as it is 
called. Thing is that the hardware needs to be reset somehow as it is 
passed through to the host. This is best done with FLR and nVidia is 
known to supply firmware patches for some of their Geforce cards with 
this support and it is said to be supported by default with their Quadro 
cards. FLR is not the only way to reset a PCI device, a reset could be 
trigged through the ACPI power management framework by temporarily 
cutting power to the affected PCI slot. These reset methods are called 
d3d0 and bus reset. The question however, is if this works on PCI cards 
that use auxiliary power directly from the PSU. There is a pdf document 
on the VMWare website 
(http://www.vmware.com/files/pdf/techpaper/vsp_4_vmdirectpath_host.pdf) 
about this:
-----------------------
Reset Method

Possible values for the reset method include flr, d3d0, link, bridge, or 
default.
The default setting is described as follows. If a device supports 
function level reset (FLR), ESX always uses FLR. If the device does not 
support FLR, ESX next defaults to link reset and bus reset in that 
order. Link reset and bus reset might prevent some devices from being 
assigned to different virtual machines, or from being assigned between 
the VMkernel and virtual machines. In the absence of FLR, it is possible 
to use PCI Power Management capability (D3 to D0 transitions) to trigger 
a reset. Most of the Intel NICs and various other HBAs support this mode.
-----------------------


There are indications from people that d3d0 also work with PCI cards 
that take power from auxiliary inputs. I suggest that you take a look at 
the following youtube clip and read the comments there:
http://www.youtube.com/watch?v=Gtmwnx-k2qg

So it seems that it works although it may be a bit more quirky. It 
doesn't hurt to take that discussion (particularly about FLR support) 
with nVidia and/or AMD.
When it comes to virtualization, the technology has come very far, but 
it is still lacking considerably when it comes to sharing GPUs and also 
to some degree when it comes to sharing I/O devices (especially when you 
intend to run many virtual machines on a single system). The GPU today 
consists of three types of components; the processing unit, graphics 
memory and the video output unit/adapter and it is not clear as to how 
to share these components seamlessly between the host and virtual 
machines with minimal overhead. Whereas there are VT-x extensions that 
allows you to pretty seamlessly share CPU cores between VMs and the host 
there are currently none for the processing unit. It is also not clear 
how the hardware can assist with sharing TV/monitor screen estate 
between machines with all 3D effects such as Aqua for Win7 and the 
whatnot enabled for all machines. Especially when considering the 
dynamics of plugging and unplugging computer monitors to 
multiport/eyefinity graphics cards and the ability to change screen 
resolution. Things are improving for sure and a lot of research is 
likely going into this. I don't know what's happening in the GPU 
frontline but I know that the next thing with passthrough is the SR-IOV 
that allows PCI units to present several virtual instances of oneself to 
several virtual machines. It's a cool thing, I recommend further reading 
about this here:
http://www.intel.com/content/www/us/en/pci-express/pci-sig-sr-iov-primer-sr-iov-technology-paper.html
http://blog.scottlowe.org/2009/12/02/what-is-sr-iov/

It is likely to take a few years before something useful will come out 
of it. In the meanwhile, unless you want to use several GPUs which might 
not be a bad thing as a lot of monitors these days have several inputs, 
you can resort to using a remote desktop client to integrate one machine 
with another. Virtualbox for example use RDP through which you can 
interact with your virtual machine. In a similar manner you can set up a 
VNC server on your Linux host and establish a connection to it through 
your Windows VM. You will not get full 3D functionality (such as Aqua) 
through the client although there is a growing support for it through 
VirtualGL extensions that are coming to VNC and perhaps the Spice 
protocol. But some clients might even allow for seamless mode that lets 
you mix Linux and Windows windows on the same desktop like this for example:
http://i.techrepublic.com.com/blogs/seamless.png
http://www.youtube.com/watch?v=eQr8iI0yZH4


Just keep in mind that this is still a little bit of uncharted territory 
so there may be a few bumps on the way and it may not work as smooth as 
you would desire.




I see that your demands are somewhat multifaceted. I believe that you 
also want to use diffent services such as using your machine as a file 
server with the possible intention of using filesystems such as ZFS. If 
you do, you should be careful with your selection of hardware for these 
particular purposes. If you want to get full protection against data 
corruption from ZFS, your choice of hardware gets rather limited when it 
comes to choice of hard drives, host bus adapter and network controller. 
The most stable implementation of ZFS is found with Illumos based 
operating systems (such as OpenIndiana, SmartOS, OmniOS, Belenix etc) or 
Solaris if you choose to download it from Oracle's website. With these 
operating systems you are most likely to want to use hardware that has 
certified drivers for it. That way you are less likely to run into 
problems later on. That implies that you will be limited to choosing 
Intel based network adapters and LSI based SAS controllers. There should 
be _no_ hardware RAID functionality in the SAS controller that merely 
should be run in IT mode (or Initiator-Target mode). That requires the 
LSI controller to be flashed with IT firmware in most cases. The 
objective here is to make sure that _all_ errors that might occur with 
the hard drives are reported all the way to the software level and that 
nothing is concealed of obfuscated by internal error handling in the 
hardware. It is therefore recommended to use SAS hard drives instead of 
S-ATA (which also are fully compatible with SAS controllers). SAS hard 
drives are not much more expensive than similar SATA drives and you get 
a higher reliability out of them. It is also recommended to have at 
least two drive redundancy simply because if one drive is dead and you 
swap it, it is not uncommon that another drive dies in the rebuild 
process of the RAID cluster because of the added strain the rebuild 
process (or 'resilvering' as it is called in Solaris terms) put on the 
drives. Of course, the system should communicate directly to the hard 
drive hardware and not be obfuscated by some virtual abstraction layer 
in between which means that you either run ZFS on the metal or through 
PCI passthrough of the SAS (and perhaps also network) adapters. Also, it 
is highly recommended that you use ECC RAM for such applications and it 
doesn't hurt to dedicate a few gigs of it to the ZFS as RAM is used for 
cache. The good news is that most motherboards with good chipsets 
support ECC RAM even though you might not find anything about it in the 
user manuals.
I admire your persistence with pursuing this undertaking and wish you 
the best of luck with it!
Robin.

On 2012-09-20 23:12, ShadesOfGrey wrote:
> I'm looking to build a new personal computer.  I want it to function 
> as a Linux desktop, provide network services for my home, and lastly, 
> occasional Windows gaming.  From what I've gathered, virtualization 
> using a Type 1 Hypervisor supporting PCI/VGA pass-through like KVM or 
> Xen would be an attractive solution for my needs.  For reference, 
> reading these threads on Ars Technica may be helpful to understand 
> where I'm coming from, 
> http://arstechnica.com/civis/viewtopic.php?f=6&t=1175674 and 
> http://arstechnica.com/civis/viewtopic.php?f=11&t=1181867. But 
> basically, I use Linux as my primary OS and would rather avoid dual 
> booting or building two boxes just to play Windows games when I want 
> to play Windows games.  I'm also intrigued by the concept of 
> virtualization and would like to experiment with it as a solution for 
> my case.
> My problem is isolating which hardware to choose, specifically which 
> combination of CPU, motherboard and video card.  Previously I had been 
> relying on web searches to glean information from gaming and 
> enthusiast web sites and tech specs from motherboard manufacturers.  
> After what I learned during my participation in the referenced threads 
> at Ars Technica, I find myself back at square one.  Instead of trying 
> to guess what hardware support KVM & Xen, and vice versa.  I'd like to 
> know what hardware KVM & Xen users are actually using to run KVM & 
> Xen? Particularly with consideration for 3D gaming and current 
> generation hardware, BTW.
> If there is need for further clarification, I'll answer any queries 
> you might have.
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxx
> http://lists.xen.org/xen-users


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users