[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] Recommendations for Virtulization Hardware

  • To: xen-users@xxxxxxxxxxxxx
  • From: ShadesOfGrey <shades_of_grey@xxxxxxxxxxxxx>
  • Date: Sun, 23 Sep 2012 23:45:50 -0400
  • Delivery-date: Mon, 24 Sep 2012 03:46:35 +0000
  • Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=dk20050327; d=earthlink.net; b=ADTKgYyPE6CncvVf6gmsGoFEP6wNEXjgIU6tjQW9Sx+GX25d/xM/JsXQOfsagfNz; h=Received:Message-ID:Date:From:User-Agent:MIME-Version:To:Subject:References:In-Reply-To:Content-Type:X-ELNK-Trace:X-Originating-IP;
  • List-id: Xen user discussion <xen-users.lists.xen.org>

Sorry for the late response, I've had a lot to digest.

On 09/21/2012 01:02 PM, Casey DeLorme wrote:
Thanks for the CC Andrew,

I responded in private since I didn't want to spam the list.

Since we're on the subject of the latest hardware, my system was using a Z68 Motherboard Chipset and Core i7 SandyBridge CPU (2600).  I just sold it to a friend and upgraded to the latest.

Now running an IvyBridge 3770, Z77 Motherboard Chipset (ASRock Z77 Extreme 9), and similar components for the rest.  The Extreme 9 price is extremely high, I wouldn't recommend it unless you have the money, their Z77 Extreme 6 is priced at the same as the Z68 Extreme4 Gen3, and has all the same features plus more USB 3.0.  I would give that one a shake if you are looking for newest components.  The hardest part of switching from the Extreme4 Gen3 was remapping the USB ports to USB controllers by PCI BDF in Linux, because I haven't found an easy way to do that using the systems information yet.

Funnily enough, the Extreme9 was one of my two original motherboard candidates, along with the Gigabyte G1.Sniper 3.  Though I've expanded that list now to include all of ASRock's Z77 line of motherboards.  I might even consider Z77 motherboards from other manufacturers, if I can confirm they support VT-d.

I've also been contemplating going with a Q77 motherboard, just to avoid the hassles of determining if a given Z77 motherboard actually has VT-d support.  The downside to the Q77 motherboards I have looked at is that they typically lack the integrated extras (e.g. additional SATA/USB controllers) that you find on Z77 motherboards.  This makes them less attractive for PCI passthrough. 

Furthermore, a respondent on the Ars Technica threads suggested that an LGA2011 could be worth a gander.  Now, when I started investigating hardware, I had dismissed this option, mostly because they were often very expensive (incl. CPU). More aggravating, the X79 chipset used on desktop LGA2011 motherboards is not officially sanctioned for VT-d.  Given it's placement as a desktop workstation chipset, the omission of VT-d just pissed me off.  I didn't even bother investigating the possibility that X79 motherboard 'unofficially' supported VT-d.  So, in order to get VT-d on LGA2011, you had to level up to a server motherboard.  Unfortunately, most of the options that had the mix of features I wanted, ended up being ridiculously expensive, particularly because those motherboards were designed with dual processors in mind.

Fortunately, there have been some changes since then.  For example, I found this Gigabyte motherboard.  It's based on the C606 server chipset but packaged as a desktop motherboard.  It is sanctioned by Intel for VT-d and has more RAM capacity.  As an added bonus, it includes an integrated SAS controller supplying eight SAS 3GB/s ports.  I'm pretty sure it's fake RAID though.  No big deal, I'll probably end up using Btrfs, mdadm, or ZFS on that controller anyway.  As things stand now, this motherboard would pretty much cost the same as the Extreme9.  Going the LGA 2011 route, however, would require I start with two discrete GPUs instead of one.  Maybe something like this, combined with something like this or this.

All of these considerations have seriously complicated my decision-making process.  I still haven't managed to pare down my hardware choices to fit the $2000 budget I have imposed on myself.  And those calculations don't even include one discrete GPU!  So far, most (not all, e.g. Extreme6) of the suggestions I get haven't helped to reduce costs.  Based on your success, I'm sorely tempted to dip into my reserve fund for building a second, more modest box:  My contingency plan if my experiments in virtualization failed to produce satisfactory results.
As soon as I'm done putting together the details, I'll append the new hardware information to the wiki.

As for changes, the new Z77 chipset has onboard USB 3.0 in exchange for some of the formerly USB 2.0 ports.  In my performance demo video I showed that USB 2.0 speeds are sub-par, but even SUB 2.0 devices can hit higher rates when connected to a USB 3.0 port.  As far as the IvyBridge CPU, onboard graphics are noticeably better in Dom0, and file decompression in my Windows HVM appears to be faster.  I have not run any benchmark tests yet.

I was kind of surprised by that.  I can understand that there would be some overhead due to the virtualized environment, but I wondered what would cause such a degradation of performance?  I mean I know USB is pretty CPU dependent.  However, with a Core i7, I figure there would be enough CPU cycles to spare that only multiple asynchronous connections would cause that kind of performance hit.

I compiled the new Xen 4.2 stable, as tagged in the new 4.2 testing branch, but I may try the older revision of Xen in my guide.  I am getting an error on the first-launch attempt of any virtual machine with passed devices.  When I try again it works, but it's 5 lines worth of errors which makes the following success seem less joyful.

I tried kernel 3.5.4, and it appears to be suffering the same passthrough bugs as 3.5.2, so I would avoid 3.5+ still.  I ended up with Kernel 3.4.11, which is working.

I'll keep all that in mind, though I don't have a problem compiling sever kernel images to play with...  BTW, would there be any significant problems, that you know of, to compiling and trying several combinations of Linux and Xen during my testing phase?  If I went with significantly different hardware, I imagine I might need different combinations of both.  But, since I haven't actually gotten to the testing phase, I don't know what easily avoidable pitfalls might be in my path.
One good bit of news was I kept my SSD, so I was able to just recreate the HVM configuration and launch Windows without problems.  Only issues were new drivers, and Windows asking to reactivate due to HW changes.

Hope this information helps,


It's all been very helpful.  I am in your debt!  I'm sure others will be too, once they make use of your guide.

BTW, another thing that I've been wondering about.  Should I treat Dom0 as an  administrative domain and create a separate DomU for my Linux desktop?  Or is it safe to reduce the overhead of having a Linux DomU and just use Dom0?  I know this is probably more of a stylistic question; after all, you were using your Dom0 for development purposes... but that's significantly different from a general purpose desktop environment.

Xen-users mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.