Re: [Xen-users] Getting ready for a computer build for IOMMU virtualization, need some input regarding sharing multiple GPUs among VMs

[Gordan Bobic <gordan@xxxxxxxxxx>]

On Sun, 22 Sep 2013 22:20:14 -0300, Zir Blazer <zir_blazer@xxxxxxxxxxx> 
wrote:
> > If you want something more recent that that, GTX680 can be
> > modified into a Quadro K5000 or half of a Grid K2, but this
> > requires a bit of soldering.
> >
> > The upshot of going for a cheap Quadro or a Quadrified
> > Nvidia card is that rebooting VMs doesn't cause problems
> > which ATI cards are widely reported to suffer from.
> I know about the possibility of hard modding GeForces into Quadros,
> but purchasing a new card for that purpose is totally out of my
> budget. So I'm limited to whatever I can do with my current ones.
> My understanding is that ATI Video Cards like the ones I already own
> were easier to passthrough than non-modded GeForces. No idea about 
> the
> VM rebooting issues or what generations it applies to.
As far as I can tell (including from personal experience up until
a couple of months ago when I finally gave up on any kind of a
solution based on ATI cards) - all ATI cards suffer the problems
mentioned - and will continue to do so until ATI do something
about it in their drivers (e.g. make sure that the driver resets
the card back to a clean slate before unloading). Given ATI's
track record of driver Quality, I doubt it will ever happen, even
though I don't imagine it is a particularly difficult thing to fix.

> > You should be aware that Intel have changed VID control on Haswell
> > and later CPUs, so undervolt tuning based on clock multiplier
> > (e.g. using something like RMClock on Windows or PHC on Linux)
> > no longer works. If you want to use this functionality, you would
> > be better off picking a pre-Haswell CPU. I have this problem with
> > my Chromebook Pixel, which runs uncomfortably (if you keep it on
> > your lap) hot, even when mostly idle.
> For mobile computers, pre-Haswell Processors VID can't be touched by
> Software, either. I purchased a year ago a Notebook with a Sandy
> Bridge and after googling around looking for info on that, there are
> no tools to undervolt neither mobile Sandy Bridge, Ivy Bridge or
> Haswell. It has to be done via BIOS only, which no Notebook BIOS will
> let you do. On Desktop platforms I think there were a few tools that
> did worked to undervolt via Software, but I'm not sure on Haswell
> generation. People that appreciates undervolt are just a small niche.
I am not sure about Sandy Bridge and Ivy Bridge, but everything up to
that has controllable VID (within the hard-limited range). I haven't
tried it on anything between Core2 and Haswell since I change my
laptops very infrequently. I do know, however, that a number of Atom
based laptops and industrial motherboards which do have
user-controllable VID - ignore it (Dell Mini, as well as Advancetech
motherboards).

Anyway, my point is that if you are concerned to any substantial
degree about power usage and heat (unlikely give your GPU power
draw), you may be better off with a previous generation CPU that
actually has working functionality for undervolting. Note that
in every case I have seen, once you adjust the voltage, the CPU
runs at that voltage under all conditions. That means that the
voltage won't drop below this when the system is idle and clocks
down. This results in the idle power usage going up rather than
down. It reduces the average power consumption of your system is
always running at 100% load, but in a typical case where it is
idle 95% of the time, the power consumption will actually go up.

So if you are after a power reduction, VID adjusting is the way
to do it.

> > I have more or less given up on buying any non-Asus motherboards.
> > Switching to an EVGA SR-2 after completely trouble-free 5 years
> > with my Asus Maximus Extreme has really shown me just how good
> > Asus are compared to other manufacturers.
> >
> > All things being equal, if I was doing my rig for similar purposes
> > as you (two virtual gaming VMs with VGA passthrough, one for me,
> > one for the wife), I would probably get something like an Asus
> > Sabertooth or Crosshair motherboard with an 8-Core AMD CPU. They
> > are reasonably priced, support ECC, and seem to have worked
> > quite well for VGA passthrough for may people on the list.
> I find hard to believe that someone here would recommend an ASUS
> Motherboard considering that they were know to be the less reliable
> when it came to IOMMU support, which is top priority for my purpose.
Less reliable than what? My experience is that if you want a
consumer grade motherboard with decent tweakability, Asus are
the only choice even remotely worth considering.

> They're fine from the Hardware side, but these days where most of the
> important components are part of either the Processor or Chipset you
> can't really go wrong unless its a very, very low end model. Most
> expensive Motherboards are full of worthless features I don't need
> (And even makes virtualization harder, like that PCIe bridge). 
> Besides
> that, what annoys me the most is the ASUS official "we don't support
> Linux, won't going to spend dev resources on fixing the BIOS so it
> follows standard specifications instead of expecting Windows to
> override settings and bad ACPI tables to make it work anyways". 
> That's
> the very reason why I'm avoiding them at all cost currently, and
> suggesting other people to do the same. Oh, and every now and then
> ASUS has several low end models that are to be avoided, too. Maximus
> Extreme, Sabertooth or Crosshair aren't budget Motherboards, most
> manufacturers don't screw up badly if you're spending around 200 U$D
> or more on a Motherboard - yet ASUS does with IOMMU support.
Who doesn't?

EVGA sure do - their flagship $600 SR-2 has all of it's PCIe slots
behind NF200 bridges.

If you want something that "just works", your only sane option is
one of a handful HP or Dell workstations (one each with any sort
of certification from hypervisor vendors) and pay the going rate
for them (which is exorbitant, but worth every penny if your time
isn't worthless). And you'll just have to live with the lack of
any kind of tweakability.

> Supermicro is quite expensive, but also a very reputable Server based
> brand. They're know for being 24/7 rock solid workhouses, reason why
> I'm spending soo much on one.
And you have it on good authority (i.e. better than the vendor's
without-prejudice pre-sales insunuation) that that particular
motherboard has all of the required features AND that they
actually tested it to ensure that virtualization and IOMMU
work fine?

> > Avoid anything featuring Nvidia NF200 PCIe bridges at all cost.
> > That way lies pain and suffering. I'm in the process of working
> > on two patches for Xen just to make things workable on my
> > EVGA SR-2 (which has ALL if it's PCIe slots behind NF200
> > bridges).
> Supermicro X10SAT got the three PCIe 1x slots and 4 USB3 ports behind
> a PLX 8606 chip. Not sure if that one is supported or not, but I 
> don't
> plan at the moment to fill those slots, neither.
Interestingly, my GTX690 seems to have 3 (!) PLX PCIe bridges
on it. I haven't modified it yet, so I cannot say how well
they work, or what odd interractions might arise between those
and the NF200.

> > I flat out refuse to run anything without ECC memory these days.
> > This is a major reason why I don't consider Core i chips an
> > option.
> There are several Xeons models that do support ECC Memory and are
> around the same price than a Core iX of similar specs. I think you
> also need a Server Motherboard based on the C222, C224 or C226
> Chipsets too, another one of Intel artificial limits to force you to
> spend on a more expensive platform. But depending on your budget and
> what you were going to buy, adding ECC support on an Intel platform
> could be relatively cheap if you pick proper parts.
Or you could just pick more or less anything made by AMD, including
desktop grade parts with a better price/performance.

> > Did you stability test them? GPUs come pre-overclocked to within
> > 1% of death from the factory.
> The default for my pair of Video Cards was 850 MHz @ 1.1V for GPU,
> which is the same than reference Radeons 5770, so these are not
> factory overclocked.
Given there is little or no margin for error in them, I consider the
_reference_ model to be pre-overclocked.

> They were capable of being 100% stable for
> BitCoin mining at 850 MHz @ 1V which was what I had them at, along
> with the VRAM @ 300 MHz.
How are you certain that they weren't making erroneous calculations?
Were you crunching everything twice and cross-comparing just to make
sure? Or are you naively assuming that something is stable just
because doesn't outright crash?

Did you stability test the hardware at those settings using available
tools (e.g. OCCT, FurMark) for 24 continuous hours per tool to
confirm error free operation?

> > I also wouldn't consider putting any non-expendable data on
> > anything but ZFS - silent corruption happens far more often
> > than most people imagine, especially on consumer grade desktop
> > disks.
> I would need to purchase a new, 2 TB+ HD to allow for ZFS redundancy
> on single disk systems. I have around 800 GB of data on this one. At
> the moment, can't.
I guess you don't consider your data that valuable to you. Each to
his own.

> > I didn't think firing up two D3D games at the same time was even
> > possible - or are you talking about just minimizing one?
> You can have two D3D games open at the same time even on WXP. 
> However,
> as there is no proper GPU priority (Like CPU), you literally have no
> control of where you want the video performance at - League of 
> Legends
> FPS are usually on the 15-20 FPS mark, and if I have Eve Online on 
> the
> screen like on the screenshot its around 7-10. Also, remember that
> both games are running simultaneously powered by my integrated Radeon
> 4200 that is 5 years or so old, even the Haswell IGP should be 3-5
> times more powerful than this.
>
> http://imageshack.us/a/img24/4909/3uck.png
Right, running windowed.

> Additionally, sometimes you MAY want to use an older Driver version
> for whatever reason. For example, for my Radeon 4200 I don't want to
> use any recent Catalyst Drivers versions because at some point they
> introduced a 2D acceleration bug that locks my machine. Check this
> Thread:
>
> http://forums.anandtech.com/showthread.php?t=2332798
That's ATI drivers for you.

> For me as a Desktop power user, the power of Xen is all about
> workarounding against OS and Software limitations that exist because
> at the time no one considered you could try doing stuff in some
> specific fashion, usually by running assymetric setups like mine (IGP
> + 2 discrete GPUs), multimonitor, etc. It is to make Frankenstein
> builds works as intended when the OS doesn't know how to do it. If my
> current machine had IOMMU support, I believe Xen could have helped me
> by letting me to have a WXP VM with the Radeon 4200 with the stable
> Drivers, then having a totally separated VM with the pair of Radeons
> 5770 with newer Drivers to mine, everything working conflictless
> assuming a perfect scenario.
That's a great theory. Unfortunately, it doesn't survive contact with
reality in most cases. On a lot of hardware you run into firmware,
hardware and driver bugs that require all kinds of hacky proprietary
patches, similar to the ones I'm having to write for running a
similar stack on my hardware.

> > My understanding was that cost effectiveness of electricity +
> ammortized
> > hardware cost of hardware was nowdays such that it makes mining
> > cost-ineffective. But whatever floats your boat.
> Where I live at, is STILL profitable, but barely so. Not enough to
> bother to be honest. I'm not doing it from some months ago due to the
> Driver issue described above. But originally it was one of the
> possible uses.
I thought most people who still mine do so on proprietary purpose
built silicon that is now available, from what I hear.

> > I think your list of requirements is already spiraling out of
> > control, and if you really are mostly a Windows user, you may
> > find the amount of effort required to achieve a system this
> > complex is not worth the benefits. As I said, I'm finding myself
> > having to write patches to get my system working passably well,
> > and the complexity level on that is considerably lower than
> > what you are proposing.
> The bad part of this is that while I do have the free time to waste
> making something out of Xen work, I'm NOT a programmer. I barely can
> do anything above a Hello World. This means that whatever I want to
> do, will be limited to what Xen can do either by itself or with
> patches, along with reading tons of guides, etc. If there are things
> that Xen currently can't do, they will be dropped out of my list of
> requeriments until the next Xen version, for as long as it is 
> possible
> to do via Software modifications.
> I may be dreaming a bit. My proposed requeriments are for my ultimate
> production setup, what can be archivable will obviously be less than
> that. However, for what you're telling me what can be currently done
> isn't really that far from that.
ASSUMING you manage to get some hardware within your budget that is
bug free for this kind of operation. That's a big assumption, and
one that is untestable until you acquire some and test it yourself,
by which point you have already spent your budget so you're stuck
with it.

This is why I'm saying it is of paramount importance that you get
yourself a motherboard that other people here have extensively
tested with VGA passthrough and large memory allocations and
verified it to work in a completely trouble-free way.

> > This was only announced as a preview feature a few days ago.
> > I wouldn't count on it being as production-ready as you might hope.
> It is not new. I recall hearing about nVidia Grid and the GPU
> Hypervisor around 6 months ago. I think I readed somewhere that they
> already suffered more than a year of delays. Considering the market
> that it is aimed to and how much the nVidia GRIDs cost, I would be
> surprise if they are still far from production ready.
That was for VMware, not Xen, AFAIK.

> > You might be interested in a recent announcement I saw on the
> xen-devel mailing list, around work on a graphics virtualization
> solution :
>
> http://lists.xenproject.org/archives/html/xen-devel/2013-09/msg00681.html
>
> Excellent to hear. Wasn't aware of that one, would give me yet 
> another
> possible choice.
Once it is production stable and ready for general public non-developer
consumption.

Gordan

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users