Xen project Mailing List

On Mon, Sep 23, 2013 at 1:09 PM, Gordan Bobic <gordan@xxxxxxxxxx> wrote:

On Sun, 22 Sep 2013 22:20:14 -0300, Zir Blazer <zir_blazer@xxxxxxxxxxx> wrote:

If you want something more recent that that, GTX680 can be
modified into a Quadro K5000 or half of a Grid K2, but this
requires a bit of soldering.

The upshot of going for a cheap Quadro or a Quadrified
Nvidia card is that rebooting VMs doesn't cause problems
which ATI cards are widely reported to suffer from.

I know about the possibility of hard modding GeForces into Quadros,
but purchasing a new card for that purpose is totally out of my
budget. So I'm limited to whatever I can do with my current ones.
My understanding is that ATI Video Cards like the ones I already own
were easier to passthrough than non-modded GeForces. No idea about the
VM rebooting issues or what generations it applies to.

As far as I can tell (including from personal experience up until
a couple of months ago when I finally gave up on any kind of a
solution based on ATI cards) - all ATI cards suffer the problems
mentioned - and will continue to do so until ATI do something
about it in their drivers (e.g. make sure that the driver resets
the card back to a clean slate before unloading). Given ATI's
track record of driver Quality, I doubt it will ever happen, even
though I don't imagine it is a particularly difficult thing to fix.

You should be aware that Intel have changed VID control on Haswell
and later CPUs, so undervolt tuning based on clock multiplier
(e.g. using something like RMClock on Windows or PHC on Linux)
no longer works. If you want to use this functionality, you would
be better off picking a pre-Haswell CPU. I have this problem with
my Chromebook Pixel, which runs uncomfortably (if you keep it on
your lap) hot, even when mostly idle.

For mobile computers, pre-Haswell Processors VID can't be touched by
Software, either. I purchased a year ago a Notebook with a Sandy
Bridge and after googling around looking for info on that, there are
no tools to undervolt neither mobile Sandy Bridge, Ivy Bridge or
Haswell. It has to be done via BIOS only, which no Notebook BIOS will
let you do. On Desktop platforms I think there were a few tools that
did worked to undervolt via Software, but I'm not sure on Haswell
generation. People that appreciates undervolt are just a small niche.

I am not sure about Sandy Bridge and Ivy Bridge, but everything up to
that has controllable VID (within the hard-limited range). I haven't
tried it on anything between Core2 and Haswell since I change my
laptops very infrequently. I do know, however, that a number of Atom
based laptops and industrial motherboards which do have
user-controllable VID - ignore it (Dell Mini, as well as Advancetech
motherboards).

Anyway, my point is that if you are concerned to any substantial
degree about power usage and heat (unlikely give your GPU power
draw), you may be better off with a previous generation CPU that
actually has working functionality for undervolting. Note that
in every case I have seen, once you adjust the voltage, the CPU
runs at that voltage under all conditions. That means that the
voltage won't drop below this when the system is idle and clocks
down. This results in the idle power usage going up rather than
down. It reduces the average power consumption of your system is
always running at 100% load, but in a typical case where it is
idle 95% of the time, the power consumption will actually go up.

So if you are after a power reduction, VID adjusting is the way
to do it.

I have more or less given up on buying any non-Asus motherboards.
Switching to an EVGA SR-2 after completely trouble-free 5 years
with my Asus Maximus Extreme has really shown me just how good
Asus are compared to other manufacturers.

All things being equal, if I was doing my rig for similar purposes
as you (two virtual gaming VMs with VGA passthrough, one for me,
one for the wife), I would probably get something like an Asus
Sabertooth or Crosshair motherboard with an 8-Core AMD CPU. They
are reasonably priced, support ECC, and seem to have worked
quite well for VGA passthrough for may people on the list.

I find hard to believe that someone here would recommend an ASUS
Motherboard considering that they were know to be the less reliable
when it came to IOMMU support, which is top priority for my purpose.

Less reliable than what? My experience is that if you want a
consumer grade motherboard with decent tweakability, Asus are
the only choice even remotely worth considering.

They're fine from the Hardware side, but these days where most of the
important components are part of either the Processor or Chipset you
can't really go wrong unless its a very, very low end model. Most
expensive Motherboards are full of worthless features I don't need
(And even makes virtualization harder, like that PCIe bridge). Besides
that, what annoys me the most is the ASUS official "we don't support
Linux, won't going to spend dev resources on fixing the BIOS so it
follows standard specifications instead of expecting Windows to
override settings and bad ACPI tables to make it work anyways". That's
the very reason why I'm avoiding them at all cost currently, and
suggesting other people to do the same. Oh, and every now and then
ASUS has several low end models that are to be avoided, too. Maximus
Extreme, Sabertooth or Crosshair aren't budget Motherboards, most
manufacturers don't screw up badly if you're spending around 200 U$D
or more on a Motherboard - yet ASUS does with IOMMU support.

Who doesn't?

EVGA sure do - their flagship $600 SR-2 has all of it's PCIe slots
behind NF200 bridges.

If you want something that "just works", your only sane option is
one of a handful HP or Dell workstations (one each with any sort
of certification from hypervisor vendors) and pay the going rate
for them (which is exorbitant, but worth every penny if your time
isn't worthless). And you'll just have to live with the lack of
any kind of tweakability.

Supermicro is quite expensive, but also a very reputable Server based
brand. They're know for being 24/7 rock solid workhouses, reason why
I'm spending soo much on one.

And you have it on good authority (i.e. better than the vendor's
without-prejudice pre-sales insunuation) that that particular
motherboard has all of the required features AND that they
actually tested it to ensure that virtualization and IOMMU
work fine?

Avoid anything featuring Nvidia NF200 PCIe bridges at all cost.
That way lies pain and suffering. I'm in the process of working
on two patches for Xen just to make things workable on my
EVGA SR-2 (which has ALL if it's PCIe slots behind NF200
bridges).

Supermicro X10SAT got the three PCIe 1x slots and 4 USB3 ports behind
a PLX 8606 chip. Not sure if that one is supported or not, but I don't
plan at the moment to fill those slots, neither.

Interestingly, my GTX690 seems to have 3 (!) PLX PCIe bridges
on it. I haven't modified it yet, so I cannot say how well
they work, or what odd interractions might arise between those
and the NF200.

I flat out refuse to run anything without ECC memory these days.
This is a major reason why I don't consider Core i chips an
option.

There are several Xeons models that do support ECC Memory and are
around the same price than a Core iX of similar specs. I think you
also need a Server Motherboard based on the C222, C224 or C226
Chipsets too, another one of Intel artificial limits to force you to
spend on a more expensive platform. But depending on your budget and
what you were going to buy, adding ECC support on an Intel platform
could be relatively cheap if you pick proper parts.

Or you could just pick more or less anything made by AMD, including
desktop grade parts with a better price/performance.

Did you stability test them? GPUs come pre-overclocked to within
1% of death from the factory.

The default for my pair of Video Cards was 850 MHz @ 1.1V for GPU,
which is the same than reference Radeons 5770, so these are not
factory overclocked.

Given there is little or no margin for error in them, I consider the
_reference_ model to be pre-overclocked.

They were capable of being 100% stable for
BitCoin mining at 850 MHz @ 1V which was what I had them at, along
with the VRAM @ 300 MHz.

How are you certain that they weren't making erroneous calculations?
Were you crunching everything twice and cross-comparing just to make
sure? Or are you naively assuming that something is stable just
because doesn't outright crash?

Did you stability test the hardware at those settings using available
tools (e.g. OCCT, FurMark) for 24 continuous hours per tool to
confirm error free operation?

I also wouldn't consider putting any non-expendable data on
anything but ZFS - silent corruption happens far more often
than most people imagine, especially on consumer grade desktop
disks.

I would need to purchase a new, 2 TB+ HD to allow for ZFS redundancy
on single disk systems. I have around 800 GB of data on this one. At
the moment, can't.

I guess you don't consider your data that valuable to you. Each to
his own.

I didn't think firing up two D3D games at the same time was even
possible - or are you talking about just minimizing one?

You can have two D3D games open at the same time even on WXP. However,
as there is no proper GPU priority (Like CPU), you literally have no
control of where you want the video performance at - League of Legends
FPS are usually on the 15-20 FPS mark, and if I have Eve Online on the
screen like on the screenshot its around 7-10. Also, remember that
both games are running simultaneously powered by my integrated Radeon
4200 that is 5 years or so old, even the Haswell IGP should be 3-5
times more powerful than this.

http://imageshack.us/a/img24/4909/3uck.png

Right, running windowed.

Additionally, sometimes you MAY want to use an older Driver version
for whatever reason. For example, for my Radeon 4200 I don't want to
use any recent Catalyst Drivers versions because at some point they
introduced a 2D acceleration bug that locks my machine. Check this
Thread:

http://forums.anandtech.com/showthread.php?t=2332798

That's ATI drivers for you.

For me as a Desktop power user, the power of Xen is all about
workarounding against OS and Software limitations that exist because
at the time no one considered you could try doing stuff in some
specific fashion, usually by running assymetric setups like mine (IGP
+ 2 discrete GPUs), multimonitor, etc. It is to make Frankenstein
builds works as intended when the OS doesn't know how to do it. If my
current machine had IOMMU support, I believe Xen could have helped me
by letting me to have a WXP VM with the Radeon 4200 with the stable
Drivers, then having a totally separated VM with the pair of Radeons
5770 with newer Drivers to mine, everything working conflictless
assuming a perfect scenario.

That's a great theory. Unfortunately, it doesn't survive contact with
reality in most cases. On a lot of hardware you run into firmware,
hardware and driver bugs that require all kinds of hacky proprietary
patches, similar to the ones I'm having to write for running a
similar stack on my hardware.

My understanding was that cost effectiveness of electricity +

ammortized

hardware cost of hardware was nowdays such that it makes mining
cost-ineffective. But whatever floats your boat.

Where I live at, is STILL profitable, but barely so. Not enough to
bother to be honest. I'm not doing it from some months ago due to the
Driver issue described above. But originally it was one of the
possible uses.

I thought most people who still mine do so on proprietary purpose
built silicon that is now available, from what I hear.

I think your list of requirements is already spiraling out of
control, and if you really are mostly a Windows user, you may
find the amount of effort required to achieve a system this
complex is not worth the benefits. As I said, I'm finding myself
having to write patches to get my system working passably well,
and the complexity level on that is considerably lower than
what you are proposing.

The bad part of this is that while I do have the free time to waste
making something out of Xen work, I'm NOT a programmer. I barely can
do anything above a Hello World. This means that whatever I want to
do, will be limited to what Xen can do either by itself or with
patches, along with reading tons of guides, etc. If there are things
that Xen currently can't do, they will be dropped out of my list of
requeriments until the next Xen version, for as long as it is possible
to do via Software modifications.
I may be dreaming a bit. My proposed requeriments are for my ultimate
production setup, what can be archivable will obviously be less than
that. However, for what you're telling me what can be currently done
isn't really that far from that.

ASSUMING you manage to get some hardware within your budget that is
bug free for this kind of operation. That's a big assumption, and
one that is untestable until you acquire some and test it yourself,
by which point you have already spent your budget so you're stuck
with it.

This is why I'm saying it is of paramount importance that you get
yourself a motherboard that other people here have extensively
tested with VGA passthrough and large memory allocations and
verified it to work in a completely trouble-free way.

This was only announced as a preview feature a few days ago.
I wouldn't count on it being as production-ready as you might hope.

It is not new. I recall hearing about nVidia Grid and the GPU
Hypervisor around 6 months ago. I think I readed somewhere that they
already suffered more than a year of delays. Considering the market
that it is aimed to and how much the nVidia GRIDs cost, I would be
surprise if they are still far from production ready.

That was for VMware, not Xen, AFAIK.

You might be interested in a recent announcement I saw on the

xen-devel mailing list, around work on a graphics virtualization
solution :

http://lists.xenproject.org/archives/html/xen-devel/2013-09/msg00681.html

Excellent to hear. Wasn't aware of that one, would give me yet another
possible choice.

Once it is production stable and ready for general public non-developer
consumption.

Gordan

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users

Re: [Xen-users] Getting ready for a computer build for IOMMU virtualization, need some input regarding sharing multiple GPUs among VMs