Distro kernel and 'virtualization server' vs. 'server that sometimes runs virtual instances' rant (was: Re: [Xen-devel] Re: [GIT PULL] Xen APIC hooks (with io_apic_ops))

[Luke S Crawford <lsc@xxxxxxxxx>]

Dan Magenheimer <dan.magenheimer@xxxxxxxxxx> writes:
> such as is needed for huge corporate data centers and "clouds".
> However, the majority of users (individuals and small businesses)
> will probably be most happy with their distro (and distro kernel)
> as dom0 since it is convenient and familiar.

Hey, I'm going to hijack this for a rant, if you don't mind.  I've 
pared back the cc list.  

Now, I'm just the janitor; I rightly belong on xen-users.  I lurk in -devel
because it's good to know what the developers are thinking.  I don't want
to come off like I think I'm smarter than anyone here; I don't.  But
I am a heavy user of Xen, and I don't think I'm an unusual user of Xen.

I've been selling VPSs using Xen since 2005.  After the 
marketing people convince the middle managers that virtualization is the 
way to go, someone like me has to actually bang on the thing with a spanner
or rub it with a greasy rag until it works.  

I also do contracting for some of those 'large corporate data centers' of 
which you speak.  (corporate data centers seem to be the worst in terms of 
operational efficiency.  do you know how many Linux installations I've seen 
where the customer pays a few hundred extra per box for integrated KVM over 
IP functionality rather than the much cheaper and more useful serial 
consoles?  Oy.  You expect me to tell you why your server crashed when 
you have no console logs of the backtrace?)

But I'm getting sidetracked.   My point is that small companies need good 
tools more than large corporations do.   The big guys can just keep 
throwing money at the problem until their stuff mostly works.  

In the Dom0, I want something as stable, minimal, standard and  
supported as possible (by supported, I really mean standard and widely
used.  The best 'support' is searching mailing lists such as this one.)

the last thing I want is all the cowboy hackery that goes into my favorite
desktop OS to be included in my Xen Dom0.   I moved to Ubuntu on my laptop
last year and I was amazed how easy it was.  everything just worked.  
making new hardware work was easier than windows.  

But do I want that on my Xen Dom0?  certainly not until you get that thing
working where I can reboot the Dom0 without killing everything.  

This is what I think is wrong about the default install of Xen;  it is setup 
so that you can run your desktop in the dom0, and spin up DomUs as needed.
It tries to be a virtualization server and a desktop at the same time,
and it gives up stability for this. 

If you've ever run a Xen host and have forgotten to change the default 
dom0-min-mem of 192MiB, you'd know most (especially x86_64) linux 
installations are not stable under load with that much memory.   Even 
if you set dom0-min-mem to a reasonable value, I've seen enough problems 
related to ballooning that I always disable it on all hosts.  (granted, 
most of those problems are on old RHEL installs)   Also, in all 
the environments I work in, both at the 'large corporate data center' 
and my own,  the configuration of the DomUs is fairly static.   Sure, it's 
kind of nice to be able to add resources without rebooting the DomU, but 
to give up stability for that is just crazy.  If your customer says "I 
want more X" it's fine to say "Ok, let me know when I can reboot you"  -  
it's not ok to crash.  

In my work, people mostly use the  'I take this Linux box, I set it up,
and I use it for three years' model.  They don't need any of the fancy 
'computing on demand'  -  they just want to move 16 of those crusty P3
servers that are killing their power bill and crashing due to bad hardware
twice a month on to a nice shiny new 8 core box with 32GiB ram and a
warranty.   I've seen lots of people who buy ec2 instances and do the
same thing; they leave it on all the time. (the basic ec2 instances are
particularly unsuited to this usage, but people do it anyhow.) 

I'm not going to say memory overcommit is never useful for anyone;
but I can say it is never useful for me.  32GiB registered ecc ddr2
is around $600.  That's not very many billable hours.  That's around
half the approximate cost of an unplanned reboot of one of my servers.
(I'm only counting money lost due to SLA and time to clean up;  if you
count loss to reputation, it gets even worse)  

My experience with memory overcommit has been that it makes your 
system either unstable, slow or both.  Now, I don't know if you could
theoretically make a zero cost memory overcommit system;  I'm just saying
that every attempt at overcommiting memory between virtual servers I have
seen ended in tears.   (heck, I've seen quite a lot of tearful endings
due to the memory overcommit linux itself does.)  This is why I ditched
FreeBSD jails and came to Xen in 2005.  

Right now, I'm using CentOS5 with the xen.org kernels, but it sure
would be nice if there was some pared down pre-built dom0 configuration 
available. (I personally give my Dom0 1024MiB out of 32GiB)  It could be
based on centos, or on ttylinux, or whatever.  just something standard, small,
and simple.  Make it good enough that people use it.  When I see a problem,
I want fifty other guys to have seen the problem first.  

I'm thinking about starting such a project myself once I get a few other 
things done.  If nothing else, I can distribute kickstart files of a minimal
dom0. Going forward, now that NetBSD 5 is out, perhaps I will switch back 
to NetBSD as my Dom0  (I switched from FreeBSD jails to NetBSD/Xen2 in 
2005.  I switched away for pae/x86_64 support.  I mean, no pae sucked, 
but the os was solid)  Unfortunately, that means I would have less 
'support' in the form of other people doing the same thing and talking 
about it in public.

RedHat is talking about doing it with KVM  -  see the Red Hat Enterprise 
Virtualization hypervisor  - they claim you will have a KVM 'dom0'  that 
uses only 64M ram- which seems funny to me, as my perception of KVM  has 
always been that it was optimized to run virtual instances as needed on 
a box that usually ran applications on the bare metal, like a desktop.  


-- 
Luke S. Crawford
http://prgmr.com/xen/  -     Hosting for the technically adept
http://nostarch.com/xen.htm  We don't assume you are stupid.  

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel