Re: [Xen-devel] XEN - networking and performance

["D. Duckworth" <donduq@xxxxxxxxxxx>]

I've removed xen-users, the subject seems appropriate for xen-devel now.

> > I tried to use xl with xen4 for a while but due to bugs and missing
> > features I had to go back to xm and xend. This is where the fun
> > begins. In the past I used xend with network-bridge and for some
> > strange reason (voodoo probably) I blindly accepted that script in
> > the past and blamed myself for not appreciating it. But let's be
> > blunt and honest: the scripts, in particular the script that
> > *modifies dom0 networking during xend startup* is the biggest piece
> > of sh!# idea I have ever seen in Xen. It creates bridges, takes
> > eth0 down, tortures dom0 with occult ip addr/route, brctl, sysctl
> > and iptables awk/sed manipulations and then it has you looking at
> > your screen yearning for the moment that ping timeouts become ping
> > replies, telling you that your box is reachable again. This script
> > is a malevolent demon from the sewers of Norman the Golgothan and
> > the worst part is that network-bridge is also still the
> 
> <laughs>
> > recommended default!
> 
> Can you point me to where it mentions that please?

Sorry for the word flood & glad you had to laugh at my cynicism, I had
wanted to use my Norman comparison at least once somewhere and this was
the perfect opportunity. I'll explain the subtlety of the above problem.

So in late 2008 installed xen, then went through xend-config.sxp a
couple of times and came across bridges as the documented feature and
then two non-descriptive oneliners of alternatives. No need for NAT in
my situation but I skipped routing due to a lack of understanding on my
side. Heh what can I say, "know thine limitations?" I did search for
info about the routing option and it did not make sense to me at the
time. I think there was no such thing on the wiki back then either. So
i shelved the idea for later consideration and a few months later there
was a wiki update that did vaguely mention routing. But I already had a
working bridge setup, a tentative one due to the mysterious voodoo
happening somewhere on the machine. I probably have made this point
enough now but if it happens to me it probably happens to others too. I
had no clue why arp was being disabled, why the mac got
fe:ff:ff:ff:ff:ff. I do now thanks to other puzzled people who emailed
their questions and got them answered.

> We realized that the networking setup is quite complex and would be
> best left in the hands of the admins. The problem is that..
> > 
> > It surely seems so. Xen's /etc/xen/scripts (another design fail, why
> > not /usr?) and udev scripts are confusing ad-hoc bloatware routines
> > and are not transparent at all. With the current xen4 I saw the
> > premature advice to more or less 'prepare for migration from xm to
> > xl'. Yet, xl supports less and is conflicting: there is no vifname,
> > no 'xl new/delete', no more python, no relocation and suddenly there
> > is a conflict between 'xm start domain' versus 'xl start
> > /etc/xen/domain'.
> > 
> > So new features emerge, adding to the confusion of the end user,
> > while old problems are not being fixed properly. I wonder why,
> > especially because it does not seem that xm and xend are the broken
> > parts that need to be replaced by an unstable interface.
> > 
> > What needs attention first and foremost are two things, first of
> > which is real and wise effort into one simple, minimal script that
> > just handles the minimum in a transparent way e.g. control the
> > hypervisor, manage vms, manage the backend. Of course networking can
> > be done on domain start too, but this has happen in an entirely
> > different way from what it does and how it does it. This is so
> > important because it gives more control to the user that runs Xen.
> > It's also a good moment to build in proper and mature support for
> > IPv6.
> > 
> > Secondly, the website and documentation should be cleaned up and
> > revised where appropriate. The current situation is a mess that has
> > a much too steep and incompatible learning curve right now - for
> > example, a bridge should just not be named eth0 and a physical
> > device should not be renamed at all. It's fundamentally wrong,
> > stupid, mad as hell and a PR failure for Xen to do it this way out
> > of the box. No matter how often and detailed it has been documented
> > on the website. 
> 
> .. the documentation and setup is sometimes quite hard. BTW, we are
> going to do on Oct 26th a Documentation Day to clean up some of this
> mess. Would you be intereted in helping along - perhaps in the
> networking Wiki?

Interested yes, perhaps. Could you please send me the details? As for
the notion that documentation and setup can be hard, that's exactly why
I, like you, suggest to move away from OS configuration as much as
possible -- obviously the hypervisor and kernel integration are the
primary domain. Daily commits by committed people seem to support this
theory.

Moving away from OS integration narrows the scope of the Xen project
which means less specific documentation needs to be maintained while
making it easier for everyone to invoke their own methods. OS
integration, even udev scripts, should move to the distribution's
developers and they can leave the rest up to the end user. Supporting
everything from within Xen is like trying to be everybody's friend. An
impossible task, just ask Ghandi, he tried it and got shot.

So take vif-nat. It does stuff with dhcp with all kinds of assumptions.
I don't even use ISC's dhcpd, I use dnsmasq. And this script does stuff
with iptables. I use shorewall and shreeked when I noticed a sudden
change in my tables. A fast but not fun two or three hours later I
found what did do it... after having searched through /usr and the domU.
So I found this whole script an abomination. Here's one other example
from the function dhcp_arg_add_entry (!)

# handle Red Hat, SUSE, and Debian styles, with or without quotes

Just...don't do this. These might be three big distributions but I
wonder whether they outnumber all users of all other distributions. And
the BSD camp doesn't seem to be really fond of Xen at the moment either.
In fact to me it seems like Xen's arrows all focus on Linux while a
bigger arrow marked KVM points towards the same direction. But I'm not
deeply involved with any project's development and never have been, so I
could be totally wrong. I don't even know Xen's project roadmap, future
outlook et cetera. I'll even admit I'm just a boob!

Still, it seems only logical to move to a more generic way of
doing things while OS/distribution devs and admins take care of their
respective domains. (Is this even remotely making sense to anyone?)

> > 
> > I propose something like the following for xen networking:
> > 
> > * Xen will not manipulate non-xen devices or a firewall under any
> >   circumstance, it might only add or substract routes and/or rules
> > from the routing tables,
> 
> Uh, what is 'non-xen' devices? Like bridges?

Yes. If I would use a Xen created bridge it has STP disabled, more
occult features, It's evil. So Xen must not create or touch those and
also not touch the routing/iptables with automated scripts. It is
clearly a much better idea to create transparent per-domain networking
options. Can be done in different ways like with pre/post-up/down
networking scripts that the dev/admin has to write. It still allows for
Xen to supply examples and suggest sane defaults. This time
transparently, making it all easier to maintain for everyone because
more modular. There can be a syntax check that spawns a warning if a
vif is configured without configured settings/scripts etc...

> 
> > * Allow for networking configuration per domU. For example let
> >   networking per device be nat, routed, bridged or custom, where
> >   all name the interface and bring it up; nat only adds the ip to
> > the routing table; routed could be an array of routes and rules
> > that need to be added or subtracted from various routing tables and
> > it might support proxyarp; bridged turns off arp, sets the mac on
> > the vif and then adds the interface to a bridge that should already
> > be created by the user; and custom is a custom set of unmanaged
> > commands after creating and destroying a domain.
> 
> You lost me. <sigh> I am using a bridge configuration and just
> do:
> 
> auto lo
> iface lo inet loopback
> 
> auto switch
> iface switch inet static
>       address 192.168.101.16
>       netmask 255.255.255.0
>       gateway 192.168.101.1
>       bridge_ports eth2
> 
> And just use that 'bridge=switch' in all my configuration. And that
> seems to work just fine - wouldn't that be best way of providing
> the first network setup to users? I would think the majority of folks
> do something akin to this?

Well I said a lot of things but not that it's broken. :) Not that I
couldn't, though. The above stuff works in Debian, but Arch Linux has no
ifup, neither does bsd. So those maintainers have to port Xen's code
with patches making it time expensive (and painfully dull). Instead the
whole routine that removes interfaces, adds new bridges, sets iptables,
etc. should simply be deleted. Not Xen's business; makes grown men cry.
How the majority of users handles it is not really relevant imho
because they are also bound to what Xen supplies. A lot of good people
see no obvious choice and a lot of good people have no clue about STP.
Consequently you'll find a lot of the same help requests on various
forums and lists when searching for clues. 

I tried to make sense of the minimum that the xen scripts do (with
xenstore, vif creation, device and udev interaction etc.) but the jungle
of 118 functions (grep '()' /etc/xen/scripts/* | wc -l, take or leave a
few) was not helping. I'd appreciate it if someone would write me a
detailed step by step hierarchy that explains what happens right in a
Xen4.2 dom0 from when a domain is created until it's running, what might
happen in between, and what happens from a domain's shutdown or crash to
domain deletion.

> > So, with that off my chest and the second line of my network-bridge
> > being the words "exit 0" Xen lets my dom0 configuration alone
> > like it is supposed to do. While KVM is becoming a 'next cool
> > thing' for many people I would still prefer a separate hypervisor
> > so now the fat just has to be removed from Xen.
> > 
> 
> I am all for removing fat. Do you have links to some of particularly
> bad Wiki pages that should be heavily audited?

I would be more entertained to see how far I can take this plan and
if/how discussion will shape it into a workable form. I think that the
entire networking section and more can be wholly rewritten together
with new scripts. Mind that I am only talking about core Xen and not
about XCP/libvirt/foo/bar, in fact I don't even have experience with
any of that and I simply assume that they handle stuff in their own way.

One question that has been bugging me is the xm/xl thing. What are the
exact plans, will xl indeed be phased in while xm will be phased out and
what will/won't be supported by xl? Or did I misinterpret it. Imho
a suggestion to switch over should only be come out when xl becomes
'distribution-grade stable'.

Right now it's just a tool worth testing.
(-That's what "she" said..)

_donduq.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel