[Xen-devel] Re: [Xen-users] XEN - networking and performance

["D. Duckworth" <donduq@xxxxxxxxxxx>]

Salutations,

From the xen-users list:
> I would like some advice from people how are/were using  Xen 3.4.2 -
> it should be a rather stable release. Dom0 is CentOS 5.5 64bit with
> Xen 3.4.2 installed with the default settings (minus cpu pinning to
> Dom0 and memory setup for 2GB). There are 2 built-in nics (Broadcom)
> and an add-on network card (Intel) with another 4 nics. Currently
> only one NIC is used for all network access, and as far as
> networking, the default setings are used - xend-config.sxp:
> (network-script network-bridge) (vif-script vif-bridge)
> 
> The questions are:
> 
> How can I improve the network performance (now all the VM are sharing
> one bridge):
> 
> a. creating multiple bridges and assigning a VM (DomU) per bridge
> b. trying to hide the NICs from Dom0 using something like "pciback
> hide" - (pointers/example of how one would do this in Centos 5.5
> would be highly appreciated...)

Xen Networking has been a thorn in my eye and a similar question has
been with me for a long time now. So prepare, for this response contains
rage.

Xen networking has room for many different approaches, yet the best
thing about its scripts is that they are not mandatory to use. You can
fully adjust the scripts to your needs or even replace them with your
own. You can find modifications on forums and blogs although most of
them just seem to be copies of few suggestions made by few people.
Logical, because quite frankly it's a pain to grope what the Xen
scripts and udev rules really do, let alone grok most of what they
do out of the box.

Right now I just care about creating my ideal networking solution, i.e.
routing, bridging and firewall stuff for vms with different roles. I am
running Xen 4.1.2-rc3-pre non-professionally on a quad core single cpu
1U server. with 4 hard drives in RAID10 configuration. The server has
one usable Ethernet port with multiple globally routable IPs. I can't
use the other ethernet port; the server has no IPMI and the ISP declines
use of two ports by one system because the data center is a no smoking
zone for both humans and routers and switches.

So the highest priority is to reach dom0 from the Internet and
therefore my grub has fallback options, one of which is a boot to Linux
with no Xen. In turn this means that dom0's networking boot scripts
may not depend on Xen at all, and Xen may not change networking in any
way unless specified. My dom0 is a minimal system that only controls vms
and networking. I want dom0 to be small and simple so the obvious
choice is Arch Linux. Dom0 should be separated from the domUs in that
the domUs cannot reach dom0 and one domU (domN) should do all
networking for the other domUs.

I tried to use xl with xen4 for a while but due to bugs and missing
features I had to go back to xm and xend. This is where the fun
begins. In the past I used xend with network-bridge and for some strange
reason (voodoo probably) I blindly accepted that script in the past and
blamed myself for not appreciating it. But let's be blunt and honest:
the scripts, in particular the script that *modifies dom0 networking
during xend startup* is the biggest piece of sh!# idea I have ever seen
in Xen. It creates bridges, takes eth0 down, tortures dom0 with occult
ip addr/route, brctl, sysctl and iptables awk/sed manipulations and then
it has you looking at your screen yearning for the moment that ping
timeouts become ping replies, telling you that your box is reachable
again. This script is a malevolent demon from the sewers of Norman the
Golgothan and the worst part is that network-bridge is also still the
recommended default!

On the more positive side there was a fantastic update in Xen 4 where
network-bridge changed a bit so that "it will not bring a bridge up if
one already exists". Whoever wrote it should get a corporate medal for
that and then a long vacation to a deserted island with an MSX II and
no floppies. How can this even be approved by Xen's senior project
manager, or is that a vacant position?

It surely seems so. Xen's /etc/xen/scripts (another design fail, why
not /usr?) and udev scripts are confusing ad-hoc bloatware routines and
are not transparent at all. With the current xen4 I saw the premature
advice to more or less 'prepare for migration from xm to xl'. Yet, xl
supports less and is conflicting: there is no vifname, no 'xl
new/delete', no more python, no relocation and suddenly there is a
conflict between 'xm start domain' versus 'xl start /etc/xen/domain'.

So new features emerge, adding to the confusion of the end user, while
old problems are not being fixed properly. I wonder why, especially
because it does not seem that xm and xend are the broken parts that
need to be replaced by an unstable interface.

What needs attention first and foremost are two things, first of which
is real and wise effort into one simple, minimal script that just
handles the minimum in a transparent way e.g. control the hypervisor,
manage vms, manage the backend. Of course networking can be done on
domain start too, but this has happen in an entirely different way from
what it does and how it does it. This is so important because it gives
more control to the user that runs Xen. It's also a good moment to
build in proper and mature support for IPv6.

Secondly, the website and documentation should be cleaned up and
revised where appropriate. The current situation is a mess that has
a much too steep and incompatible learning curve right now - for
example, a bridge should just not be named eth0 and a physical device
should not be renamed at all. It's fundamentally wrong, stupid, mad as
hell and a PR failure for Xen to do it this way out of the box. No
matter how often and detailed it has been documented on the website. 

I propose something like the following for xen networking:

* Xen will not manipulate non-xen devices or a firewall under any
  circumstance, it might only add or substract routes and/or rules from
  the routing tables,
* Allow for networking configuration per domU. For example let
  networking per device be nat, routed, bridged or custom, where
  all name the interface and bring it up; nat only adds the ip to the
  routing table; routed could be an array of routes and rules that need
  to be added or subtracted from various routing tables and it might
  support proxyarp; bridged turns off arp, sets the mac on the vif and
  then adds the interface to a bridge that should already be created by
  the user; and custom is a custom set of unmanaged commands after
  creating and destroying a domain.

I am aware that this can already be done with Xen. However, that
process is quite arbitrary and it does things no one asked for. So one
has to read the scripts. For example with the iptables part of
vif-bridge. It is not handled transparently, it is quite arbitrary and
it automatically executes for all vms that are being started. This leads
you to wonder what more it does without you knowing it...

So, with that off my chest and the second line of my network-bridge
being the words "exit 0" Xen lets my dom0 configuration alone
like it is supposed to do. While KVM is becoming a 'next cool
thing' for many people I would still prefer a separate hypervisor so now
the fat just has to be removed from Xen.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel