Re: [Xen-devel] Re: [Xen-users] XEN - networking and performance

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

On Sat, Oct 08, 2011 at 07:07:01PM +0000, D. Duckworth wrote:
> Salutations,
> 
> From the xen-users list:
> > I would like some advice from people how are/were using  Xen 3.4.2 -
> > it should be a rather stable release. Dom0 is CentOS 5.5 64bit with
> > Xen 3.4.2 installed with the default settings (minus cpu pinning to
> > Dom0 and memory setup for 2GB). There are 2 built-in nics (Broadcom)
> > and an add-on network card (Intel) with another 4 nics. Currently
> > only one NIC is used for all network access, and as far as
> > networking, the default setings are used - xend-config.sxp:
> > (network-script network-bridge) (vif-script vif-bridge)
> > 
> > The questions are:
> > 
> > How can I improve the network performance (now all the VM are sharing
> > one bridge):
> > 
> > a. creating multiple bridges and assigning a VM (DomU) per bridge
> > b. trying to hide the NICs from Dom0 using something like "pciback
> > hide" - (pointers/example of how one would do this in Centos 5.5
> > would be highly appreciated...)
> 
> Xen Networking has been a thorn in my eye and a similar question has
> been with me for a long time now. So prepare, for this response contains
> rage.
> 
> Xen networking has room for many different approaches, yet the best
> thing about its scripts is that they are not mandatory to use. You can
> fully adjust the scripts to your needs or even replace them with your
> own. You can find modifications on forums and blogs although most of
> them just seem to be copies of few suggestions made by few people.
> Logical, because quite frankly it's a pain to grope what the Xen
> scripts and udev rules really do, let alone grok most of what they
> do out of the box.
> 
> Right now I just care about creating my ideal networking solution, i.e.
> routing, bridging and firewall stuff for vms with different roles. I am
> running Xen 4.1.2-rc3-pre non-professionally on a quad core single cpu
> 1U server. with 4 hard drives in RAID10 configuration. The server has
> one usable Ethernet port with multiple globally routable IPs. I can't
> use the other ethernet port; the server has no IPMI and the ISP declines
> use of two ports by one system because the data center is a no smoking
> zone for both humans and routers and switches.
> 
> So the highest priority is to reach dom0 from the Internet and
> therefore my grub has fallback options, one of which is a boot to Linux
> with no Xen. In turn this means that dom0's networking boot scripts
> may not depend on Xen at all, and Xen may not change networking in any
> way unless specified. My dom0 is a minimal system that only controls vms
> and networking. I want dom0 to be small and simple so the obvious
> choice is Arch Linux. Dom0 should be separated from the domUs in that
> the domUs cannot reach dom0 and one domU (domN) should do all
> networking for the other domUs.
> 
> I tried to use xl with xen4 for a while but due to bugs and missing
> features I had to go back to xm and xend. This is where the fun
> begins. In the past I used xend with network-bridge and for some strange
> reason (voodoo probably) I blindly accepted that script in the past and
> blamed myself for not appreciating it. But let's be blunt and honest:
> the scripts, in particular the script that *modifies dom0 networking
> during xend startup* is the biggest piece of sh!# idea I have ever seen
> in Xen. It creates bridges, takes eth0 down, tortures dom0 with occult
> ip addr/route, brctl, sysctl and iptables awk/sed manipulations and then
> it has you looking at your screen yearning for the moment that ping
> timeouts become ping replies, telling you that your box is reachable
> again. This script is a malevolent demon from the sewers of Norman the
> Golgothan and the worst part is that network-bridge is also still the

<laughs>
> recommended default!

Can you point me to where it mentions that please?
> 
> On the more positive side there was a fantastic update in Xen 4 where
> network-bridge changed a bit so that "it will not bring a bridge up if
> one already exists". Whoever wrote it should get a corporate medal for
> that and then a long vacation to a deserted island with an MSX II and
> no floppies. How can this even be approved by Xen's senior project
> manager, or is that a vacant position?

We realized that the networking setup is quite complex and would be best
left in the hands of the admins. The problem is that..
> 
> It surely seems so. Xen's /etc/xen/scripts (another design fail, why
> not /usr?) and udev scripts are confusing ad-hoc bloatware routines and
> are not transparent at all. With the current xen4 I saw the premature
> advice to more or less 'prepare for migration from xm to xl'. Yet, xl
> supports less and is conflicting: there is no vifname, no 'xl
> new/delete', no more python, no relocation and suddenly there is a
> conflict between 'xm start domain' versus 'xl start /etc/xen/domain'.
> 
> So new features emerge, adding to the confusion of the end user, while
> old problems are not being fixed properly. I wonder why, especially
> because it does not seem that xm and xend are the broken parts that
> need to be replaced by an unstable interface.
> 
> What needs attention first and foremost are two things, first of which
> is real and wise effort into one simple, minimal script that just
> handles the minimum in a transparent way e.g. control the hypervisor,
> manage vms, manage the backend. Of course networking can be done on
> domain start too, but this has happen in an entirely different way from
> what it does and how it does it. This is so important because it gives
> more control to the user that runs Xen. It's also a good moment to
> build in proper and mature support for IPv6.
> 
> Secondly, the website and documentation should be cleaned up and
> revised where appropriate. The current situation is a mess that has
> a much too steep and incompatible learning curve right now - for
> example, a bridge should just not be named eth0 and a physical device
> should not be renamed at all. It's fundamentally wrong, stupid, mad as
> hell and a PR failure for Xen to do it this way out of the box. No
> matter how often and detailed it has been documented on the website. 

.. the documentation and setup is sometimes quite hard. BTW, we are
going to do on Oct 26th a Documentation Day to clean up some of this mess.
Would you be intereted in helping along - perhaps in the networking Wiki?
> 
> I propose something like the following for xen networking:
> 
> * Xen will not manipulate non-xen devices or a firewall under any
>   circumstance, it might only add or substract routes and/or rules from
>   the routing tables,

Uh, what is 'non-xen' devices? Like bridges?

> * Allow for networking configuration per domU. For example let
>   networking per device be nat, routed, bridged or custom, where
>   all name the interface and bring it up; nat only adds the ip to the
>   routing table; routed could be an array of routes and rules that need
>   to be added or subtracted from various routing tables and it might
>   support proxyarp; bridged turns off arp, sets the mac on the vif and
>   then adds the interface to a bridge that should already be created by
>   the user; and custom is a custom set of unmanaged commands after
>   creating and destroying a domain.

You lost me. <sigh> I am using a bridge configuration and just
do:

auto lo
iface lo inet loopback

auto switch
iface switch inet static
        address 192.168.101.16
        netmask 255.255.255.0
        gateway 192.168.101.1
        bridge_ports eth2

And just use that 'bridge=switch' in all my configuration. And that
seems to work just fine - wouldn't that be best way of providing
the first network setup to users? I would think the majority of folks
do something akin to this?

> 
> I am aware that this can already be done with Xen. However, that
> process is quite arbitrary and it does things no one asked for. So one
> has to read the scripts. For example with the iptables part of
> vif-bridge. It is not handled transparently, it is quite arbitrary and
> it automatically executes for all vms that are being started. This leads
> you to wonder what more it does without you knowing it...
> 
> So, with that off my chest and the second line of my network-bridge
> being the words "exit 0" Xen lets my dom0 configuration alone
> like it is supposed to do. While KVM is becoming a 'next cool
> thing' for many people I would still prefer a separate hypervisor so now
> the fat just has to be removed from Xen.
> 

I am all for removing fat. Do you have links to some of particularly
bad Wiki pages that should be heavily audited?

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel