[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-API] xcp-xapi bonding

  • To: xen-api@xxxxxxxxxxxxx
  • From: Carlos Reategui <creategui@xxxxxxxxx>
  • Date: Mon, 11 Mar 2013 22:27:43 -0700
  • Delivery-date: Tue, 12 Mar 2013 05:27:57 +0000
  • List-id: User and development list for XCP and XAPI <xen-api.lists.xen.org>

On Mon, Mar 11, 2013 at 6:13 PM, Alexandre Kouznetsov <alk@xxxxxxxxxx> wrote:
Hi.  Thanks for taking the time. 

El 11/03/13 18:08, Carlos Reategui escribió:

So looks like I need to setup the bonding using the xe create-bond
command just as if I were on XCP/XS.  However it is not clear to me how
my /etc/network/interfaces file should be set.  Does it matter?  Should
I set the interfaces to manual?
I have not used XenAPI over Ubuntu (only plain Xen on Debian, and XCP), but in case of XCP it is supposed to manage the networking setup by itself.
Close enough. 


auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

Do I also need to set up the bridge in this file or will that be done
automatically by xcp-xapi. Eg:

auto xenbr0
iface xenbr0 inet dhcp
bridge_ports eth0

auto xenbr1
iface xenbr1 inet dhcp
bridge_ports eth1
As a reference, very unrelated to Xen-API or XCP, please note that this would not create any "bond", but a plain "bridge". The bonding magic is done by vSwitch in XCP (and probably XenAPI). Other observation is, that Debian based distribution does not need to have eth0 or eth1 interface described or set up, in order to make a valid bridge via /etc/network/interfaces file.
Understood, but do I need to define a bridge to both eth0 and 1 in the interfaces file in order to bond them.  Hmm as I type that  it does not sound right.  If I am going to bond eth0 and 1, do I need to even bother setting up bridges for them?  Seems like a bridge is a "network" in xapi which is what I'll be creating in xe.

Look at this example:
auto xenbr01
iface xenbr01 inet manual
  bridge_ports eth1
  bridge_stp off
  bridge_maxwait 0
  bridge_fd 5

This creates a bridge, set it up automatically on boot, without need to declare anything about eth1 itself. After configured, ifconfig shows eth1 in UP state, but there is no need for any "auto eth1" or "iface eth1" lines.
I guess the assumption here is that xapi picks this up when it is first setup, or do you add "xenbr01" as a network using xe yourself.

Using xe I am guessing I would then do the following:

ETH0=`xe pif-list device=eth0 --minimal`
ETH1=`xe pif-list device=eth1 --minimal`
NETID=`xe network-create name-label=bond0`
BONDID=`xe bond-create network-uuid=$NETID pif-uuids=$ETH0,$ETH1`

Note that if your host is a part of a cluster, "xe pif-list device=eth0" will return a list of PIF's instead of a single one. 

The commands you listed should work, but they will bond your first two interfaces together. Your administration interface could behave funny.
By administration did you mean management?  I want this bond to be my management and guest traffic interface. 
To set the bond mode would I do that on NETID or BONDID above?
Consider specifying "mode=" on your "xe bond-create". In any case, the "mode" is bond's property, not network's.
good to know. 

 What is
the preferred bond mode on an L2 switch that only supports static LAG?
"balance-slb" or "active-backup" should work, depending on what you want. "lacp" will definitely not.
I'll try slb. It seems similar to alb.

Check this reference:

  Same with setting management to true, which UUID?
If you are talking about management interface, then it's a pif's property (the bond will have one associated).
"xe pif-list" will tell.
Ok that makes sense. 

If I want to use static IP addresses would I then pif-reconfigure-ip on
the NETID?
pif-reconfigure-ip is done on PIFID, not NETID.
Ok so I do that on the bond.

 What happens at reboot time will xcp take over, remember the
previous setting and ignore the interfaces file?
Here I fail you.
In XCP, the network configuration is definitely persistent across reboots, the configuration files are well hidden behind xe-toolstack.

 Would it be better to
not use dhcp in the interfaces file?
Yes, much better. On any server. Except, maybe, very special cases.
I'm just going to use it for initial setup where it gets from a temporary ip pool. Then switch to static when configuring via xe.

 Can I define  xenbr0 and 1 in the
interfaces file as manual?  I do need at least eth0 to have an IP so
that I can initially install xcp-xapi.
I would expect that manual alterations of interfaces file, once xapi is installed, would conflict with it. But never tried.
I would suggest to leave eth0 plain at the beginning, and create any bridge or other fancy configuration by using xe-toolstack means.

For my NFS network using a bond of eth2 and 3 I was thinking of keeping
them from xcp and just telling it to "xe pif-forget" on the uuids for
those.  Is that ok?  Im assuming as long as the host knows about that
network it should be fine.
Making this network not to be attached to the new VM's would be enough:
NEWNET=`xe network-list name-label="PUT YOURS HERE" --minimal`
xe network-param-set uuid=${NEWNET} other-config:automatic=false

If you do pif-forget, most likely xapi will be unable to manage a bond using this interface. Unless you intend to manage this interfaces completely out of the scope of xapi.
Yes.  That was the plan.  I was going to create an LACP bond in the interfaces file for my storage network since my storage switch does support LACP.

If this is documented somewhere let me know (I am aware of the XS docs
which do cover some of what I am asking but not in this debian
based environment).
Not the low level config, as I believe.
In XCP case (and I believe it applys to xapi as well), it is supposed to be managed via xe interface. No hand under the hood, unless you know what you are doing.
Some bit uncomfortable at the beginning.

On Sat, Mar 9, 2013 at 12:46 AM, Carlos Reategui <carlos@xxxxxxxxxxxx
<mailto:carlos@xxxxxxxxxxxx>> wrote:
    In my setup xe network-list and xe pif-list are now inconsistent
    with interfaces file and I think this is what is causing me some
>From XCP experience, I treat the internal configuration database as a fragile thing. So, if a hack have introduces any inconsistency, it's better to make sure that part of configuration is overwritten by xe-toolstack. Sometimes it's possible to do that without a re-install, sometimes not. For my experiments, I have set up a automated PXE installation environment, so the re-installs are not so painful.

 I have 4 nics and I am creating 2 separate bonds and then
    bridging them to xenbr0 and 1. One for nfs access to my primary
    storage and the other for all the other vm traffic.  The bonds work
    great on the host but the VMs that start up are having networking
    issues.  This was all setup in the interfaces file and I have not
    done anything network related with xe which I think is my problem
    because xe network lists xenbr0-3 linked to eth0-3 instead of xenbr0
    and 1 linked to bond0 and 1.
Besides what I have written above, check your interfaces naming order.

    Any pointers as to the right way to do this would be greatly
By the book. Forget the mad skills for a while. I was unable to get anything working on XCP until I made it that way.


Once again thank you very much for your response.  This will narrow down my experimenting and save me time.  Once I get this right, I'll be adding this host to cloudstack.

Alexandre Kouznetsov

Xen-api mailing list

Xen-api mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.