Re: [Xen-users] High availability Xen with bonding

[Bart Coninckx <bart.coninckx@xxxxxxxxxx>]

On Friday 15 October 2010 13:44:42 Eric van Blokland wrote:
> Hello everyone,
> 
> A few days back I decided to give Ethernet port bonding in Xen another try.
> I've never been able to get it to work properly and after a short search I
> found the network-bridge-bonding script shipped with CentOS-5 probably
> wasn't going to solve my problems. Instead of searching for a tailored
> solution, I thought it would be a good idea to take a leap and try to
> understand Xen networking a bit better and refresh my networking
> knowledge.
> 
> I came up with a configuration I'm currently testing on a pre-production
> environment and I would love to hear from some experts whether what I'm
> doing is sane or that I'm just plain lucky my ping requests are actually
> coming through. Please don't hesitate to give remarks on efficiency,
> security or questioning my sanity. Especially because networking is
> ordinarily not my piece of cake.
> 
> The hardware setup is simple. We have a physical machine with to gigabit
> Ethernet nics, eth0 and eth1, each connected to its own switch. We want to
> achieve failover by bonding (bond0). We're using mode 0 (round robin) to
> gain some bandwidth as well.
> 
> In this test setup, I'm only running one DomU.
> 
> Now for the configuration. I haven't written a wrapper script for the
> purpose of testing and am just using the ifcfgs.
> 
> First of all I've disabled the bridge script in xend-config:
> (network-script '/bin/true')
> 
> Then setup the interfaces:
> 
> # ifcfg-eth0
> DEVICE=eth0
> USERCTL=no
> BOOTPROTO=none
> MACADDR=00:1E:C9:BB:3B:DE
> ONBOOT=yes
> MASTER=bond0
> SLAVE=yes
> 
> # ifcfg-eth1
> DEVICE=eth1
> USERCTL=no
> BOOTPROTO=none
> MACADDR=00:1E:C9:BB:3B:DF
> ONBOOT=yes
> MASTER=bond0
> SLAVE=yes
> 
> # ifcfg-bond0
> DEVICE=bond0
> BOOTPROTO=none
> ONBOOT=yes
> TYPE=Ethernet
> BONDING_OPTS='mode=0 miimon=100'
> BRIDGE=xenbr0
> 
> # ifcfg-xenbr0
> DEVICE=xenbr0
> BOOTPROTO=none
> ONBOOT=yes
> TYPE=Bridge
> STP=OFF
> FD=9
> HELLO=2
> MAXAGE=12
> 
> # ifcfg-veth0
> DEVICE=veth0
> BOOTPROTO=none
> ONBOOT=yes
> IPADDR=192.168.1.11
> NETMASK=255.255.255.0
> GATEWAY=192.168.1.1
> # Our Dom0 will be needing a HW address
> MACADDR=00:16:3E:00:01:00
> 
> # ifcfg-vif0.0
> DEVICE=vif0.0
> BOOTPROTO=none
> ONBOOT=yes
> BRIDGE=xenbr0
> 
> Whatever firewalling I'm going to do, I presume it would be either on bond0
> and/or veth0. Our DomU's will probably do their own firewalling anyway as
> we can't predict what services they will be running. For the sake of
> efficiency and my lack of iptables knowledge, I've changed some bridge
> settings in systctl.conf:
> 
> # Excerpt sysctl.conf
> net.bridge.bridge-nf-filter-vlan-tagged = 0
> net.bridge.bridge-nf-call-ip6tables = 0
> net.bridge.bridge-nf-call-iptables = 0
> net.bridge.bridge-nf-call-arptables = 0
> 
> Finally it seems the ARP request aren't coming through. I fixed that with
> the following commands:
> 
> ip link set eth0 arp off
> ip link set eth1 arp off
> ip link set bond0 arp off
> ip link set vif0.0 arp off
> ip link set vif1.0 arp off
> 
> A wrapper script could do this for me in the future, as I haven't found an
> ifcfg setting to do this for me. The requirement to disable ARP to achieve
> proper, unrouted bridging, limits the use of certain bonding modes, but I
> don't think I care.
> 
> Ifconfig output after setup:
> 
> bond0     Link encap:Ethernet  HWaddr 00:1E:C9:BB:3B:DE
>           inet6 addr: fe80::21e:c9ff:febb:3bde/64 Scope:Link
>           UP BROADCAST RUNNING NOARP MASTER MULTICAST  MTU:1500  Metric:1
>           RX packets:27407897 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:29528747 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:677011912 (645.6 MiB)  TX bytes:2070391875 (1.9 GiB)
> 
> eth0      Link encap:Ethernet  HWaddr 00:1E:C9:BB:3B:DE
>           UP BROADCAST RUNNING NOARP SLAVE MULTICAST  MTU:1500  Metric:1
>           RX packets:15509051 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:14731788 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:2338232100 (2.1 GiB)  TX bytes:3153334765 (2.9 GiB)
>           Interrupt:16 Memory:dfdf0000-dfe00000
> 
> eth1      Link encap:Ethernet  HWaddr 00:1E:C9:BB:3B:DE
>           UP BROADCAST RUNNING NOARP SLAVE MULTICAST  MTU:1500  Metric:1
>           RX packets:11898846 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:14796959 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:2633747108 (2.4 GiB)  TX bytes:3212024406 (2.9 GiB)
>           Interrupt:17 Memory:dfef0000-dff00000
> 
> lo        Link encap:Local Loopback
>           inet addr:127.0.0.1  Mask:255.0.0.0
>           inet6 addr: ::1/128 Scope:Host
>           UP LOOPBACK RUNNING  MTU:16436  Metric:1
>           RX packets:23829 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:23829 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:5148013 (4.9 MiB)  TX bytes:5148013 (4.9 MiB)
> 
> veth0     Link encap:Ethernet  HWaddr 00:16:3E:00:01:00
>           inet addr:192.168.1.11  Bcast:192.168.1.255  Mask:255.255.255.0
>           inet6 addr: fe80::216:3eff:fe00:100/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:27197523 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:14952082 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:541462356 (516.3 MiB)  TX bytes:807191362 (769.7 MiB)
> 
> vif0.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
>           UP BROADCAST RUNNING NOARP MULTICAST  MTU:1500  Metric:1
>           RX packets:14952092 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:27197458 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:807192826 (769.7 MiB)  TX bytes:541406194 (516.3 MiB)
> 
> vif1.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
>           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
>           RX packets:197922 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:245822 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:32
>           RX bytes:193048753 (184.1 MiB)  TX bytes:26837000 (25.5 MiB)
> 
> virbr0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00
>           inet addr:192.168.122.1  Bcast:192.168.122.255 
> Mask:255.255.255.0 inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
> 
> xenbr0    Link encap:Ethernet  HWaddr 00:1E:C9:BB:3B:DE
>           inet6 addr: fe80::21e:c9ff:febb:3bde/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:48495 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:3432718 (3.2 MiB)  TX bytes:398 (398.0 b)
> 
> I hope I haven't forgot a vital setting. Please let me know what you think.
> 
> Kind regards,
> 
> Eric


Eric,

curious to see your progress in this. Up till now I tackled network redundancy 
with multipathing, not with bonding. However, this does not provide added 
speed, though theoretically it should. So I recently decided to switch to 
bonding for the hypervisors in their connections to iSCSI, using rr and 
running over seperate switches, just like you but I'm not at the point of 
installing domU's, so I can't really comment on how and if it works. Will know 
next week though so I will return to this post with my findings...

B.








_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users