RE: [Xen-users] Re: domU network fails under load - vif breaks

["Dustin Henning" <Dustin.Henning@xxxxxxxxxxx>]

        I have a backup running on four Windows HVM domains (I know you asked 
for someone using PV domUs, but these are running on the PV network interfaces, 
as I am using James Harper's GPLPV drivers).  All four DomUs are on the same 
Dom0, which has one gigabit connection.  The backups are going toward a 
separate physical backup server.  Here is a summary of said backups:
  
HVM1:
Start backup objects time 09/05/2008 1:11:04 AM
Elapsed backup objects time 2 hours,14 minutes,30 seconds
Completed 61743 objects, 9.33 GB (100%)
HVM2: 
Start backup objects time 09/05/2008 2:18:36 AM
Elapsed backup objects time 57 minutes,30 seconds
Completed 38071 objects, 6.05 GB (100%)
HVM3: 
Start backup objects time 09/05/2008 2:21:41 AM
Elapsed backup objects time 57 minutes,15 seconds
Completed 36175 objects, 5.94 GB (100%)
HVM4: 
Start backup objects time 09/05/2008 1:17:47 AM
Elapsed backup objects time 1 hour,30 minutes,30 seconds
Completed 36101 objects, 5.92 GB (100%)

        As you can see, these backups overlap, so the network usage across the 
bridge/NIC would be even higher than what it takes to back up 6 GiB in just 
under an hour (I suspect that is quite a bit more than 3.6Mbps, but I haven't 
calculated or benchmarked).  Also, as you can see here, they have been running 
for some time:

[virtadmin@virt1 ~]$ sudo /usr/sbin/xm list 
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  2858     4     r----- 160986.6
xm1                                         17  2048     1     ------ 233816.8
xm2                                         18  1024     1     -b---- 188676.6
xm3                                         19  1024     1     -b---- 166046.2
xm4                                         20  1024     1     -b---- 171760.6
[virtadmin@virt1 ~]$ uptime
 10:50:40 up 65 days, 20:45,  1 user,  load average: 0.00, 0.01, 0.00

        The DomUs are currently IDs 17-20 because of reboots for software 
installs, updates, and whatnot.  xm1's output from systeminfo shows it has been 
up for 17 Days, 18 Hours, 24 Minutes, 31 Seconds.  This is with the backup 
mentioned above running 5 times a week.
 
        Anyway, I am running Fedora 8 with a kernel from their repo.  I have 
attached the .config for that kernel in case you want to compare.  The version 
matches yours, but it's hard telling what additional patches the Fedora kernel 
might include (though technically said patches would be available with the 
source, which should be downloadable from multiple mirrors, so I guess it's not 
so hard telling).  Hope this is helpful,
        Dustin

-----Original Message-----
From: xen-users-bounces@xxxxxxxxxxxxxxxxxxx 
[mailto:xen-users-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Paul
Sent: Thursday, September 04, 2008 18:43
Cc: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-users] Re: domU network fails under load - vif breaks

>
> On Sep 1, 2008, at 8:57 AM, Paul wrote:
>
>   
>
>     At a certain point that I have not established exactly, the
>     network load takes out the pv network. For example, if I initiate
>     a bittorrent session in a pv domU, I get a slow build up of
>     network load, and then connectivity is lost to both of the pv
>     domU's. If I console into them, they cannot ping outside the
>     network, but they can ping their own interfaces. A tcpdump on the
>     dom0 physical shows no traffic. However, during all this, the hvm
>     domains are able to use their network connections without issues. 
>
> I have experienced the same issue with Ubuntu 8.04. The only solution 
> I have found is to limit the rate of the nic so that it doesn't reach 
> a transfer rate that breaks the networking.
> Pepe
>   
Thanks Pepe.  My last test showed the network breaking at 3.6Mbs  - too 
slow to lower the nic rate, and having networking running that slow 
would make the vm unusable for its purpose.

I have upgraded Xen from 3.2.1 to 3.3, and the problem remains, which 
means I have changed:

1) Change domU kernel (2.6.21, 2.6.24, 2.6.25)
2) Change domU userland (gentoo, ubuntu)
3) Change dom0 physical nic + driver (Realtek 8169 -> 8168)
4) Change Xen version (3.2.1 -> 3.3)

The only thing I haven't changed is the dom0 kernel. I am using a stock 
2.6.21-xen gentoo kernel, so it would be great if anyone watching this 
that has pv domUs working under a 2.6.21 kernel would post their .config 
so I can compare it to mine.

Thanks,

Paul

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

Attachment: config-2.6.21.7-3.fc8xen
Description: Binary data

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users