[Xen-users] Re: Problem with network for new domU's (Grant table problem/limit?) (vif vif-247-0: 28 mapping shared-frames)

["PCextreme B.V. - Wido den Hollander" <wido@xxxxxxxxxxxx>]

Hello,

The last few days i did some further research.

It seems i have run into some limit for the Xen grant tables (although,
that was what people told me).

I tried to replicate the error so i set up a new server. Since the
machine wich is having the troubles is a pretty heavy machine (64GB ram)
i could not exactly replicate it.

So i set up a new server with 6GB ram, but with exactly the same
versions of Xen and i was NOT able to reproduce the error.

I created 115 domU's (with 48MB of memory) and they all had network
connectivity.

So i started reading more about the Grant tables and found out that also
vbd's are mapped in these tables.

For the test i used this config for the domU:

*****************************************************************
kernel = '/boot/vmlinuz-2.6.24-22-xen'
ramdisk = '/boot/initrd.img-2.6.24-22-xen'
memory = 512
vcpus = 1

root = '/dev/xvda1 ro'

disk = [
                  'phy:/dev/xen-domains-root/vps78-root,xvda1,w'
                  'phy:/dev/xen-domains-swap/vps78-swap,xvda2,w'
              ]

name = 'vps78'

vif = [ 'mac=00:16:3e:12:de:e4,vifname=vps78,bridge=vlanbr710' ]

on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'

extra='xencons=tty1 rootflags=quota'
*****************************************************************

This did NOT work, it resulted in the following message:

vif vif-279-0: 28 mapping shared-frames 768/769 port 8

After searching in drivers/xen/interface.c i found out that it was bad,
it is a error with code 28.

As i ran out of options i remove the xvda2 line from my domU's config
and i'll be damned, the network connectivity was back!

I created a new domU (#80) and this had the same troubles, no network.
But when i removed the xvda2 line (for the swap) the network was OK.

So for the last test i added the swap after it was created:

root@vps-pool-01:~# xm block-attach
vps78 /dev/xen-domains-swap/vps78-swap xvda2 w
root@vps-pool-01:~# dmesg
[2540234.810635] blkback: ring-ref 299, event-channel 8, protocol 1
(x86_64-abi)
[2540234.810813] vbd vbd-280-51714: 28 mapping ring-ref 299 port 8
root@vps-pool-01:~#

As you can see i have the same kind of error, mapping the vbd fails.

Today i ordered a new server wich is exactly the same as the machine
wich is running into this limit. So i will be able to reproduce a exact
setup and hopefully reproduce the errors as well.

I hope to receive this server within one week.

But in the meantime i hope their is somebody who has any clues of what
this could be.

root@vps-pool-01:~# xm info
host                   : vps-pool-01.xen.pcextreme.nl
release                : 2.6.24-22-xen
version                : #1 SMP Mon Nov 24 21:35:54 UTC 2008
machine                : x86_64
nr_cpus                : 8
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 1
cpu_mhz                : 2500
hw_caps                :
bfebfbff:20100800:00000000:00000140:040ce3bd:00000000:00000001
total_memory           : 64766
free_memory            : 26797
max_free_memory        : 26797
max_para_memory        : 26793
max_hvm_memory         : 26722
node_to_cpu            : node0:0-7
xen_major              : 3
xen_minor              : 2
xen_extra              : .1-rc1-pre
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64 
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)
cc_compile_by          : buildd
cc_compile_domain      : buildd
cc_compile_date        : Fri Apr 11 01:13:51 UTC 2008
xend_config_format     : 4
root@vps-pool-01:~#

Thank you in advance!
-  
Met vriendelijke groet,

Wido den Hollander
Hoofd Systeembeheer / CSO
Telefoon Support Nederland: 0900 9633 (45 cpm)
Telefoon Support BelgiÃ: 0900 70312 (45 cpm)
Telefoon Direct: (+31) (0)20 50 60 104
Fax: +31 (0)20 50 60 111
E-mail: support@xxxxxxxxxxxx
Website: http://www.pcextreme.nl
Kennisbank: http://support.pcextreme.nl/
Netwerkstatus: http://nmc.pcextreme.nl


On Thu, 2009-03-12 at 18:47 +0100, PCextreme B.V. - Wido den Hollander
wrote:
> Hello,
> 
> Today i ran into a problem with new domU's on my Xen 3.2 system.
> 
> On this server i have 78 domU's running, but when creating domU #79 the
> network for this domU fails.
> 
> I allready submitted a bugreport with Ubuntu and please forgive me for
> just posting a link:
> https://bugs.launchpad.net/ubuntu/+source/xen-meta/+bug/341846
> 
> Since this machine is already in production i can't simply reboot or
> build a new kernel.
> 
> Could somebody give me a clue where to find the solution?
> 
> -  
> Met vriendelijke groet,
> 
> Wido den Hollander
> Hoofd Systeembeheer / CSO
> Telefoon Support Nederland: 0900 9633 (45 cpm)
> Telefoon Support BelgiÃ: 0900 70312 (45 cpm)
> Telefoon Direct: (+31) (0)20 50 60 104
> Fax: +31 (0)20 50 60 111
> E-mail: support@xxxxxxxxxxxx
> Website: http://www.pcextreme.nl
> Kennisbank: http://support.pcextreme.nl/
> Netwerkstatus: http://nmc.pcextreme.nl
>

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users