|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-users] Re: Problem with network for new domU's (Grant table problem/limit?) (vif vif-247-0: 28 mapping shared-frames)
Hello, I have been able to reproduce the bug and found out what goes wrong. After creating 79 VM's with all 2 vbd (root disk and swap) and one vif i ran into the problem again. One error caught my attention (and is only mentioned once!): [ 628.040958] No available IRQ to bind to: increase NR_DYNIRQS. After some searching i found: * https://bugzilla.redhat.com/show_bug.cgi?id=442736 * http://www.nabble.com/Unable-to-start-more-than-103-VMs-td11420853.html I downloaded the source of "linux-image-2.6.24-23-xen" and modified "debian/binary-custom.d/xen/patchset/001-xen-base.patch" on line 87148 Here you should change NR_DYNIRQS from 256 to 1024. When you have a VM with 3 "devices" (in my case 2x vbd, 1x vif) you reserve 3 Dynamic IRQ's per domU. 79 * 3 = 237 IRQ's The default limit of 256 is reached since the dom0 also reserves some IRQ's. You can find out how many IRQ's you are using with: grep Dynamic-irq /proc/interrupts |wc -l I have now been able to address over 256 IRQ's and start 90 VM's without any troubles. I think Ubuntu should raise the limit from 256 to 512 or 1024 OR make it a option wich can be passed to the kernel on boot (like suggested in the RedHat bugreport) I hope this helps more sysadmins in the future! - Met vriendelijke groet, Wido den Hollander Hoofd Systeembeheer / CSO Telefoon Support Nederland: 0900 9633 (45 cpm) Telefoon Support BelgiÃ: 0900 70312 (45 cpm) Telefoon Direct: (+31) (0)20 50 60 104 Fax: +31 (0)20 50 60 111 E-mail: support@xxxxxxxxxxxx Website: http://www.pcextreme.nl Kennisbank: http://support.pcextreme.nl/ Netwerkstatus: http://nmc.pcextreme.nl On Tue, 2009-03-17 at 14:59 +0100, PCextreme B.V. - Wido den Hollander wrote: > Hello, > > The last few days i did some further research. > > It seems i have run into some limit for the Xen grant tables (although, > that was what people told me). > > I tried to replicate the error so i set up a new server. Since the > machine wich is having the troubles is a pretty heavy machine (64GB ram) > i could not exactly replicate it. > > So i set up a new server with 6GB ram, but with exactly the same > versions of Xen and i was NOT able to reproduce the error. > > I created 115 domU's (with 48MB of memory) and they all had network > connectivity. > > So i started reading more about the Grant tables and found out that also > vbd's are mapped in these tables. > > For the test i used this config for the domU: > > ***************************************************************** > kernel = '/boot/vmlinuz-2.6.24-22-xen' > ramdisk = '/boot/initrd.img-2.6.24-22-xen' > memory = 512 > vcpus = 1 > > root = '/dev/xvda1 ro' > > disk = [ > 'phy:/dev/xen-domains-root/vps78-root,xvda1,w' > 'phy:/dev/xen-domains-swap/vps78-swap,xvda2,w' > ] > > name = 'vps78' > > vif = [ 'mac=00:16:3e:12:de:e4,vifname=vps78,bridge=vlanbr710' ] > > on_poweroff = 'destroy' > on_reboot = 'restart' > on_crash = 'restart' > > extra='xencons=tty1 rootflags=quota' > ***************************************************************** > > This did NOT work, it resulted in the following message: > > vif vif-279-0: 28 mapping shared-frames 768/769 port 8 > > After searching in drivers/xen/interface.c i found out that it was bad, > it is a error with code 28. > > As i ran out of options i remove the xvda2 line from my domU's config > and i'll be damned, the network connectivity was back! > > I created a new domU (#80) and this had the same troubles, no network. > But when i removed the xvda2 line (for the swap) the network was OK. > > So for the last test i added the swap after it was created: > > root@vps-pool-01:~# xm block-attach > vps78 /dev/xen-domains-swap/vps78-swap xvda2 w > root@vps-pool-01:~# dmesg > [2540234.810635] blkback: ring-ref 299, event-channel 8, protocol 1 > (x86_64-abi) > [2540234.810813] vbd vbd-280-51714: 28 mapping ring-ref 299 port 8 > root@vps-pool-01:~# > > As you can see i have the same kind of error, mapping the vbd fails. > > Today i ordered a new server wich is exactly the same as the machine > wich is running into this limit. So i will be able to reproduce a exact > setup and hopefully reproduce the errors as well. > > I hope to receive this server within one week. > > But in the meantime i hope their is somebody who has any clues of what > this could be. > > root@vps-pool-01:~# xm info > host : vps-pool-01.xen.pcextreme.nl > release : 2.6.24-22-xen > version : #1 SMP Mon Nov 24 21:35:54 UTC 2008 > machine : x86_64 > nr_cpus : 8 > nr_nodes : 1 > cores_per_socket : 4 > threads_per_core : 1 > cpu_mhz : 2500 > hw_caps : > bfebfbff:20100800:00000000:00000140:040ce3bd:00000000:00000001 > total_memory : 64766 > free_memory : 26797 > max_free_memory : 26797 > max_para_memory : 26793 > max_hvm_memory : 26722 > node_to_cpu : node0:0-7 > xen_major : 3 > xen_minor : 2 > xen_extra : .1-rc1-pre > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 > hvm-3.0-x86_32p hvm-3.0-x86_64 > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xffff800000000000 > xen_changeset : unavailable > cc_compiler : gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) > cc_compile_by : buildd > cc_compile_domain : buildd > cc_compile_date : Fri Apr 11 01:13:51 UTC 2008 > xend_config_format : 4 > root@vps-pool-01:~# > > Thank you in advance! > - > Met vriendelijke groet, > > Wido den Hollander > Hoofd Systeembeheer / CSO > Telefoon Support Nederland: 0900 9633 (45 cpm) > Telefoon Support BelgiÃ: 0900 70312 (45 cpm) > Telefoon Direct: (+31) (0)20 50 60 104 > Fax: +31 (0)20 50 60 111 > E-mail: support@xxxxxxxxxxxx > Website: http://www.pcextreme.nl > Kennisbank: http://support.pcextreme.nl/ > Netwerkstatus: http://nmc.pcextreme.nl > > > On Thu, 2009-03-12 at 18:47 +0100, PCextreme B.V. - Wido den Hollander > wrote: > > Hello, > > > > Today i ran into a problem with new domU's on my Xen 3.2 system. > > > > On this server i have 78 domU's running, but when creating domU #79 the > > network for this domU fails. > > > > I allready submitted a bugreport with Ubuntu and please forgive me for > > just posting a link: > > https://bugs.launchpad.net/ubuntu/+source/xen-meta/+bug/341846 > > > > Since this machine is already in production i can't simply reboot or > > build a new kernel. > > > > Could somebody give me a clue where to find the solution? > > > > - > > Met vriendelijke groet, > > > > Wido den Hollander > > Hoofd Systeembeheer / CSO > > Telefoon Support Nederland: 0900 9633 (45 cpm) > > Telefoon Support BelgiÃ: 0900 70312 (45 cpm) > > Telefoon Direct: (+31) (0)20 50 60 104 > > Fax: +31 (0)20 50 60 111 > > E-mail: support@xxxxxxxxxxxx > > Website: http://www.pcextreme.nl > > Kennisbank: http://support.pcextreme.nl/ > > Netwerkstatus: http://nmc.pcextreme.nl > > Attachment:
signature.asc _______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-users
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |