[Xen-devel] Re: many guests on Xen 4 NR_DYNAMIC_IRQS success report...

[Luke S Crawford <lsc@xxxxxxxxx>]

so I am tenitively moving towards the xen 4/pv_ops kernel (2.6.32.18, 
xen/stable-2.6.32.x as of yesterday).  

I need a lot of guests, as our smallest hardware has 32GiB ram, and our median
guest has 256MiB ram.
 
we followed
http://lists.xensource.com/archives/html/xen-devel/2010-04/msg01537.html

and changed

NR_DYNAMIC_IRQS in arch/x86/include/asm/irq_vectors.h to 2048. 
  
we changed the .config like so:
  
 CONFIG_SPARSE_IRQ=n 
 
then we applied the following patch to arch/x86/include/asm/irq_vectors.h:


  --- ./linux-2.6-pvops.git/arch/x86/include/asm/irq_vectors.h.old        
2010-08-11 15:40:53.000000000 +0000
  +++ ./linux-2.6-pvops.git/arch/x86/include/asm/irq_vectors.h    2010-08-11 
15:41:39.000000000 +0000
  @@ -167,7 +167,7 @@
   # if defined(CONFIG_X86_IO_APIC) && defined(CONFIG_SPARSE_IRQ)
   extern int nr_dynamic_irqs;
   # else
  -#  define NR_DYNAMIC_IRQS                       256
  +#  define NR_DYNAMIC_IRQS                       2048
   # endif
   #endif
  
and we now can get to 338 guests (with three disks each) on our
64GiB 16 core test server.   the 339th guest appears to start, but 
instead sits there on a blank console.   
 
338 guests is enough for me, as my 64GiB boxes are reserved for 1GiB 
guests and up, and even on the 512MiB and smaller guests, the 32GiB 
servers rarely exceed 100 guests.  

Now, if you start more guests, (we started 479) while xl works, xm hangs 
forever.  the domains we start after 338 are unpingable.  Xend can not 
restart.   xenconsoled keeps dying.   We kill guests down to 300, restart 
xend and xenconsoled, and everything appears to be happy once again.  

Anyhow, uh, like I said, it appears to work just fine for 300 guests
and below, which is enough for me.  I'm guessing that much above 300 
guests (338, I believe, is the tipping point when you use my setup)
we just run out of interupts.

[root@chessboard ~]# xl list |wc -l
300
[root@chessboard ~]# wc -l /proc/interrupts 
1830 /proc/interrupts

so that all looks right to me.  Now, it'd be a nice safety feature if
xendomains didn't hoark when it went over the limit, as I need that
to keep the existing domains running, but it looks like I've got enough
headroom here that it's not a big deal.

-- 
Luke S. Crawford
http://prgmr.com/xen/         -   Hosting for the technically adept
http://nostarch.com/xen.htm   -   We don't assume you are stupid.  

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel