[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] Performance questions.



I have a problem I'd like to get some input on. Here's my setup:
2x4 core Intel(R) Xeon(R) CPU E5320 @ 1.86GHz
16GB Memory
4x Barracuda 7200.10 SATA 3.0Gb/s 160-GB Hard Drive
4port 3ware 9550SXU-4LP, with the 4 drives in 2 Raid1 mirrors.
Dom-0 Centos 5.1 x86_64
kernel 2.6.18-53.1.6.el5xen

I installed 32 identical "bare metals" Centos 5.1 x86_64 instances. I put a php script that simply calculates pi to a certain degree of accuracy, and adjusted it so it took about .4 seconds to run on one of the Dom-Us. Then from another server, I am using the 'ab' command from apache to benchmark performance on the 32 servers simultaniously. Here's an example of the command I am running:

ab -n 20 -c 20 http://192.168.1.201/pi.php &

I loop this 32 times, once for each server. It's right around a concurrency of 20, and a count of anything higher, when I start getting time outs from 'ab'.
apr_poll: The timeout specified has expired (70007)

Out of the batch of 32, I will invariably get between 3 and 16 that return this error. If I lower the concurrency to 19, I'll get between 0 and 5 time outs. 18 and below, I don't get any failures. Over 20, I the number of servers that time out do increase, but very slowly.

I'm simply trying to find the threshold or cap that I am hitting so I can give a suggestion on how to monitor and manage the highest number of virtual machines possible.

What I've tried looking for is a correlation between when the time out occurs, and the following list of indicators on both the Dom-0 as well as all the Dom-Us.
Interrupts from vmstat
Context Switches from vmstat
CPU usage from vmstat/top/xentop
apache connections from the server-status module
load average

I would prefer to be able to track this sort of behavior from Dom-0. I have test results from running various different combinations of virtual machines, and concurrency levels from ab, but havn't found the underlying cause yet. Any help or pointers on what else to look for would be appreciated.


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.