Re: [Xen-users] domU has better I/O performance than dom0?

[Mark Williamson <mark.williamson@xxxxxxxxxxxx>]

I don't have an explanation, but I can tell you that when we first tested the 
current IO architecture (which was before Xen 2 was released) we found that 
performance on /certain benchmarks/ could be improved in the domU relative to 
both dom0 and to native execution.  We never traced down exactly why this was 
happening, but suspected it was somehow due to the two layers of IO that were 
going on (e.g. perhaps a fortuitous interaction of the schedulers in dom0 and 
domU, some weird behaviour of the Linux IO stack).

Nonetheless I'm not surprised if some operations are still faster in dom0 - it 
does, after all, have less layers to pass through to perform the IO.

Cheers,
Mark

On Wednesday 28 November 2007, Troels Arvin wrote:
> Hello,
>
> I've done some I/O benchmarks on an RHEL 5.1-based xen setup. The main
> (dom0) server is an x86_64 host with a FC-connected IBM SAN. The guest
> servers are paravirtualized.
>
> I used bonnie++ to stress-test and to try to analyze I/O performance.
> Bonnie++ was run with this command, current working directory being the
> relevant part of the file system:
> bonnie++ -n 4 -s 20g -x 5 -u nobody
>
> The chunk size (20g) was much larger than the memory available for the
> dom0/domU (both having ÂGB RAM available).
> Testing ran for several hours. No stability problems were seen.
>
> Bonnie++ was run on the dom0 and one of the domUs, making sure that the
> involved disks were otherwise idle, or very close to being idle. I.e., a
> quiet (and somewhat slow) part of the SAN was used.
>
> The storage area was made available for the dom0 as a file system on "raw
> device" (no use of logical volume management on the operating system).
> After testing on the dom0, the same device was unmounted, the file system
> was re-created and subsequently allocated to the domU as a "phy"-device;
> the phy-device was then mounted to a suitable mountpoint on the domU.
>
> The tests were run on different times of the day. The dom0 test was
> partly run during work hours where there may have been contention on the
> SAN signalling infrastructure (switch/storage HBAs), although we
> generally believe that we don't have a major bottleneck on the SAN
> optical pathways. The domU test was run during night-time where I/O
> pressure on the SAN infrastructure is probably somewhat lower (although
> various backup and batch jobs make sure that the SAN is never sleeping).
>
> The results were averaged, after filtering out result which seemed
> atypical.
>
> Bonnie wasn't able to detect a difference in the file-creation
> performance, so these values aren't included.
>
> The results:
>
> Host  |     Sequential Output       | Sequential Input  | Random seeks |
>
>       |           (K/sec)           |      (K/sec)      |   (/sec)     |
>       | Per Char | Block  | Rewrite | Per Char | Block  |              |
>
> ======+=============================+===================+==============+
> dom0  |   56739  |  96529 |  46524  |   58346  | 119830 |       94     |
> ------+----------+--------+---------+----------+--------+--------------+
> domU  |   56186  | 112796 |  50178  |   65325  | 202569 |      148     |
> ======+=============================+===================+==============+
> domU  |          |        |         |          |        |              |
> gain% |       -1 |     17 |      8  |      12  |     69 |       56     |
>
> In other words: I've found that my domU's I/O performance generally
> surpasses that of my dom0(!).
>
> On a side note: Running mke2fs went much, much faster on the dom0 than on
> the domU. So for this kind of I/O, the pattern seems to break.
>
> Am I just being an ignorant benchmark-idiot, or could this kind of result
> actually be expected and/or explained?
>
> Is bonnie++ a bad storage benchmarking tool? - If so: What else is better?



-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users