Re: [Xen-users] Severe slow disk write issue on LVM-deployed DomU

[g <g.d.monnezza@xxxxxxxxxx>]

Hi Adam. Thanks for you quick answer. Am very happy to see that somebody 
is involved in this issue. I feel less alone :)
Here below my reply, on line point by point:

On 08/01/2013 05:27 AM, Adam Goryachev wrote:
> On 01/08/13 12:56, g wrote:
> > Hi everyone.
> > I have a really severe performance issue on disk write on my domU's PV
> > guest.
> > - Xen 4.1 fresh install on Debian Wheezy (kernel 3.2.0-4-amd64) via
> > apt-get
> > - Simple disk (no raid) for Linux/Xen Dom0 installation
> > - LVM2 on top of raid5 disk raid5 disk for storing DomU
> > - Single LV for each DomU
> >
> > The problem
> > While write speed (mesured via dd command) on Dom0 is near 163MB/s on
> > the mounted DomU LV , the same dd command provides 8MB/s if executed
> > within
> > the running DomU (with "noop" I/O scheduler set)
> > Running "top" in DomU, i  get high load average due to high %wa (no
> > high CPU load)
> >
> > DomU disk is submitted to Xen via configuration file with:
> > disk  = [ 'phy:/dev/xenstore/test,xvda1,w' ]
> >
> > If the same DomU is deployed on file image ("test.img" cloned from
> > "test" LV), write speed rise up to 63MB/s
> > disk  = [ 'file:/home/test.img,xvda1,w' ]
> >
> > Obviously I can provide further detailed configuration parameters
> >
> > Does anyone faced to the same problem? Does anyone can point me in the
> > right direction?
> A couple of things to check.
>
> shutdown the VM, and check the read/write speed on the LV from the dom0
> /dev/xenstore/test
The speed on LVM /dev/xenstore/test when DomU is powered off is 
impressive: 163MB/s
> Most likely, you have one or more snapshots of the LV, and this totally
> kills performance. You might get away with a single snapshot and
> performance might still be acceptable.
No snapshots active (never run any snapshot on this new Dom0 server)

> Also, try adjusting the value of /sys/block/md1/md/stripe_cache_size
> I ended up using 4096 which seemed to provide the best performance (I'm
> using 4 x SSD in RAID5).
I'm not using software raid. Are you writing "md1" as generic block 
device? In my case which block device should I adjust?
> Verify performance on dom0 first, then compare it to the performance
> within the domu.
Performance for disk write in Dom0 are great (> 20) if compared with 
DomU deployed on LVM volume.
Disk write speed  on DomU deployed  on file-image disk are half the 
speed of Dom0, but it's dangerous working with fileimage disk
> Make sure you allocate sufficient CPU resources to dom0 as well, or
> performance of the domu can suffer.
On this server (HP Proliant) I have 8GB RAM, 2x 8-core Intel CPUs, only 
one domU with 1 vCpu, 1GB RAM, no swap
On another server (same model, old xen linux kernel 2.6.32-5) where the 
problems began, I have 24GB RAM, 2x 8-core Intel CPUs, DomU with 4 
vCPUs, 4GB swap
On both the server the problem is the same: 8MB/s disk write speed on 
DomU if deployed on an LVM2 volume (unpartioned, with ext3 or ext4 FS)
Thanks to anyone can give help to this issue
giuliano


> Regards,
> Adam
>
> --
> Adam Goryachev Website Managers www.websitemanagers.com.au
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxx
> http://lists.xen.org/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users