[Xen-users] Slow disk access on forwarded RAID array

[Mauro Condarelli <mc5686@xxxxxxxxx>]

Hi,
I get a sluggish (600kb/s) read access on a Raid1 (mirror) disk array.
Is this to be expected?
What am I doing wrong?

Configuration follows (if more details are needed I'm ready to provide them, 
just ask):

Both Dom0 and DomU are fairly simple debian Wheezy installs.

The server "real" hardware is not state-of-the-art anymore, but it is still a 
reasonably powerful machine: AMD Phenom(tm) II X6 1055T / 8Gb DDR3 RAM

Setup was done following "beginner-guide" (after that I switched to xl 
toolchain).

Dom0 has one plain disk (boot/root/LVM) and two RAID1 arrays (these were on two 
different machines and I rewired them to the server:
> root@vmrunner:~# fdisk -l /dev/sde
>
> Disk /dev/sde: 320.1 GB, 320072933376 bytes
> 255 heads, 63 sectors/track, 38913 cylinders, total 625142448 sectors
> Units = sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x000d11c8
>
>    Device Boot      Start         End      Blocks   Id  System
> /dev/sde1   *        2048      585727      291840   83  Linux
> /dev/sde2          585728    12304383     5859328   82  Linux swap / Solaris
> /dev/sde3        12304384    41601023    14648320   83  Linux
> /dev/sde4        41601024   625141759   291770368   8e  Linux LVM
> root@vmrunner:~# mdadm --detail --scan
> ARRAY /dev/md127 metadata=0.90 UUID=075741b5:c25af231:bfe7d838:0da5cb4d
> ARRAY /dev/md/store metadata=1.2 name=store 
> UUID=b277d0c1:0ade7e6a:d0139b97:ac1a295b
DomU configuration is straightforward:
> bootloader = '/usr/lib/xen-4.1/bin/pygrub'
> vcpus       = '2'
> memory      = '512'
> root        = '/dev/xvda2 ro'
> disk        = [
>                   'phy:/dev/vg0/fileserver-pv-guest-disk,xvda2,w',
>                   'phy:/dev/vg0/fileserver-pv-guest-swap,xvda1,w',
>                   'phy:/dev/md126,xvda3,w',
>                   'phy:/dev/md127,xvda4,w'
>               ]
> name        = 'fileserver-pv-guest'
> dhcp        = 'dhcp'
> vif         = [ 'mac=00:16:3E:59:55:AD' ]
> on_poweroff = 'destroy'
> on_reboot   = 'restart'
> on_crash    = 'restart'
Notice I have a rather small memory (512Mb), but this DomU is going to be 
"just" a file server (nfs+cifs).

Also the configuration on DomU is quite straightforward:
> root@fileserver-pv-guest:/usr/share/doc/fio# cat /etc/fstab
> proc            /proc           proc    defaults        0       0
> devpts          /dev/pts        devpts rw,noexec,nosuid,gid=5,mode=620 0  0
> /dev/xvda1 none swap sw 0 0
> /dev/xvda2 / ext3 noatime,nodiratime,errors=remount-ro 0 1
> /dev/xvda3 /srv/shares/Store ext4 noatime,nodiratime,errors=remount-ro 0 2
> /dev/xvda4 /srv/shares/Store/private ext4 noatime,nodiratime,errors=remount-ro 
> 0 2
But performance is NOT good:
> root@fileserver-pv-guest:~# cat rendom-read-test.fio
> ; random read of 128mb of data
>
> [random-read]
> rw=randread
> size=128m
> directory=/srv/shares/Store/Store/tmp/
> root@fileserver-pv-guest:~# fio rendom-read-test.fio
> random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
> 2.0.8
> Starting 1 process
> random-read: Laying out IO file(s) (1 file(s) / 128MB)
> Jobs: 1 (f=1): [r] [100.0% done] [1704K/0K /s] [426 /0  iops] [eta 00m:00s]
> random-read: (groupid=0, jobs=1): err= 0: pid=4028
>   read : io=131072KB, bw=677296 B/s, iops=165 , runt=198167msec
>     clat (usec): min=118 , max=964702 , avg=6040.88, stdev=7552.44
>      lat (usec): min=119 , max=964703 , avg=6041.92, stdev=7552.44
>     clat percentiles (usec):
>      |  1.00th=[  126],  5.00th=[  131], 10.00th=[  141], 20.00th=[  151],
>      | 30.00th=[  167], 40.00th=[ 3888], 50.00th=[ 5728], 60.00th=[ 7520],
>      | 70.00th=[ 9280], 80.00th=[11072], 90.00th=[12864], 95.00th=[13888],
>      | 99.00th=[18048], 99.50th=[25984], 99.90th=[29824], 99.95th=[33536],
>      | 99.99th=[68096]
>     bw (KB/s)  : min=  211, max= 1689, per=100.00%, avg=661.45, stdev=108.43
>     lat (usec) : 250=31.42%, 500=0.25%, 750=0.34%, 1000=0.07%
>     lat (msec) : 2=0.22%, 4=8.33%, 10=33.08%, 20=25.44%, 50=0.84%
>     lat (msec) : 100=0.01%, 250=0.01%, 1000=0.01%
>   cpu          : usr=0.34%, sys=0.17%, ctx=32880, majf=0, minf=24
>   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>      issued    : total=r=32768/w=0/d=0, short=r=0/w=0/d=0
>
> Run status group 0 (all jobs):
>    READ: io=131072KB, aggrb=661KB/s, minb=661KB/s, maxb=661KB/s, 
> mint=198167msec, maxt=198167msec
>
> Disk stats (read/write):
>   xvda3: ios=32853/6, merge=0/2, ticks=199448/5640, in_queue=205200, util=99.82%
Any hint/pointer welcome
Regards
Mauro

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users