[Xen-devel] 4.2.1: Poor write performance for DomU.

[Steven Haigh <netwiz@xxxxxxxxx>]

Hi guys,

Firstly, please CC me in to any replies as I'm not a subscriber these days.

I've been trying to debug a problem with Xen 4.2.1 where I am unable to 
achieve more than ~50Mb/sec sustained sequential write to a disk. The 
DomU is configured as such:
name            = "zeus.vm"
memory          = 1024
vcpus           = 2
cpus            = "1-3"
disk            = [ 'phy:/dev/RAID1/zeus.vm,xvda,w', 
'phy:/dev/vg_raid6/fileshare,xvdb,w' ]
vif             = [ "mac=02:16:36:35:35:09, bridge=br203, 
vifname=vm.zeus.203", "mac=10:16:36:35:35:09, bridge=br10, 
vifname=vm.zeus.10" ]
bootloader      = "pygrub"

on_poweroff     = 'destroy'
on_reboot       = 'restart'
on_crash        = 'restart'

I have tested the underlying LVM config by mounting 
/dev/vg_raid6/fileshare from within the Dom0 and running bonnie++ as a 
benchmark:
Version  1.96       ------Sequential Output------ --Sequential Input- 
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP 
/sec %CP
xenhost.lan.crc. 2G   667  96 186976  21 80430  14   956  95 290591  26 
373.7   8
Latency             26416us     212ms     168ms   35494us   35989us 83759us
Version  1.96       ------Sequential Create------ --------Random 
Create--------
xenhost.lan.crc.id. -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
/sec %CP
                 16 14901  32 +++++ +++ 19672  39 15307  34 +++++ +++ 
18158  37
Latency             17838us     141us     298us     365us     133us 296us

~186Mb/sec write, ~290Mb/sec read. Awesome.

I then started a single DomU which gets passed /dev/vg_raid6/fileshare 
through as xvdb. It is then mounted in /mnt/fileshare/. I then ran 
bonnie++ again in the DomU:
Version  1.96       ------Sequential Output------ --Sequential Input- 
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP 
/sec %CP
zeus.crc.id.au   2G   658  96 50618   8 42398  10  1138  99 267568  30 
494.9  11
Latency             22959us     226ms     311ms   14617us   41816us 72814us
Version  1.96       ------Sequential Create------ --------Random 
Create--------
zeus.crc.id.au      -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
/sec %CP
                 16 21749  59 +++++ +++ 31089  73 23283  64 +++++ +++ 
31114  75
Latency             18989us     164us     928us     480us      26us  87us

~50Mb/sec write, ~267Mb/sec read. Not so awesome.

/dev/vg_raid6/fileshare exists as an LV on /dev/md2:

# lvdisplay vg_raid6/fileshare
  --- Logical volume ---
  LV Path                /dev/vg_raid6/fileshare
  LV Name                fileshare
  VG Name                vg_raid6
  LV UUID                cwC0yK-Xr56-WB5v-10bw-3AZT-pYj0-piWett
  LV Write Access        read/write
  LV Creation host, time xenhost.lan.crc.id.au, 2013-02-18 20:59:40 +1100
  LV Status              available
  # open                 1
  LV Size                2.50 TiB
  Current LE             655360
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     1024
  Block device           253:5


md2 : active raid6 sdd[4] sdc[0] sde[1] sdf[5]
      3907026688 blocks super 1.2 level 6, 128k chunk, algorithm 2 
[4/4] [UUUU]
Heres a quick output of 'xm info' - although its full VM load is running 
now:
# xm info
host                   : xenhost.lan.crc.id.au
release                : 3.7.9-1.el6xen.x86_64
version                : #1 SMP Mon Feb 18 14:46:35 EST 2013
machine                : x86_64
nr_cpus                : 4
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 1
cpu_mhz                : 3303
hw_caps                : 
bfebfbff:28100800:00000000:00003f40:179ae3bf:00000000:00000001:00000000
virt_caps              : hvm
total_memory           : 8116
free_memory            : 1346
free_cpus              : 0
xen_major              : 4
xen_minor              : 2
xen_extra              : .1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : dom0_mem=1024M cpufreq=xen dom0_max_vcpus=1 
dom0_vcpus_pin
cc_compiler            : gcc (GCC) 4.4.6 20120305 (Red Hat 4.4.6-4)
cc_compile_by          : mockbuild
cc_compile_domain      : crc.id.au
cc_compile_date        : Sat Feb 16 19:16:38 EST 2013
xend_config_format     : 4

In a nutshell, does anyone know *why* I am only able to get ~50Mb/sec 
sequential writes to the DomU? It certainly isn't a problem getting 
normal speeds to the LV while mounted in the Dom0.
All OS are Scientific Linux 6.3. The Dom0 runs packages from my 
kernel-xen repo (http://au1.mirror.crc.id.au/repo/el6/x86_64/). The DomU 
is completely stock packages.
--
Steven Haigh

Email: netwiz@xxxxxxxxx
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel