Re: [Xen-users] Xen 3.4.3 - bad IO performance on drbd devices

[Felix Botner <botner@xxxxxxxxxxxxx>]

Hi

some more information about the xen drbd performance problem.
Any test without memory restriction for dom0. The network connection beetween 
the drbd servers limits the troughput to ~230 MB for each drbd device (2 x 
1000 Mbps cards, bond mode 0). 



bonnie on the local drive with hypevisor
~ 400 MB/sec

bonnie on the connected drbd device with hypevisor
~ 170 MB/sec (bonnie on one drbd device/server)
~ 110 MB/sec (bonnie on both drbd devices/servers)

bonnie on the connected drbd device without hypevisor
~ 230 MB/sec (bonnie on one drbd device/server)
~ 200 MB/sec (bonnie on both drbd devices/servers)

bonnie on the disconnected drbd device with hypevisor
~ 300 MB/sec

bonnie on the disconnected drbd device without hypevisor
~ 360 MB/sec

What interests me is the throughput when writing to both servers. With the XEN 
hypevisor i get 110 MB/sec on each drbd device (220 MB/sec io throughput on 
each server because drbd writes "locale" and "remote"). Without the 
hypervisor i get 200 MB/sec on each drbd device (400 MB/sec io throughput on 
each server, the maximum of what the io backend allows).

But even with only one drbd device the io is without hypevisor much better 
(230 MB/sec without and 170 MB/sec with hypervisor)

The strange thing is, a drbd resync gives me with or without hypervisor ~ 230 
MB/sec (one drbd device). And when i start one server without hypervisor, 
bonnie gives me ~ 230 MB/sec on that server (and on the other server, because 
also writes remote).


Any hints?


xm info
host                   : samla
release                : 2.6.32-ucs9-xen-amd64
version                : #1 SMP Thu Jul 22 04:32:22 UTC 2010
machine                : x86_64
nr_cpus                : 8
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 1
cpu_mhz                : 2133
hw_caps                : 
bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000
virt_caps              : hvm
total_memory           : 24564
free_memory            : 22230
node_to_cpu            : node0:0-7
node_to_memory         : node0:22230
xen_major              : 3
xen_minor              : 4
xen_extra              : .3
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.3.2 (Debian 4.3.2-1.1.13.200909082302)
cc_compile_by          : root
cc_compile_domain      : [unknown]
cc_compile_date        : Thu Jun 17 14:54:34 UTC 2010
xend_config_format     : 4

Am Dienstag 20 Juli 2010 15:54:21 schrieb Felix Botner:
> Am Dienstag 20 Juli 2010 14:35:30 schrieb Pasi Kärkkäinen:
> > On Tue, Jul 20, 2010 at 10:46:34AM +0200, Felix Botner wrote:
> > > Hi everyone,
> > >
> > > i have two servers installed with a debian/lenny based os (64bit), a
> > > debian/sid based kernel 2.6.32-xen-amd64 and xen 3.4.3-4. Each server
> > > has two drbd devices (protocol c, formatted with ext4) and is primary
> > > for one of them. Every drbd pair has a dedicated network interface (a
> > > bond mode 0 interface with two 1000 Mbps cards).
> >
> > <snip>
> >
> > > The io performance on the connected drbd devices is significantly worse
> > > if i start the kernel with the xen hypervisor (with "kernel
> > > /boot/xen-3.4.3.gz"). Without the hypervisor (but the same kernel) the
> > > systems are about 50% faster.
> >
> > Are you measuring from dom0 or from a guest?
>
> From dom0, there are no guests at the moment.
>
> > > Why is there such a difference?
> > > Can i optimize my xend (i already added dom0_mem=2048M dom0_max_vcpus=2
> > > dom0_vcpus_pin as boot parameter with no effect)?
> > > Are there any known issues using XEN and bonding/drbd?
> > >
> > > Feel free to ask for more information about the system or the setup.
> >
> > How much memory does your server have?
> > ie. how much ram do you have when you boot it baremetal without Xen.
>
> ~20GB without Xen. Now i removed the hypervisor param dom0_mem=2048M from
> menu.lst (in xend-config set (dom0-min-mem 196) and (enable-dom0-ballooning
> yes)), rebooted and as far as i know there should be no memory restriction
> for dom0. "xm list" shows me the complete memory for dom0:
>
> server1-> xm list
> Name                                        ID   Mem VCPUs      State
> Time(s)
> Domain-0                                     0 18323     2     r-----  
> 1785.0
>
> server2-> xm list
> Name                                        ID   Mem VCPUs      State
> Time(s)
> Domain-0                                     0 22375     2     r-----  
> 1754.2
>
> But bonnie++ gives me still bad results:
>
> server1,60000M,,,113444,30,55569,13,,,231555,23,622.6,0,16,25634,70,+++++,+
>++, +++++,+++,+++++,+++,+++++,+++,+++++,+++
> server2,60000M,,,114014,31,53864,13,,,243541,27,617.4,0,16,+++++,+++,+++++,
> +++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
>
> So i don't think caching is the issue, or?
>
> bye
>
> Felix
>
> > Remember all the free memory of the host/dom0 will be used by linux
> > pagecache.. So if you limit dom0 to 2GB, it'll have less cache than the
> > baremetal case.
> >
> > -- Pasi


-- 
Felix Botner

Open Source Software Engineer

Univention GmbH
Linux for your business
Mary-Somerville-Str.1
28359 Bremen
Tel. : +49 421 22232-0
Fax : +49 421 22232-99

botner@xxxxxxxxxxxxx>
http://www.univention.de

Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876 

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users