Re: [Xen-users] Performance problems with domU on drbd

[Florian Tischler <flo_list2007@xxxxxxxx>]

Am Freitag, 15. Februar 2008 schrieb Reto Gantenbein:
> Hello everybody
>
> After 3 days messing around I'm at my wits' end. I have two Sun X4200M2
> servers which I would like to setup as Xen dom0s with a drbd-mirrored
> file system for paravirtualized domU images.
>
> I setup an LVM2 on top of DRBD to create the domU volumes and to be able
> to backup the domUs with LVM snapshot.
>
> I was able to configure this setup and make it run, but when accessing
> the block-frontend device in domU this is terrible slow. E.g. copying a
> 5G file via scp and gigabit Ethernet onto the domU leads to a lot of
> stalls during transfer. The peak transfer rate is about 5MB/s every 1-2

Hi,

Quite a late answer, but maybe better than no answer at all...

I am currently working on a Xen based 2 nodes Linux-HA cluster.
System is Gentoo Hardened AMD64 (multilib) with Xen 3.2 and Kernel 
2.6.20-Xen/2.6.21-Xen on a HP Proliant DL380G5. I am also using LVM2 & DRBD 
8.0.11.

Until now i have not discovered any problems with performance. The system is 
not in production use but testing is already quite far and if nothing goes 
wrong i will put it in production use soon. The speed of syncing 
portage tree or compiling programs in a domU does not feel much slower than 
on dom0.

I have done some basic bonni benchmarks in a domU. (this domU is on a drbd 
disk which is on a lvm volume) Here are the results.

bonnie -s 2048 -m xentest
             -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per 
Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU 
K/sec %CPU  /sec %CPU
xentest  2048 55503 81.0 84324 15.4 41231  1.9 42244 27.5 159844  1.4 891.5  
0.0

Probably not related to your problem - in my opinion your al-extents in your 
drbd.conf are set quite low. this hurts performance. Default is 127, i am 
using 257.

I don´t know if this answer is helpful for you... if you want to have more 
details on my setup let me know. Basically there seems to be not much 
difference, your configs below no not much differ from my configs.

Florian

> minutes but generally it's about a few hundred KB/s. When copying the
> same file onto the DRBD volume mounted directly in dom0 it works with
> the expected 30-40MB/s. Another test was to sync the portage tree but
> also this takes maybe about 20 minutes or spontaneously raises some
> timeout errors. Sometimes after a while the disk fully detaches from the
> domU, so every call of a binary in domU leads to a "Input/output error".
> I guess the domU kernel adds this troubles, but why? There are no
> statements in any log files. E.g. /var/log/xen/xend-debug.log doesn't
> notify the detached block device.
>
> I use Gentoo Linux and tried so far with xen-3.1.2 and xen-3.2.0,
> xen-sources-2.6.18-r8, xen-sources-2.6.20-r6 and xen-sources-2.6.22,
> drbd-8.0.7, drbd-8.0.8 and drbd-8.0.11.
>
> The longer the more I expect a general problem with Xen and this setup,
> a kernel problem or a misconfiguration. Attached are the most important
> configurations for this setup.
>
> Are there any hints or other experiences for such a setup? All ideas are
> welcome.
>
> Thanks for reading and giving any hints...
> Cheers Reto Gantenbein
>
>
> ################ Configurations #################################
>
> /etc/drbd.conf
> ,----------------------------------------------------------------
>
> | global {
> |     usage-count no;
> |     dialog-refresh 1;
> | }
> |
> | common {
> |     startup {
> |         wfc-timeout 60;
> |         degr-wfc-timeout 60;
> |     }
> |
> |     disk {
> |       on-io-error detach;
> |         # also tried 'pass-on' with same behavior
> |     }
> |
> |     syncer {
> |         rate 500M;
> |         al-extents 80;
> |     }
> | }
> |
> | resource "gentoo" {
> |     protocol C;
> |     net {
> |         # required for xen live migration
> |         allow-two-primaries;
> |       after-sb-0pri discard-zero-changes;
> |       after-sb-1pri discard-secondary;
> |    }
> |
> |    on vserver03 {
> |       device        /dev/drbd1;
> |               disk          /dev/xen-vg/gentoo;
> |         address     10.0.0.1:7789;
> |       meta-disk     internal;
> |    }
> |
> |    on vserver04 {
> |      device         /dev/drbd1;
> |      disk           /dev/xen-vg/gentoo;
> |      address        10.0.0.2:7789;
> |      meta-disk      internal;
> |    }
> | }
>
> `----------------------------------------------------------------
>
> /etc/xen/gentoo
> ,----------------------------------------------------------------
>
> | name = "gentoo"
> |
> | kernel = "/cluster/xen/kernel/vmlinuz-gentoo-domU"
> | extra = ""
> | root = "/dev/hda1"
> |
> | memory = 4096
> | vcpus = 3
> | disk = [ 'drbd:gentoo,hda1,w' ]
> | vif = [ 'mac=00:40:a6:00:45:01,bridge=xenbr-core',
> |         'mac=00:40:a6:00:45:02,bridge=xenbr-public' ]
>
> `----------------------------------------------------------------
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users



-- 
Florian Tischler
mailto:flo_list2007@xxxxxxxx
icq#11754147

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users