[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] Re: Xen, LVM, DRBD, Linux-HA



Ross S. W. Walker wrote:
Please do not hijack threads, please put all your drbd questions
to the drbd-user list.

This is a xen issue.

I'm pretty sure its not a problem with drbd itself but with drbd under xen and probably with a clash between xen networking and disk IO.

For people who are trying to use drbd under xen this is an important issue.


If properly setup and maintained drbd should work properly.

I've been checking and many of our xen servers are showing similar packet loss under disk IO load. However, so far, only drbd shows such catastrophic problems.

Please advise as to how to configure such that disk IO does not kill the network, as this *seems* to be what is happening here.



-Ross
-----Original Message-----
From: xen-users-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-users-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Steve Wray
Sent: Tuesday, April 22, 2008 4:19 PM
To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-users] Re: Xen, LVM, DRBD, Linux-HA

I am going to reply to this thread but I'm going to start from something new as it doesn't seem to be covered in this thread so far.

I have been testing drbd under Xen and found some very disturbing things.

I'd like to implement this in a production system but this scares the hell out of me...

I have two Dom0 servers connected with a crossover cable between two gigabit e1000 NICs. No switch involved.

One DomU on each server with a 20G drbd device shared between them.


The drbd config contains:

   syncer {
     rate 10M;
     group 1;
     al-extents 257;
   }

   net {
     on-disconnect reconnect;
   }


so the net section is working at defaults. At first I had thought that the problems I was seeing was due to timeout values etc and tried various parameters in the net section but nothing made any difference.

When, on the current secondary node, I execute

drbdadm invalidate all

I get frequent errors such as:

drbd0: PingAck did not arrive in time.
drbd0: drbd0_asender [1572]: cstate SyncSource --> NetworkFailure
drbd0: asender terminated
drbd0: drbd_send_block() failed
drbd0: drbd0_receiver [1562]: cstate NetworkFailure --> BrokenPipe
drbd0: short read expecting header on sock: r=-512
drbd0: worker terminated
drbd0: ASSERT( mdev->ee_in_use == 0 ) in /usr/src/modules/drbd/drbd/drbd_receiver.c:1880
drbd0: drbd0_receiver [1562]: cstate BrokenPipe --> Unconnected
drbd0: Connection lost.


I observe the xm top in both Dom0's and I note a HUGE amount of dropped RX packets being reported on both DomU's vif interfaces. The dropping of RX packets is continuous throughout the drbd resync and grows extremely large.

The ifconfig output within the DomU's do not show any dropped packets.

I have used iperf to test the performance of the crossover link and it is fine when there is no drbd syncing going on.

I have tried various things such as setting sysctl.conf options:

net.core.rmem_default=65536
net.core.wmem_default=65536
net.core.rmem_max=16777216
net.core.wmem_max=16777216

net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

but so far the only thing that prevents the "PingAck did not arrive in time" errors is to take the sync rate down to 1M.



My Xen version info is:

Xen version 3.0.3-1 (Debian 3.0.3-0-4)


Please advise...

Thanks!

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.