RE: [Xen-users] Re: Snapshotting LVM backed guests from dom0

[Jeff Sturm <jeff.sturm@xxxxxxxxxx>]

Chris,

Saw your original post, but hesitated to respond, since I'm not really an 
expert on either Linux block I/O or NFS.  Anyway...

On Sat, Apr 17, 2010 at 2:53 PM, chris <tknchris@xxxxxxxxx> wrote:
> Just looking for some feedback from other people who do this. I know
> its not a good "backup" method but "crash consistent" images have been
> very useful for me in disaster situations just to get OS running
> quickly then restore data from a data backup. My typical setup is to
> put the LV in snapshot mode while guest is running then dd the data to
> a backup file which is on a NFS mount point. The thing that seems to
> be happening is that the VM's performance gets pretty poor during the
> time the copy is happening.

We see this all the time on Linux hosts.  One process with heavy I/O can starve 
others.

I'm not quite sure why but I suspect it has something to do with the unified 
buffer cache.  When reading a large volume with "normal" I/O, buffer pages 
might get quickly replaced with pages that are never going to be read again, 
and your buffer cache hit ratio suffers.  Every other process on the affected 
host that needs to do I/O may experience longer latency as a result.  With Xen, 
that includes any domU.

A quick fix that worked for us:  Direct I/O.  Run your "dd" command with 
"iflag=direct" and/or "oflag=direct", if your version supports it (definitely 
works on CentOS 5.x, definitely *not* on CentOS 4.x).  This bypasses the buffer 
cache completely and forces dd to read/write direct to the underlying disk 
device.  Make sure you use an ample block size ("bs=64k" or larger) so the copy 
will finish in reasonable time.

Not sure if that'll work properly with NFS, however.  (Having been badly burned 
by NFS numerous times I tend to not use it on production hosts.)  To copy disks 
from one host to another, we resort to tricks like piping over ssh (e.g. "dd 
if=<somefile> iflag=direct bs=256k | ssh <otherhost> -c 'dd of=<otherfile> 
oflag=direct bs=256k'").  These copies run slow, but steady.   Importantly they 
run with minimal impact on other processing going on at the time.

> 3.   Tried nice-ing the dd to lowest priority and qemu-dm to highest

"nice" applies only to CPU scheduling and probably isn't helpful for this.  You 
could try playing with ionice, which lets you override scheduling priorities on 
a per-process basis.

Jeff



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users