[Xen-devel] Improving domU restore time

[Rafal Wojtczuk <rafal@xxxxxxxxxxxxxxxxxxxxxx>]

Hello,
I would be grateful for the comments on possible methods to improve domain
restore performance. Focusing on the PV case, if it matters.
1) xen-4.0.0
I see a similar problem to the one reported at the thread at
http://lists.xensource.com/archives/html/xen-devel/2010-05/msg00677.html

Dom0 is 2.6.32.9-7.pvops0 x86_64, xen-4.0.0 x86_64. 
[user@qubes ~]$ xm create /dev/null
        kernel=/boot/vmlinuz-2.6.32.9-7.pvops0.qubes.x86_64 
        root=/dev/mapper/dmroot extra="rootdelay=1000" memory=400
...wait a second...
[user@qubes ~]$ xm save null nullsave
[user@qubes ~]$ time cat nullsave >/dev/null
...
[user@qubes ~]$ time cat nullsave >/dev/null
...
[user@qubes ~]$ time cat nullsave >/dev/null
real    0m0.173s
user    0m0.010s
sys     0m0.164s
/* sits nicely in the cache, let's restore... */
[user@qubes ~]$ time xm restore nullsave
real    0m9.189s
user    0m0.151s
sys     0m0.039s

According to systemtap, xc_restore uses 3812s of CPU time; besides it being
a lot, what uses the remaining 6s ? Just as reported previously, there are 
some errors in xend.log

[2010-05-25 10:49:02 2392] DEBUG (XendCheckpoint:286) restore:shadow=0x0,
_static_max=0x19000000, _static_min=0x0, 
[2010-05-25 10:49:02 2392] DEBUG (XendCheckpoint:305) [xc_restore]:
/usr/lib64/xen/bin/xc_restore 39 3 1 2 0 0 0 0
[2010-05-25 10:49:02 2392] INFO (XendCheckpoint:423) xc_domain_restore
start: p2m_size = 19000
[2010-05-25 10:49:02 2392] INFO (XendCheckpoint:423) Reloading memory pages:
0%
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:423) ERROR Internal error:
Error when reading batch size
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:423) ERROR Internal error:
error when buffering batch, finishing
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:423) 
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:4100%
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:423) Memory reloaded (0
pages)
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:423) read VCPU 0
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:423) Completed checkpoint
load
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:423) Domain ready to be
built.
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:423) Restore exit with rc=0

Note, xc_restore on xen-3.4.3 works much faster (and with no warnings in the
log), with the same dom0 pvops kernel.

Ok, so there is some issue here. Some more generic thoughts below.

2) xen-3.4.3
Firstly, /etc/xen/scripts/block in xen-3.4.3 tries to do something like
for i in /dev/loop* ; do
        losetup $i
so, spawn one losetup process per each existing /dev/loopX; it hogs CPU, 
especially if your system comes with maxloops=255 :). So,
let's replace it with the xen-4.0.0 version, where this problem is fixed (it 
uses losetup -a, hurray).
Then, restore time for a 400MB domain, with the restore file in the cache,
with 4 vbds backed by /dev/loopX, with one vif, is ca 2.7s real time.
According to systemtap, the CPU time requirements are
xend threads- 0.363s
udevd(in dom0) - 0.007s
/etc/xen/scripts/block and its children - 1.075s
xc_restore - 1.368s
/etc/xen/scripts/vif-bridge (in netvm) - 0.130s

The obvious idea to improve /etc/xen/scripts/block shell script execution time 
is to recode it, in some other language that will not spawn hundreds of 
processes to do its job.

Now, xc_restore.
a) Is it correct that when xc_restore runs, the target domain memory is already
zeroed (because hypervisor scrubs free memory, before it is assigned to a
new domain) ? So, xc_save could check whether a given page contains only
zeroes and if so, omit it in the savefile. This could result in quite
significant savings when
- we save a freshly booted domain, or if we can zero out free memory in the 
  domain before saving
- we plan to restore multiple times from the same savefile (yes, vbd must be
restored in this case too).

b) xen-3.4.3/xc_restore reads data from savefile in 4k portions - so, one
read syscall per page. Make it read in larger chunks. It looks it is fixed in
xen-4.0.0, is this correct ?

Also, it looks really excessive that basically copying 400MB of memory takes 
over 1.3s cpu time. Is IOCTL_PRIVCMD_MMAPBATCH the culprit (its
dom0 kernel code ? Xen mm code ? hypercall overhead ? ), anything 
else ?
I am aware that in the usual cases, xc_restore is not the bottleneck 
(savefile reads from the disk or the network is), but in case we can fetch 
savefile quickly, it matters.

Is 3.4.3 branch still being developed, or pure maintenance mode only, so new 
code should be prepared for 4.0.0 ? 

Regards,
Rafal Wojtczuk
Principal Researcher
Invisible Things Lab, Qubes-os project

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel