[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] live saving of domU



Andres Lagar Cavilla wrote:
Anthony Liguori wrote

Moreover, you cannot dump the state of a domain after a pause and expect it to ever run again.

Guests are aware of the physical addresses of the memory that's been allocated to them. Because of this, to save a domain's state in a restorable way you need the guest to "canonicalize" itself. The only way to do this today is through a suspend operation which happens to be a subop of shutdown. Shutdowns are non-recoverable so you cannot use this as a snapshotting mechanism.
My understanding is that the guest only canonicalizes the store and console mfn's and places them on the shared info frame which is passed to the suspend hypercall. The rest of the canonicalizations are done by dom0 user-space code (xc_linux_save).

Sort of. When you pause a domain, it could be doing something like a PTE update in which case it has a PFN in a register (or on the stack somewhere). Part of the reason for having a suspend entry point in the kernel is to ensure that we're in a consistent state.

The guest never really shuts down: it issues the suspend hypercall and waits for it to return. This could happen months later when the domain is resumed :) The suspend hypercall executing in xen is the one that pauses all vcpus and kills the domain.

Actually, take a look at what HYPERVISOR_suspend is:

static inline int
HYPERVISOR_suspend(
   unsigned long srec)
{
   struct sched_shutdown sched_shutdown = {
       .reason = SHUTDOWN_suspend
   };

   int rc = _hypercall3(int, sched_op, SCHEDOP_shutdown,
                &sched_shutdown, srec);

   if (rc == -ENOSYS)
       rc = _hypercall3(int, sched_op_compat, SCHEDOP_shutdown,
                SHUTDOWN_suspend, srec);

   return rc;
}

It's just a shutdown op. It's the same hypercall as reboot/halt and the hypervisor doesn't do anything differently for these calls. What happens next is that the hypervisor stops scheduling the domain and sets the 's' flag in the domain's state. The userspace tools will see that the domain has now "suspended" by checking the shutdown reason and begin the teardown process.

This of course implies that the userspace tools have to remember which domains have issued which requests (so they know who to check these things for). This is why we have the Xend daemon--to keep track of this information.

Is it feasible to use a different hypercall that pauses the domain but doesn't kill it, and once xc_linux_save is done checkpointing have it issue a dom0_op that unpauses the domain?

A domain is "killed" with a dom0_op of domain_destroy which is invoked by Xend. The problem with checkpointing is that once the 's' bit has been set on a domain, there's no way to unset that bit.

For filesystem corruption you're gonna have to hack up your own thing. Probably a CoW solution, where you begin a new "epoch" when resuming from the checkpoint.

Especially with something like dm-userspace on the horizon, this would (hopefully) soon be a moot issue. It just leaves us with this pesky problem of suspend.

Regards,

Anthony Liguori

Andres

The closest thing you can achieve is a localhost migration. There are some caveats to this, of course. The first is that you need to have as much memory as the domain has available since you'll have a copy of the domain created briefly while the migration takes place. Migrations are also quite intrusive since they involve tearing down and bringing up all the devices.

I've gotten a lot of requests for light weight checkpointing. AFAIK, noone is actually working on it though.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.