Re: [PATCH v14 0/3] VM forking

[Tamas K Lengyel <tamas@xxxxxxxxxxxxx>]

On Wed, Apr 8, 2020 at 5:15 AM Jan Beulich <jbeulich@xxxxxxxx> wrote:
>
> On 06.04.2020 17:20, Tamas K Lengyel wrote:
> > The following series implements VM forking for Intel HVM guests to allow for
> > the fast creation of identical VMs without the assosciated high startup 
> > costs
> > of booting or restoring the VM from a savefile.
> >
> > JIRA issue: https://xenproject.atlassian.net/browse/XEN-89
> >
> > The fork operation is implemented as part of the "xl fork-vm" command:
> >     xl fork-vm -C <config> -Q <qemu-save-file> -m <max-vcpus> <parent_domid>
> >
> > By default a fully functional fork is created. The user is in charge 
> > however to
> > create the appropriate config file for the fork and to generate the QEMU 
> > save
> > file before the fork-vm call is made. The config file needs to give the
> > fork a new name at minimum but other settings may also require changes. 
> > Certain
> > settings in the config file of both the parent and the fork have to be set 
> > to
> > default. Details are documented.
> >
> > The interface also allows to split the forking into two steps:
> >     xl fork-vm --launch-dm no \
> >                -m <max-vcpus> \
> >                -p <parent_domid>
> >     xl fork-vm --launch-dm late \
> >                -C <config_file_for_fork> \
> >                -Q <qemu_save_file> \
> >                <fork_domid>
> >
> > The split creation model is useful when the VM needs to be created as fast 
> > as
> > possible. The forked VM can be unpaused without the device model being 
> > launched
> > to be monitored and accessed via VMI. Note however that without its device
> > model running (depending on what is executing in the VM) it is bound to
> > misbehave or even crash when its trying to access devices that would be
> > emulated by QEMU. We anticipate that for certain use-cases this would be an
> > acceptable situation, in case for example when fuzzing is performed of code
> > segments that don't access such devices.
> >
> > Launching the device model requires the QEMU Xen savefile to be generated
> > manually from the parent VM. This can be accomplished simply by connecting 
> > to
> > its QMP socket and issuing the "xen-save-devices-state" command. For example
> > using the standard tool socat these commands can be used to generate the 
> > file:
> >     socat - UNIX-CONNECT:/var/run/xen/qmp-libxl-<parent_domid>
> >     { "execute": "qmp_capabilities" }
> >     { "execute": "xen-save-devices-state", \
> >         "arguments": { "filename": "/path/to/save/qemu_state", \
> >                         "live": false} }
> >
> > At runtime the forked VM starts running with an empty p2m which gets lazily
> > populated when the VM generates EPT faults, similar to how altp2m views are
> > populated. If the memory access is a read-only access, the p2m entry is
> > populated with a memory shared entry with its parent. For write memory 
> > accesses
> > or in case memory sharing wasn't possible (for example in case a reference 
> > is
> > held by a third party), a new page is allocated and the page contents are
> > copied over from the parent VM. Forks can be further forked if needed, thus
> > allowing for further memory savings.
> >
> > A VM fork reset hypercall is also added that allows the fork to be reset to 
> > the
> > state it was just after a fork, also accessible via xl:
> >     xl fork-vm --fork-reset -p <fork_domid>
> >
> > This is an optimization for cases where the forks are very short-lived and 
> > run
> > without a device model, so resetting saves some time compared to creating a
> > brand new fork provided the fork has not aquired a lot of memory. If the 
> > fork
> > has a lot of memory deduplicated it is likely going to be faster to create a
> > new fork from scratch and asynchronously destroying the old one.
> >
> > The series has been tested with Windows VMs and functions as expected. Linux
> > VMs when forked from a running VM will have a frozen VNC screen. Linux VMs 
> > at
> > this time can only be forked with a working device model when the parent VM 
> > was
> > restored from a snapshot using "xl restore -p". This is a known limitation.
> > Also note that PVHVM/PVH Linux guests have not been tested. Forking most 
> > likely
> > works but PV devices and drivers would require additional wiring to set 
> > things
> > up properly since the guests are unaware of the forking taking place, unlike
> > the save/restore routine where the guest is made aware of the procedure.
> >
> > Forking time has been measured to be 0.0007s, device model launch to be 
> > around
> > 1s depending largely on the number of devices being emulated. Fork resets 
> > have
> > been measured to be 0.0001s under the optimal circumstances.
> >
> > New in v14:
> >     minor adjustments
> >
> > Patch 1 implements the VM fork
> > Patch 2 implements fork reset operation
> > Patch 3 adds the toolstack-side code implementing VM forking and reset
> >
> > Tamas K Lengyel (3):
> >   xen/mem_sharing: VM forking
> >   x86/mem_sharing: reset a fork
>
> I've applied these two, but ...
>
> >   xen/tools: VM forking toolstack side
>
> ... since this one doesn't have any ack or alike I'll defer to
> the tool stack maintainers here.
>

Thanks! I haven't got much feedback on the toolstack side in a while
now. We had a discussion on the design of the xl interface early on
but that was about it. Hopefully the tool stack maintainers get a
chance to look at it and get it merged now that the hypervisor side is
done.

Tamas