Re: [Xen-devel] [PATCH v9 3/5] xen/mem_sharing: VM forking

[Tamas K Lengyel <tamas@xxxxxxxxxxxxx>]

On Tue, Feb 25, 2020 at 5:06 AM Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
>
> On Tue, Feb 25, 2020 at 04:43:30AM -0700, Tamas K Lengyel wrote:
> > On Tue, Feb 25, 2020 at 3:05 AM Roger Pau Monné <roger.pau@xxxxxxxxxx> 
> > wrote:
> > >
> > > On Fri, Feb 21, 2020 at 10:49:21AM -0800, Tamas K Lengyel wrote:
> > > > VM forking is the process of creating a domain with an empty memory 
> > > > space and a
> > > > parent domain specified from which to populate the memory when 
> > > > necessary. For
> > > > the new domain to be functional the VM state is copied over as part of 
> > > > the fork
> > > > operation (HVM params, hap allocation, etc).
> > >
> > > I've just realized that I'm not sure how are special pages handled,
> > > what happens to the xenstore, console, shared info pages, or the
> > > vcpu info pages if the parent has those registered?
> >
> > The fork starts out with an empty p2m so these pages are not present.
> > In case the guest itself tries to access these pages, or a foreign
> > mapping is set up for them, then that will trigger fork_page from the
> > parent. I see that in the VM restore path clears the pages for
> > HVM_PARAM_CONSOLE_PFN, HVM_PARAM_STORE_PFN, HVM_PARAM_IOREQ_PFN and
> > HVM_PARAM_BUFIOREQ_PFN but doesn't really explain why that would be
> > required. I can certainly add the same special handling for these HVM
> > params when they get copied during the fork hypercall.
>
> Those params are likely set by the toolstack when creating the domain.
> I think you will have to at least copy the values from the parent and
> maybe pre-populate them when forking, that depends on whether internal
> Xen accesses will also trigger the populate logic.

All params and the hvm context gets copied from the parent already.
Whether or not these pages need to be manually populated, I'm not
sure. I haven't noticed any issues when forking a standard VM and not
pre-populating these pages. But I guess it doesn't hurt to be as close
to the save/restore routine as possible.

>
> > As for the vcpu info page, I'm not sure where that gets allocated and
> > mapped normally. I don't see any special handling for that in the
> > save/restore paths.
>
> The shared info page, the vcpu info and the stolen time pages are tear
> down during suspend/restore, and the guest re-registers them when
> resuming. Since the guest is not aware of the fork happening, you will
> have to populate those on fork and make sure the domain fields
> pointing to them are updated, so that Xen can continue updating this
> shared areas.

Could you point me to where this code lives or where these pages are
being mapped into the guest? I've been grepping for a while now and
it's not clear to me still.

>
> > We use the standard createdomain hypercall to
> > setup the VM that will be used for the fork, then use vcpu_create to
> > bring up the vCPUs and just load them with the hvm context, pretty
> > much the same way the restore path would.
>
> Depending on how much of the creation process you skip you might end
> up missing bits, there are a bunch of hypercalls involved in domain
> creation.

I modeled the forking processes heavily on the save/restore routines.
So everything that gets called during a VM restore gets called during
a VM fork, except it happens within Xen instead of the toolstack
issuing a bunch of hypercalls since that's a lot a faster. The only
difference between a save/restore and a fork should be that the p2m of
the fork isn't populated since that happens during runtime.

>
> Note also that during a restore the guest is aware of such process,
> and will know that it needs to re-register some stuff, but that's not
> the case when forking, since the guest is not aware you need to make
> sure everything is in place. There are also the grant table pages,
> which I think should also be handled somehow (or we need to at least
> note this isn't handled and will need fixing).

True, the fork is not aware that something happened (and we want to
keep it that way). But right now everything seems to "just work" as
far as a standard VM is used. There must be a million cornercases left
that I haven't covered for sure.

Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel