Re: Mapping memory into a domain

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Tue, May 06, 2025 at 04:56:12PM -0400, Demi Marie Obenour wrote:
> On 5/6/25 9:06 AM, Alejandro Vallejo wrote:
> > On Tue May 6, 2025 at 3:02 AM CEST, Demi Marie Obenour wrote:
> >> On 5/5/25 7:32 AM, Alejandro Vallejo wrote:
> >>> I suppose this is still about multiplexing the GPU driver the way we
> >>> last discussed at Xen Summit?
> >>>
> >>> On Mon May 5, 2025 at 12:51 AM CEST, Demi Marie Obenour wrote:
> >>>> What are the appropriate Xen internal functions for:
> >>>>
> >>>> 1. Turning a PFN into an MFN?
> >>>> 2. Mapping an MFN into a guest?
> >>>> 3. Unmapping that MFN from a guest?
> >>>
> >>> The p2m is the single source of truth about such mappings.
> >>>
> >>> This is all racy business. You want to keep the p2m lock for the full
> >>> duration of whatever operation you wish do, or you risk another CPU
> >>> taking it and pulling the rug under your feet at the most inconvenient
> >>> time.
> >>>
> >>> In general all this faff is hidden under way too many layers beneath
> >>> copy_{to,from}_guest(). Other p2m manipulation high-level constructs
> >>> that might do interesting things worth looking at may be 
> >>> {map,unmap}_mmio_region()
> >>>
> >>> Note that not every pfn has an associated mfn. Not even every valid pfn
> >>> has necessarily an associated mfn (there's pod). And all of this is
> >>> volatile business in the presence of a baloon driver or vPCI placing
> >>> mmio windows over guest memory.
> >>
> >> Can I check that POD is not in use?  
> > 
> > Maybe, but now you're reaching exponential complexity considering each
> > individual knob of the p2m into account.
> > 
> >>
> >>> In general anything up this alley would need a cohesive pair for
> >>> map/unmap and a credible plan for concurrency and how it's all handled
> >>> in conjunction with other bits that touch the p2m.
> >>
> >> Is taking the p2m lock for the entire operation a reasonable approach
> >> for concurrency?  Will this cause too much lock contention?
> > 
> > Maybe. It'd be fine for a page. Likely not so for several GiB if they
> > aren't already superpages.
> > 
> >>
> >>>> The first patch I am going to send with this information is a 
> >>>> documentation
> >>>> patch so that others do not need to figure this out for themselves.
> >>>> I remember being unsure even after looking through the source code, which
> >>>> is why I am asking here.
> >>>
> >>> That's not surprising. There's per-arch stuff, per-p2mtype stuff,
> >>> per-guesttype stuff. Plus madness like on-demand memory. It's no wonder
> >>> such helpers don't exist and the general manipulations are hard to
> >>> explain.
> >>
> >> Is this a task that is only suitable for someone who has several years
> >> experience working on Xen, or is it something that would make sense for
> >> someone who is less experienced?
> > 
> > The p2m is a very complex beast that integrates more features than I
> > care to count. It requires a lot of prior knowledge. Whoever does it
> > must know Xen fairly well in many configurations.
> > 
> > The real problem is finding the right primitives that do what you want
> > without overcomplicating everything else, preserving system security
> > invariants and have benign (and ideally clear) edge cases.
> > 
> > This was the last email you sent (I think?). Has any of the requirements
> > changed in any direction?
> > 
> >   https://lore.kernel.org/xen-devel/Z5794ysNE4KDkFuT@itl-email/
> 
> Map and Revoke are still needed, with the same requirements as described
> in this email.  Steal and Return were needed for GPU shared virtual memory,
> but it has been decided to not support this with virtio-GPU, so these
> primitives are no longer needed.
> 
> > Something I'm missing there is how everything works without Xen. That
> > might help (me, at least) guage what could prove enough to support the
> > usecase. Are there sequence diagrams anywhere about how this whole thing
> > works without Xen? I vaguely remember you showing something last year in
> > Xen Summit in the design session, but my memory isn't that good :)

Hello,

Sorry, possibly replying a bit out of context here.

Since I will mention this in several places: p2m is the second stage
page-tables used by Xen for PVH and HVM guests.  A p2m violation is
the equivalent of a page-fault for guest p2m accesses.

> A Linux driver that needs access to userspace memory
> pages can get it in two different ways:
> 
> 1. It can pin the pages using the pin_user_pages family of APIs.
>    If these functions succeed, the driver is guaranteed to be able
>    to access the pages until it unpins them.  However, this also
>    means that the pages cannot be paged out or migrated.  Furthermore,
>    file-backed pages cannot be safely pinned, and pinning GPU memory
>    isn’t supported.  (At a minimum, it would prevent the pages from
>    migrating from system RAM to VRAM, so all access by a dGPU would
>    cross the PCIe bus, which would be very slow.)

>From a Xen p2m this is all fine - Xen will never remove pages from the
p2m unless it's requested to.  So the pining, while needed on the Linux
side, doesn't need to be propagated to Xen I would think.

> 
> 2. It can grab the *current* location of the pages and register an
>    MMU notifier.  This works for GPU memory and file-backed memory.
>    However, when the invalidate_range function of this callback, the
>    driver *must* stop all further accesses to the pages.
> 
>    The invalidate_range callback is not allowed to block for a long
>    period of time.  My understanding is that things like dirty page
>    writeback are blocked while the callback is in progress.  My
>    understanding is also that the callback is not allowed to fail.
>    I believe it can return a retryable error but I don’t think that
>    it is allowed to keep failing forever.
> 
>    Linux’s grant table driver actually had a bug in this area, which
>    led to deadlocks.  I fixed that a while back.
> 
> KVM implements the second option: it maps pages into the stage-2
> page tables (or shadow page tables, if that is chosen) and unmaps
> them when the invalidate_range callback is called.

I assume this map and unmap is done by the host as a result of some
guest action?

> Furthermore,
> if a page fault happens while the page is unmapped, KVM will try
> to bring the pages back into memory so the guest can access it.

You could likely handle this in Xen in the following way:

 - A device model will get p2m violations forwarded, as it's the same
   model that's used to handle emulation of device MMIO.  You will
   need to register an ioreq server to request those faults to be
   forwarded, I think the hardware domain kernel will handle those?

 - Allow ioreqs to signal to Xen that a guest operation must be
   retried.  IOW: resume guest execution without advancing the IP.

I think this last bit is the one that will require changes to Xen, so
that you can add a type of ioreq reply that implies a retry from the
guest context.

> For GPU acceleration via virtio-GPU native contexts to work,
> the Xen interface driver needs to do the same thing with GPU
> buffers that KVM does: it needs to fault the pages into guest
> memory on-demand and revoke access to the pages when the host
> kernel demands them back.  There really is no alternative that
> I am aware of.  The need to handle guest page faults doesn’t
> come from the host kernel, but rather from guest userspace.

I'm a bit confused with this last sentence, the "page faults"
mentioned here are p2m violations I think?

Hope this makes some sense.

Regards, Roger.