Re: Mapping memory into a domain

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Fri, May 09, 2025 at 11:47:36AM +0200, Alejandro Vallejo wrote:
> >>>>> A Linux driver that needs access to userspace memory
> >>>>> pages can get it in two different ways:
> >>>>>
> >>>>> 1. It can pin the pages using the pin_user_pages family of APIs.
> >>>>>    If these functions succeed, the driver is guaranteed to be able
> >>>>>    to access the pages until it unpins them.  However, this also
> >>>>>    means that the pages cannot be paged out or migrated.  Furthermore,
> >>>>>    file-backed pages cannot be safely pinned, and pinning GPU memory
> >>>>>    isn’t supported.  (At a minimum, it would prevent the pages from
> >>>>>    migrating from system RAM to VRAM, so all access by a dGPU would
> >>>>>    cross the PCIe bus, which would be very slow.)
> >>>>
> >>>> From a Xen p2m this is all fine - Xen will never remove pages from the
> >>>> p2m unless it's requested to.  So the pining, while needed on the Linux
> >>>> side, doesn't need to be propagated to Xen I would think.
> 
> It might still be helpful to have the concept of pinning to avoid them
> being evicted for other reasons (ballooning?). I don't think it'd be
> sane to allow returning to Xen a page that a domain ever shared with a
> device.

If mapped using the p2m_mmio_direct type in the p2m a domain won't be
able to balloon them out.  It would also be misguided for a guest
kernel to attempt to balloon out memory that I presume will be inside
of a PCI device BAR from the guest point of view.

> re: being requested. Are there real promises from Xen to that effect? I
> could make a hypervisor oversubscribing on memory that swaps non-IOVA
> mem in and out to disk, moving it around all the time and it would be
> compliant with the current behaviour AIUI, but it wouldn't work with
> this scheme, because the mfn's would be off more often than not.

Even if Xen supported domain memory swapping, that could never be used
with domains that have devices attached, as it's not possible to fixup
the p2m on IOMMU fault and retry the access.

Not sure you could even move mfns around, as you would need an atomic
way to copy the previous page contents and set the PTE to point to the
new page.

Unless you want to get into a (IMO) complicated scheme where the
domain notifies the hypervisor which ranges are being used for device
DMA accesses (and thus requires guest kernel changes), I think
swapping of guest memory when there are assigned devices is a no-go.

Xen has (or had? as I never actually seen it being used) a mechanism
to swap domain memory to a dom0 file (see tools/xenpaging.c).  However
more than one provider had mentioned to me that one feature they
particularly preferred of Xen over KVM is that it would never swap
guest memory.  Not sure if that's still the case, but some struggled
to prevent KVM from swapping guest memory, and got complains of
slowness from their tenants.

For the purposes of getting a prototype I would suggest that you
assume p2m memory cannot be randomly swapped out, unless requested by
either the guest or the control domain.

> >>>
> >>> If pinning were enough things would be simple, but sadly it’s not.
> >>>
> >>>>> 2. It can grab the *current* location of the pages and register an
> >>>>>    MMU notifier.  This works for GPU memory and file-backed memory.
> >>>>>    However, when the invalidate_range function of this callback, the
> >>>>>    driver *must* stop all further accesses to the pages.
> >>>>>
> >>>>>    The invalidate_range callback is not allowed to block for a long
> >>>>>    period of time.  My understanding is that things like dirty page
> >>>>>    writeback are blocked while the callback is in progress.  My
> >>>>>    understanding is also that the callback is not allowed to fail.
> >>>>>    I believe it can return a retryable error but I don’t think that
> >>>>>    it is allowed to keep failing forever.
> >>>>>
> >>>>>    Linux’s grant table driver actually had a bug in this area, which
> >>>>>    led to deadlocks.  I fixed that a while back.
> >>>>>
> >>>>> KVM implements the second option: it maps pages into the stage-2
> >>>>> page tables (or shadow page tables, if that is chosen) and unmaps
> >>>>> them when the invalidate_range callback is called.
> 
> I'm still lost as to what is where, who initiates what and what the end
> goal is. Is this about using userspace memory in dom0, and THEN sharing
> that with guests for as long as its live? And make enough magic so the
> guests don't notice the transitionary period in which there may not be
> any memory?
> 
> Or is this about using domU memory for the driver living in dom0?
> 
> Or is this about something else entirely?
> 
> For my own education. Is the following sequence diagram remotely accurate?
> 
> dom0                              domU
>  |                                  |
>  |---+                              |
>  |   | use gfn3 in the driver       |
>  |   | (mapped on user thread)      |
>  |<--+                              |
>  |                                  |
>  |  map mfn(gfn3) in domU BAR       |
>  |--------------------------------->|
>  |                              +---|
>  |              happily use BAR |   |
>  |                              +-->|
>  |---+                              |
>  |   | mmu notifier for gfn3        |
>  |   | (invalidate_range)           |
>  |<--+                              |
>  |                                  |
>  |  unmap mfn(gfn3)                 |
>  |--------------------------------->| <--- Plus some means to making guest 
>  |---+                          +---|      vCPUs pause on access.
>  |   | reclaim gfn3    block on |   |
>  |<--+                 access   |   |
>  |                              |   |
>  |---+                          |   |
>  |   | use gfn7 in the driver   |   |
>  |   | (mapped on user thread)  |   |
>  |<--+                          |   |
>  |                              |   |
>  |  map mfn(gfn7) in domU BAR   |   |
>  |------------------------------+-->| <--- Unpause blocked domU vCPUs

The guest vCPU will already pause on access if there's a p2m
violation, until the ioreq has completed and the vCPU execution can
resume.  That's in control of the ioreq server that handles the
request.

I don't know about the dom0 user-space part, but that's possibly of no
concern for the implementation side in Xen?

My understanding of the actions needed from the Xen side is:

 1. Map either RAM owned by the hardware domain or an MMIO page into
    a domain p2m.
 2. Remove entries from a domain p2m.
 3. Handle p2m violations resulting from guest accesses, using 1. and
    force a guest access retry (or emulate the access).

1. Can possibly be done with XEN_DOMCTL_memory_mapping and
XENMEM_add_to_physmap_batch, but as I understood it it's not ideal.
Demi would like a way to use the same hypercall to map either RAM or
IOMEM into a domain p2m.

2. What hypercall to use depends on how the memory is mapped.

3. ioreq servers will already get requests for accesses to unmapped
regions they have registered for.  If the access is to be retried we
need to expand ioreq interface a bit to handle this case.  Adding a
new ioreq state like STATE_IORESP_RETRY might be enough?  Maybe I'm
being naive though.

>  |                                  |
> 
> >>> - The switch from “emulated MMIO” to “MMIO or real RAM” needs to
> >>>   be atomic from the guest’s perspective.
> >> 
> >> Updates of p2m PTEs are always atomic.
> > That’s good.
> 
> Updates to a single PTE are atomic, sure. But mapping/unmapping sizes
> not congruent with a whole superpage size (i.e: 256 KiB, more than a
> page, less than a superpage) wouldn't be, as far as the guest is
> concerned.

I've assumed the question was towards PTE updates, as to whether
PTE entries where always consistent.

> But if my understanding above is correct maybe it doesn't matter? It
> only needs to be atomic wrt the hypercall that requests it, so that the
> gfn is never reused while the guest p2m still holds that mfn.

I think it only matters that the PTE is always consistent, either
mapped or unmapped (and thus generate an ioreq request on access when
unmapped).

Regards, Roger.