Re: Mapping memory into a domain

[Alejandro Vallejo <agarciav@xxxxxxx>]

>>>>> A Linux driver that needs access to userspace memory
>>>>> pages can get it in two different ways:
>>>>>
>>>>> 1. It can pin the pages using the pin_user_pages family of APIs.
>>>>>    If these functions succeed, the driver is guaranteed to be able
>>>>>    to access the pages until it unpins them.  However, this also
>>>>>    means that the pages cannot be paged out or migrated.  Furthermore,
>>>>>    file-backed pages cannot be safely pinned, and pinning GPU memory
>>>>>    isn’t supported.  (At a minimum, it would prevent the pages from
>>>>>    migrating from system RAM to VRAM, so all access by a dGPU would
>>>>>    cross the PCIe bus, which would be very slow.)
>>>>
>>>> From a Xen p2m this is all fine - Xen will never remove pages from the
>>>> p2m unless it's requested to.  So the pining, while needed on the Linux
>>>> side, doesn't need to be propagated to Xen I would think.

It might still be helpful to have the concept of pinning to avoid them
being evicted for other reasons (ballooning?). I don't think it'd be
sane to allow returning to Xen a page that a domain ever shared with a
device.

re: being requested. Are there real promises from Xen to that effect? I
could make a hypervisor oversubscribing on memory that swaps non-IOVA
mem in and out to disk, moving it around all the time and it would be
compliant with the current behaviour AIUI, but it wouldn't work with
this scheme, because the mfn's would be off more often than not.

>>>
>>> If pinning were enough things would be simple, but sadly it’s not.
>>>
>>>>> 2. It can grab the *current* location of the pages and register an
>>>>>    MMU notifier.  This works for GPU memory and file-backed memory.
>>>>>    However, when the invalidate_range function of this callback, the
>>>>>    driver *must* stop all further accesses to the pages.
>>>>>
>>>>>    The invalidate_range callback is not allowed to block for a long
>>>>>    period of time.  My understanding is that things like dirty page
>>>>>    writeback are blocked while the callback is in progress.  My
>>>>>    understanding is also that the callback is not allowed to fail.
>>>>>    I believe it can return a retryable error but I don’t think that
>>>>>    it is allowed to keep failing forever.
>>>>>
>>>>>    Linux’s grant table driver actually had a bug in this area, which
>>>>>    led to deadlocks.  I fixed that a while back.
>>>>>
>>>>> KVM implements the second option: it maps pages into the stage-2
>>>>> page tables (or shadow page tables, if that is chosen) and unmaps
>>>>> them when the invalidate_range callback is called.

I'm still lost as to what is where, who initiates what and what the end
goal is. Is this about using userspace memory in dom0, and THEN sharing
that with guests for as long as its live? And make enough magic so the
guests don't notice the transitionary period in which there may not be
any memory?

Or is this about using domU memory for the driver living in dom0?

Or is this about something else entirely?

For my own education. Is the following sequence diagram remotely accurate?

dom0                              domU
 |                                  |
 |---+                              |
 |   | use gfn3 in the driver       |
 |   | (mapped on user thread)      |
 |<--+                              |
 |                                  |
 |  map mfn(gfn3) in domU BAR       |
 |--------------------------------->|
 |                              +---|
 |              happily use BAR |   |
 |                              +-->|
 |---+                              |
 |   | mmu notifier for gfn3        |
 |   | (invalidate_range)           |
 |<--+                              |
 |                                  |
 |  unmap mfn(gfn3)                 |
 |--------------------------------->| <--- Plus some means to making guest 
 |---+                          +---|      vCPUs pause on access.
 |   | reclaim gfn3    block on |   |
 |<--+                 access   |   |
 |                              |   |
 |---+                          |   |
 |   | use gfn7 in the driver   |   |
 |   | (mapped on user thread)  |   |
 |<--+                          |   |
 |                              |   |
 |  map mfn(gfn7) in domU BAR   |   |
 |------------------------------+-->| <--- Unpause blocked domU vCPUs
 |                                  |

>>> - The switch from “emulated MMIO” to “MMIO or real RAM” needs to
>>>   be atomic from the guest’s perspective.
>> 
>> Updates of p2m PTEs are always atomic.
> That’s good.

Updates to a single PTE are atomic, sure. But mapping/unmapping sizes
not congruent with a whole superpage size (i.e: 256 KiB, more than a
page, less than a superpage) wouldn't be, as far as the guest is
concerned.

But if my understanding above is correct maybe it doesn't matter? It
only needs to be atomic wrt the hypercall that requests it, so that the
gfn is never reused while the guest p2m still holds that mfn.

Cheers,
Alejandro