|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: Mapping memory into a domain
On 5/9/25 5:47 AM, Alejandro Vallejo wrote: >>>>>> A Linux driver that needs access to userspace memory >>>>>> pages can get it in two different ways: >>>>>> >>>>>> 1. It can pin the pages using the pin_user_pages family of APIs. >>>>>> If these functions succeed, the driver is guaranteed to be able >>>>>> to access the pages until it unpins them. However, this also >>>>>> means that the pages cannot be paged out or migrated. Furthermore, >>>>>> file-backed pages cannot be safely pinned, and pinning GPU memory >>>>>> isn’t supported. (At a minimum, it would prevent the pages from >>>>>> migrating from system RAM to VRAM, so all access by a dGPU would >>>>>> cross the PCIe bus, which would be very slow.) >>>>> >>>>> From a Xen p2m this is all fine - Xen will never remove pages from the >>>>> p2m unless it's requested to. So the pining, while needed on the Linux >>>>> side, doesn't need to be propagated to Xen I would think. > > It might still be helpful to have the concept of pinning to avoid them > being evicted for other reasons (ballooning?). I don't think it'd be > sane to allow returning to Xen a page that a domain ever shared with a > device. Memory mapped into a domain using this API must not be mapped into the IOMMU contexts of any devices assigned to the frontend. This ensures that the backend unexpectedly reclaiming the memory doesn’t cause problems, even if the frontend has an assigned PCI device. If the frontend tries to perform DMA from a different PCI device into memory mapped into a domain using this API, it’s a frontend bug and the IOMMU must block the access. >>>> If pinning were enough things would be simple, but sadly it’s not. >>>> >>>>>> 2. It can grab the *current* location of the pages and register an >>>>>> MMU notifier. This works for GPU memory and file-backed memory. >>>>>> However, when the invalidate_range function of this callback, the >>>>>> driver *must* stop all further accesses to the pages. >>>>>> >>>>>> The invalidate_range callback is not allowed to block for a long >>>>>> period of time. My understanding is that things like dirty page >>>>>> writeback are blocked while the callback is in progress. My >>>>>> understanding is also that the callback is not allowed to fail. >>>>>> I believe it can return a retryable error but I don’t think that >>>>>> it is allowed to keep failing forever. >>>>>> >>>>>> Linux’s grant table driver actually had a bug in this area, which >>>>>> led to deadlocks. I fixed that a while back. >>>>>> >>>>>> KVM implements the second option: it maps pages into the stage-2 >>>>>> page tables (or shadow page tables, if that is chosen) and unmaps >>>>>> them when the invalidate_range callback is called. > > I'm still lost as to what is where, who initiates what and what the end > goal is. Is this about using userspace memory in dom0, and THEN sharing > that with guests for as long as its live? And make enough magic so the > guests don't notice the transitionary period in which there may not be > any memory? It is exactly about this, except that the backend domain can be any domain that is privileged over the frontend domain, rather than only dom0 or the hardware domain. > Or is this about using domU memory for the driver living in dom0? Unfortunately no. This would be a better fit to how Xen is designed, but it isn’t compatible with Linux’s memory management requirements. > Or is this about something else entirely? > > For my own education. Is the following sequence diagram remotely accurate? > > dom0 domU > | | > |---+ | > | | use gfn3 in the driver | > | | (mapped on user thread) | > |<--+ | > | | > | map mfn(gfn3) in domU BAR | > |--------------------------------->| > | +---| > | happily use BAR | | > | +-->| > |---+ | > | | mmu notifier for gfn3 | > | | (invalidate_range) | > |<--+ | > | | > | unmap mfn(gfn3) | > |--------------------------------->| <--- Plus some means to making guest > |---+ +---| vCPUs pause on access. > | | reclaim gfn3 block on | | > |<--+ access | | > | | | > |---+ | | > | | use gfn7 in the driver | | > | | (mapped on user thread) | | > |<--+ | | > | | | > | map mfn(gfn7) in domU BAR | | > |------------------------------+-->| <--- Unpause blocked domU vCPUs > | | This diagram is accurate. >>>> - The switch from “emulated MMIO” to “MMIO or real RAM” needs to >>>> be atomic from the guest’s perspective. >>> >>> Updates of p2m PTEs are always atomic. >> That’s good. > > Updates to a single PTE are atomic, sure. But mapping/unmapping sizes > not congruent with a whole superpage size (i.e: 256 KiB, more than a > page, less than a superpage) wouldn't be, as far as the guest is > concerned. > > But if my understanding above is correct maybe it doesn't matter? It > only needs to be atomic wrt the hypercall that requests it, so that the > gfn is never reused while the guest p2m still holds that mfn. Correct. -- Sincerely, Demi Marie Obenour (she/her/hers) Attachment:
OpenPGP_0xB288B55FFF9C22C1.asc Attachment:
OpenPGP_signature.asc
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |