|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: Mapping memory into a domain
On 5/9/25 6:50 AM, Roger Pau Monné wrote: > On Fri, May 09, 2025 at 11:47:36AM +0200, Alejandro Vallejo wrote: >>>>>>> A Linux driver that needs access to userspace memory >>>>>>> pages can get it in two different ways: >>>>>>> >>>>>>> 1. It can pin the pages using the pin_user_pages family of APIs. >>>>>>> If these functions succeed, the driver is guaranteed to be able >>>>>>> to access the pages until it unpins them. However, this also >>>>>>> means that the pages cannot be paged out or migrated. Furthermore, >>>>>>> file-backed pages cannot be safely pinned, and pinning GPU memory >>>>>>> isn’t supported. (At a minimum, it would prevent the pages from >>>>>>> migrating from system RAM to VRAM, so all access by a dGPU would >>>>>>> cross the PCIe bus, which would be very slow.) >>>>>> >>>>>> From a Xen p2m this is all fine - Xen will never remove pages from the >>>>>> p2m unless it's requested to. So the pining, while needed on the Linux >>>>>> side, doesn't need to be propagated to Xen I would think. >> >> It might still be helpful to have the concept of pinning to avoid them >> being evicted for other reasons (ballooning?). I don't think it'd be >> sane to allow returning to Xen a page that a domain ever shared with a >> device. > > If mapped using the p2m_mmio_direct type in the p2m a domain won't be > able to balloon them out. It would also be misguided for a guest > kernel to attempt to balloon out memory that I presume will be inside > of a PCI device BAR from the guest point of view. Indeed it will be inside a BAR. >> re: being requested. Are there real promises from Xen to that effect? I >> could make a hypervisor oversubscribing on memory that swaps non-IOVA >> mem in and out to disk, moving it around all the time and it would be >> compliant with the current behaviour AIUI, but it wouldn't work with >> this scheme, because the mfn's would be off more often than not. > > Even if Xen supported domain memory swapping, that could never be used > with domains that have devices attached, as it's not possible to fixup > the p2m on IOMMU fault and retry the access. > > Not sure you could even move mfns around, as you would need an atomic > way to copy the previous page contents and set the PTE to point to the > new page. > > Unless you want to get into a (IMO) complicated scheme where the > domain notifies the hypervisor which ranges are being used for device > DMA accesses (and thus requires guest kernel changes), I think > swapping of guest memory when there are assigned devices is a no-go. > > Xen has (or had? as I never actually seen it being used) a mechanism > to swap domain memory to a dom0 file (see tools/xenpaging.c). However > more than one provider had mentioned to me that one feature they > particularly preferred of Xen over KVM is that it would never swap > guest memory. Not sure if that's still the case, but some struggled > to prevent KVM from swapping guest memory, and got complains of > slowness from their tenants. > > For the purposes of getting a prototype I would suggest that you > assume p2m memory cannot be randomly swapped out, unless requested by > either the guest or the control domain. The API being discussed here needs to support frontends that have assigned PCI devices, but the pages should never be mapped into the frontend domain’s IOMMU context. If the frontend tries to DMA into one of these pages it’s a frontend bug. >>>>> If pinning were enough things would be simple, but sadly it’s not. >>>>> >>>>>>> 2. It can grab the *current* location of the pages and register an >>>>>>> MMU notifier. This works for GPU memory and file-backed memory. >>>>>>> However, when the invalidate_range function of this callback, the >>>>>>> driver *must* stop all further accesses to the pages. >>>>>>> >>>>>>> The invalidate_range callback is not allowed to block for a long >>>>>>> period of time. My understanding is that things like dirty page >>>>>>> writeback are blocked while the callback is in progress. My >>>>>>> understanding is also that the callback is not allowed to fail. >>>>>>> I believe it can return a retryable error but I don’t think that >>>>>>> it is allowed to keep failing forever. >>>>>>> >>>>>>> Linux’s grant table driver actually had a bug in this area, which >>>>>>> led to deadlocks. I fixed that a while back. >>>>>>> >>>>>>> KVM implements the second option: it maps pages into the stage-2 >>>>>>> page tables (or shadow page tables, if that is chosen) and unmaps >>>>>>> them when the invalidate_range callback is called. >> >> I'm still lost as to what is where, who initiates what and what the end >> goal is. Is this about using userspace memory in dom0, and THEN sharing >> that with guests for as long as its live? And make enough magic so the >> guests don't notice the transitionary period in which there may not be >> any memory? >> >> Or is this about using domU memory for the driver living in dom0? >> >> Or is this about something else entirely? >> >> For my own education. Is the following sequence diagram remotely accurate? >> >> dom0 domU >> | | >> |---+ | >> | | use gfn3 in the driver | >> | | (mapped on user thread) | >> |<--+ | >> | | >> | map mfn(gfn3) in domU BAR | >> |--------------------------------->| >> | +---| >> | happily use BAR | | >> | +-->| >> |---+ | >> | | mmu notifier for gfn3 | >> | | (invalidate_range) | >> |<--+ | >> | | >> | unmap mfn(gfn3) | >> |--------------------------------->| <--- Plus some means to making guest >> |---+ +---| vCPUs pause on access. >> | | reclaim gfn3 block on | | >> |<--+ access | | >> | | | >> |---+ | | >> | | use gfn7 in the driver | | >> | | (mapped on user thread) | | >> |<--+ | | >> | | | >> | map mfn(gfn7) in domU BAR | | >> |------------------------------+-->| <--- Unpause blocked domU vCPUs > > The guest vCPU will already pause on access if there's a p2m > violation, until the ioreq has completed and the vCPU execution can > resume. That's in control of the ioreq server that handles the > request. > > I don't know about the dom0 user-space part, but that's possibly of no > concern for the implementation side in Xen? I believe so, yes. > My understanding of the actions needed from the Xen side is: > > 1. Map either RAM owned by the hardware domain or an MMIO page into > a domain p2m. > 2. Remove entries from a domain p2m. > 3. Handle p2m violations resulting from guest accesses, using 1. and > force a guest access retry (or emulate the access). > > 1. Can possibly be done with XEN_DOMCTL_memory_mapping and > XENMEM_add_to_physmap_batch, but as I understood it it's not ideal. > Demi would like a way to use the same hypercall to map either RAM or > IOMEM into a domain p2m. Indeed so, and also the backend domain might be a driver domain instead of the hardware domain. It needs to have privilege over the frontend, but it should not need privilege over the whole system. > 2. What hypercall to use depends on how the memory is mapped. > > 3. ioreq servers will already get requests for accesses to unmapped > regions they have registered for. If the access is to be retried we > need to expand ioreq interface a bit to handle this case. Adding a > new ioreq state like STATE_IORESP_RETRY might be enough? Maybe I'm > being naive though. This is where an implementation in a real userspace emulator would be very useful, to ensure that the API being implemented is actually usable in practice. >>>>> - The switch from “emulated MMIO” to “MMIO or real RAM” needs to >>>>> be atomic from the guest’s perspective. >>>> >>>> Updates of p2m PTEs are always atomic. >>> That’s good. >> >> Updates to a single PTE are atomic, sure. But mapping/unmapping sizes >> not congruent with a whole superpage size (i.e: 256 KiB, more than a >> page, less than a superpage) wouldn't be, as far as the guest is >> concerned. > > I've assumed the question was towards PTE updates, as to whether > PTE entries where always consistent. > >> But if my understanding above is correct maybe it doesn't matter? It >> only needs to be atomic wrt the hypercall that requests it, so that the >> gfn is never reused while the guest p2m still holds that mfn. > > I think it only matters that the PTE is always consistent, either > mapped or unmapped (and thus generate an ioreq request on access when > unmapped). You are correct. -- Sincerely, Demi Marie Obenour (she/her/hers) Attachment:
OpenPGP_0xB288B55FFF9C22C1.asc Attachment:
OpenPGP_signature.asc
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |