|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: Mapping memory into a domain
On 5/7/25 1:39 PM, Roger Pau Monné wrote:
> On Tue, May 06, 2025 at 04:56:12PM -0400, Demi Marie Obenour wrote:
>> On 5/6/25 9:06 AM, Alejandro Vallejo wrote:
>>> On Tue May 6, 2025 at 3:02 AM CEST, Demi Marie Obenour wrote:
>>>> On 5/5/25 7:32 AM, Alejandro Vallejo wrote:
>>>>> I suppose this is still about multiplexing the GPU driver the way we
>>>>> last discussed at Xen Summit?
>>>>>
>>>>> On Mon May 5, 2025 at 12:51 AM CEST, Demi Marie Obenour wrote:
>>>>>> What are the appropriate Xen internal functions for:
>>>>>>
>>>>>> 1. Turning a PFN into an MFN?
>>>>>> 2. Mapping an MFN into a guest?
>>>>>> 3. Unmapping that MFN from a guest?
>>>>>
>>>>> The p2m is the single source of truth about such mappings.
>>>>>
>>>>> This is all racy business. You want to keep the p2m lock for the full
>>>>> duration of whatever operation you wish do, or you risk another CPU
>>>>> taking it and pulling the rug under your feet at the most inconvenient
>>>>> time.
>>>>>
>>>>> In general all this faff is hidden under way too many layers beneath
>>>>> copy_{to,from}_guest(). Other p2m manipulation high-level constructs
>>>>> that might do interesting things worth looking at may be
>>>>> {map,unmap}_mmio_region()
>>>>>
>>>>> Note that not every pfn has an associated mfn. Not even every valid pfn
>>>>> has necessarily an associated mfn (there's pod). And all of this is
>>>>> volatile business in the presence of a baloon driver or vPCI placing
>>>>> mmio windows over guest memory.
>>>>
>>>> Can I check that POD is not in use?
>>>
>>> Maybe, but now you're reaching exponential complexity considering each
>>> individual knob of the p2m into account.
>>>
>>>>
>>>>> In general anything up this alley would need a cohesive pair for
>>>>> map/unmap and a credible plan for concurrency and how it's all handled
>>>>> in conjunction with other bits that touch the p2m.
>>>>
>>>> Is taking the p2m lock for the entire operation a reasonable approach
>>>> for concurrency? Will this cause too much lock contention?
>>>
>>> Maybe. It'd be fine for a page. Likely not so for several GiB if they
>>> aren't already superpages.
>>>
>>>>
>>>>>> The first patch I am going to send with this information is a
>>>>>> documentation
>>>>>> patch so that others do not need to figure this out for themselves.
>>>>>> I remember being unsure even after looking through the source code, which
>>>>>> is why I am asking here.
>>>>>
>>>>> That's not surprising. There's per-arch stuff, per-p2mtype stuff,
>>>>> per-guesttype stuff. Plus madness like on-demand memory. It's no wonder
>>>>> such helpers don't exist and the general manipulations are hard to
>>>>> explain.
>>>>
>>>> Is this a task that is only suitable for someone who has several years
>>>> experience working on Xen, or is it something that would make sense for
>>>> someone who is less experienced?
>>>
>>> The p2m is a very complex beast that integrates more features than I
>>> care to count. It requires a lot of prior knowledge. Whoever does it
>>> must know Xen fairly well in many configurations.
>>>
>>> The real problem is finding the right primitives that do what you want
>>> without overcomplicating everything else, preserving system security
>>> invariants and have benign (and ideally clear) edge cases.
>>>
>>> This was the last email you sent (I think?). Has any of the requirements
>>> changed in any direction?
>>>
>>> https://lore.kernel.org/xen-devel/Z5794ysNE4KDkFuT@itl-email/
>>
>> Map and Revoke are still needed, with the same requirements as described
>> in this email. Steal and Return were needed for GPU shared virtual memory,
>> but it has been decided to not support this with virtio-GPU, so these
>> primitives are no longer needed.
>>
>>> Something I'm missing there is how everything works without Xen. That
>>> might help (me, at least) guage what could prove enough to support the
>>> usecase. Are there sequence diagrams anywhere about how this whole thing
>>> works without Xen? I vaguely remember you showing something last year in
>>> Xen Summit in the design session, but my memory isn't that good :)
>
> Hello,
>
> Sorry, possibly replying a bit out of context here.
>
> Since I will mention this in several places: p2m is the second stage
> page-tables used by Xen for PVH and HVM guests. A p2m violation is
> the equivalent of a page-fault for guest p2m accesses.
>
>> A Linux driver that needs access to userspace memory
>> pages can get it in two different ways:
>>
>> 1. It can pin the pages using the pin_user_pages family of APIs.
>> If these functions succeed, the driver is guaranteed to be able
>> to access the pages until it unpins them. However, this also
>> means that the pages cannot be paged out or migrated. Furthermore,
>> file-backed pages cannot be safely pinned, and pinning GPU memory
>> isn’t supported. (At a minimum, it would prevent the pages from
>> migrating from system RAM to VRAM, so all access by a dGPU would
>> cross the PCIe bus, which would be very slow.)
>
> From a Xen p2m this is all fine - Xen will never remove pages from the
> p2m unless it's requested to. So the pining, while needed on the Linux
> side, doesn't need to be propagated to Xen I would think.
If pinning were enough things would be simple, but sadly it’s not.
>> 2. It can grab the *current* location of the pages and register an
>> MMU notifier. This works for GPU memory and file-backed memory.
>> However, when the invalidate_range function of this callback, the
>> driver *must* stop all further accesses to the pages.
>>
>> The invalidate_range callback is not allowed to block for a long
>> period of time. My understanding is that things like dirty page
>> writeback are blocked while the callback is in progress. My
>> understanding is also that the callback is not allowed to fail.
>> I believe it can return a retryable error but I don’t think that
>> it is allowed to keep failing forever.
>>
>> Linux’s grant table driver actually had a bug in this area, which
>> led to deadlocks. I fixed that a while back.
>>
>> KVM implements the second option: it maps pages into the stage-2
>> page tables (or shadow page tables, if that is chosen) and unmaps
>> them when the invalidate_range callback is called.
>
> I assume this map and unmap is done by the host as a result of some
> guest action?
Unmapping can happen at any time for any or no reason. Semantically,
it would be correct to only map the pages in response to a p2m violation,
but for performance it might be better to map the pages eagerly instead.
>> Furthermore,
>> if a page fault happens while the page is unmapped, KVM will try
>> to bring the pages back into memory so the guest can access it.
>
> You could likely handle this in Xen in the following way:
>
> - A device model will get p2m violations forwarded, as it's the same
> model that's used to handle emulation of device MMIO. You will
> need to register an ioreq server to request those faults to be
> forwarded, I think the hardware domain kernel will handle those?
>
> - Allow ioreqs to signal to Xen that a guest operation must be
> retried. IOW: resume guest execution without advancing the IP.
>
> I think this last bit is the one that will require changes to Xen, so
> that you can add a type of ioreq reply that implies a retry from the
> guest context.
I’m not actually sure if this is needed, though it would be nice. It
might be possible for Xen to instead emulate the current instruction and
continue, with the ioreq server just returning the current value of the
pages. What I’m more concerned about is being able to provide a page
into the p2m so that the *next* access doesn’t fault, and being able
to remove that page from the p2m so that the next access *does* fault.
Are there any hypercalls that can be used for these operations right
now? If not, which Xen functions would one use to implement them?
Some notes:
- The p2m might need to be made to point to a PCI BAR or system RAM.
The guest kernel and host userspace don’t know which, and in any
case don’t need to care. The host kernel knows, but I don’t know
if the information is exposed to the Xen driver.
- If the p2m needs to point to system RAM, the RAM will be memory
that belongs to the backend.
- If the p2m needs to point to a PCI BAR, it will initially need
to point to a real PCI device that is owned by the backend.
- The switch from “emulated MMIO” to “MMIO or real RAM” needs to
be atomic from the guest’s perspective.
>> For GPU acceleration via virtio-GPU native contexts to work,
>> the Xen interface driver needs to do the same thing with GPU
>> buffers that KVM does: it needs to fault the pages into guest
>> memory on-demand and revoke access to the pages when the host
>> kernel demands them back. There really is no alternative that
>> I am aware of. The need to handle guest page faults doesn’t
>> come from the host kernel, but rather from guest userspace.
>
> I'm a bit confused with this last sentence, the "page faults"
> mentioned here are p2m violations I think?
Yes, they are.
> Hope this makes some sense.
It does.
--
Sincerely,
Demi Marie Obenour (she/her/hers)Attachment:
OpenPGP_0xB288B55FFF9C22C1.asc Attachment:
OpenPGP_signature.asc
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |