Re: Pinned, non-revocable mappings of VRAM: will bad things happen?

[Christian König <christian.koenig@xxxxxxx>]

On 4/17/26 21:35, Demi Marie Obenour wrote:
> On 4/17/26 03:53, Christian König wrote:
>> On 4/16/26 18:13, Demi Marie Obenour wrote:
>>> On 4/16/26 05:57, Christian König wrote:
...
>>> Unless I am
>>> mistaken, client workloads are highly non-uniform: a single game or
>>> compute job might well use more VRAM than every other program on the
>>> system combined.
>>
>> Yeah, perfectly correct.
>>
>>> Are these workloads impossible to make work well with pinning?
>>
>> No, as long as you don't know the workload beforehand, e.g. when you define 
>> the limit.
>>
>> I mean that's why basically everybody avoids pinning and assigning fixed 
>> amounts of resources.
>>
>> Even if you can make it work technically pinning usually results in a rather 
>> bad end user experience.
>>
>> Regards,
>> Christian.
> 
> Do drivers and programs assume that they can access VRAM from the CPU?

Yes, and that is actually really important for performance.

That's why Alex and I came up with the idea of using the resize able BAR 
feature to access all of VRAM on modern GPUs.

There are a couple of hacks which have been implemented over the years for 
exotic platforms were MMIO/VRAM access was problematic. For example on a page 
fault you use a GPU DMA engine to copy the VRAM buffer into system memory, make 
the CPU memory access and then copy it back again on demand at the next command 
submission.

But all of those hacks are basically just prove of concepts and result in 
completely unusable performance.

> Are any of the following reasonable options?
> 
> 1. Change the guest kernel to only map (and thus pin) a small subset
>    of VRAM at any given time.  If unmapped VRAM is accessed the guest
>    traps the page fault, evicts an old VRAM mapping, and creates a
>    new one.

Yeah, that could potentially work.

This is basically what we do on the host kernel driver when we can't resize the 
BAR for some reason. In that use case VRAM buffers are shuffled in and out of 
the CPU accessible window of VRAM on demand.

> 2. Pretend that resizable BAR is not enabled, so the guest doesn't
>    think it can map much of VRAM at once.  If resizable BAR is enabled
>    on the host, it might be possible to split the large BAR mapping
>    in a lot of ways.

That won't work. The userspace parts of the driver stack don't care how large 
the BAR to access VRAM with the CPU is.

The expectation is that the kernel driver makes thing CPU accessible as needed 
in the page fault handler.

It is still a good idea for your solution #1 to give the amount of "pin-able" 
VRAM to the userspace stack as CPU visible VRAM limit so that test cases and 
applications try to lower their usage of VRAM, e.g. use system memory bounce 
buffers when possible.

> Or does Xen really need to allow the host to handle guest page faults?
> That adds a huge amount of complexity to trusted and security-critical
> parts of the system, so it really is a last resort.  Putting the
> complexity in to the guest virtio-GPU driver is vastly preferable if
> it can be made to work well.

Well the nested page fault handling KVM offers has proven to be extremely 
useful. So when XEN can't do this it is clearly lacking an important feature.

But I have one question: When XEN has a problem handling faults from the guest 
on the host then how does that work for system memory mappings?

There is really no difference between VRAM and system memory in the handling 
for the GPU driver stack.

Regards,
Christian.