Re: Design session "MSI-X support with Linux stubdomain" notes

[Jan Beulich <jbeulich@xxxxxxxx>]

On 26.09.2022 14:43, Marek Marczykowski-Górecki wrote:
> On Thu, Sep 22, 2022 at 08:00:00PM +0200, Jan Beulich wrote:
>> On 22.09.2022 18:05, Anthony PERARD wrote:
>>> WARNING: Notes missing at the beginning of the meeting.
>>>
>>> session description:
>>>> Currently a HVM with PCI passthrough and Qemu Linux stubdomain doesn’t
>>>> support MSI-X. For the device to (partially) work, Qemu needs a patch 
>>>> masking
>>>> MSI-X from the PCI config space. Some drivers are not happy about that, 
>>>> which
>>>> is understandable (device natively supports MSI-X, so fallback path are
>>>> rarely tested).
>>>>
>>>> This is mostly (?) about qemu accessing /dev/mem directly (here:
>>>> https://github.com/qemu/qemu/blob/master/hw/xen/xen_pt_msi.c#L579) - lets
>>>> discuss alternative interface that stubdomain could use.
>>>
>>>
>>>
>>> when qemu forward interrupt,
>>>     for correct mask bit, it read physical mask bit.
>>>     an hypercall would make sense.
>>>     -> benefit, mask bit in hardware will be what hypervisor desire, and 
>>> device model desire.
>>>     from guest point of view, interrupt should be unmask.
>>>
>>> interrupt request are first forwarded to qemu, so xen have to do some post 
>>> processing once request comes back from qemu.
>>>     it's weird..
>>>
>>> someone should have a look, and rationalize this weird path.
>>>
>>> Xen tries to not forward everything to qemu.
>>>
>>> why don't we do that in xen.
>>>     there's already code in xen for that.
>>
>> So what I didn't pay enough attention to when talking was that the
>> completion logic in Xen is for writes only. Maybe something similar
>> can be had for reads as well, but that's to be checked ...
> 
> I spent some time trying to follow that part of qemu, and I think it
> reads vector control only on the write path, to keep some bits
> unchanged, and also detect whether Xen masked it behind qemu's back.
> My understanding is, since 484d7c852e "x86/MSI-X: track host and guest
> mask-all requests separately" it is unnecessary, because Xen will
> remember guest's intention, so qemu can simply use its own internal
> state and act on that (guest writes will go through qemu, so it should
> have up to date view from guest's point of view).
> 
> As for PBA access, it is read by qemu only to pass it to the guest. I'm
> not sure whether qemu should use hypercall to retrieve it, or maybe
> Xen should fixup value itself on the read path.

Forwarding the access to qemu just for qemu to use a hypercall to obtain
the value needed seems backwards to me. If we need new code in Xen, we
can as well handle the read directly I think, without involving qemu.

Jan

> I did some preliminary patch here:
> https://github.com/marmarek/qubes-vmm-xen-stubdom-linux/commit/80cf769f3659aa0d7f2b5598bf862d83da28807e
> 
> but it does not work yet. It seems I haven't undo MSI-X hiding enough
> (lspci inside the guest doesn't report MSI-X at all). This I will figure
> out, but I'd appreciate comments about how to handle PBA best.
>