Re: [PATCH] PCI/MSI: Fix masking MSI/MSI-X on Xen PV

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Mon, Oct 25, 2021 at 02:02:38PM +0100, David Woodhouse wrote:
> On Mon, 2021-10-25 at 14:58 +0200, Roger Pau Monné wrote:
> > On Mon, Oct 25, 2021 at 12:53:31PM +0100, David Woodhouse wrote:
> > > On Mon, 2021-10-25 at 13:43 +0200, Roger Pau Monné wrote:
> > > > It's kind of optional for HVM guests, as it depends on
> > > > XENFEAT_hvm_pirqs, which sadly gets unconditionally set for HVM
> > > > guests, thus dropping any benefits from having hardware assisted APIC
> > > > virtualization or posted interrupts support.
> > > 
> > > Indeed. After implementing PIRQ support for Xen guests running under
> > > KVM, I spent a "happy" couple of days swearing at it because it
> > > actually *worked* if something would just *unmask* the f***ing MSI, but
> > > the guest inexplicably (to me) didn't do that.
> > > 
> > > Took me a while to work out that Xen itself is *snooping* on the MSI
> > > table writes even while they are *masked*, to capture the magic MSI
> > > message (with vector==0) which means it's actually a PIRQ# in the
> > > destination ID bits, and then magically unmask the MSI when the guest
> > > binds that PIRQ to an event channel.
> > > 
> > > I did not enjoy implementing that part.
> > 
> > I can see that. It's even better because none of this is actually
> > documented.
> 
> Indeed. I still haven't worked out if/how Xen actually *masks* the
> corresponding MSI-X again. It can't do so when the evtchn is masked,
> since that's just a bit in the shinfo page. So while the evtchn is
> masked, the MSI can still be screaming into the void?

I think so, it's quite weird because as a side effect of this mangling
Xen is transforming an edge triggered interrupt to a level triggered
one, as masked event channels belonging to MSI vectors will get set to
pending.

So it's not entirely screaming into the void because it will get
(wrongly) set as pending when masked.

> Perhaps it does so when the PIRQ is unbound from the evtchn?
> 
> > > FWIW the *guest* could potentlaly be smarter here and elect not to use
> > > PIRQs when hardware assisted vAPIC is present. Aren't there some bits
> > > in the CPUID that Xen advertises, which indicate that? 
> > 
> > Yes, it's in leaf 0x40000x04. FWIW, I would also be fine with removing
> > XENFEAT_hvm_pirqs, as I don't think diverging from the hardware
> > specifications gives us much benefit. We avoid a couple of vm exits
> > for sure, but the cost of having all those modifications in guest
> > OSes is not worth it.
> 
> These days with posted interrupts, it doesn't even save us any vmexits;
> it's all that additional guest complexity just to give us *more*
> vmexits than we would have had :)

Oh indeed. I was thinking about hardware without APIC hw
virtualization or posted interrupts.

Roger.