Re: [PATCH] PCI/MSI: Fix masking MSI/MSI-X on Xen PV

[David Woodhouse <dwmw2@xxxxxxxxxxxxx>]

On Mon, 2021-10-25 at 14:58 +0200, Roger Pau Monné wrote:
> On Mon, Oct 25, 2021 at 12:53:31PM +0100, David Woodhouse wrote:
> > On Mon, 2021-10-25 at 13:43 +0200, Roger Pau Monné wrote:
> > > It's kind of optional for HVM guests, as it depends on
> > > XENFEAT_hvm_pirqs, which sadly gets unconditionally set for HVM
> > > guests, thus dropping any benefits from having hardware assisted APIC
> > > virtualization or posted interrupts support.
> > 
> > Indeed. After implementing PIRQ support for Xen guests running under
> > KVM, I spent a "happy" couple of days swearing at it because it
> > actually *worked* if something would just *unmask* the f***ing MSI, but
> > the guest inexplicably (to me) didn't do that.
> > 
> > Took me a while to work out that Xen itself is *snooping* on the MSI
> > table writes even while they are *masked*, to capture the magic MSI
> > message (with vector==0) which means it's actually a PIRQ# in the
> > destination ID bits, and then magically unmask the MSI when the guest
> > binds that PIRQ to an event channel.
> > 
> > I did not enjoy implementing that part.
> 
> I can see that. It's even better because none of this is actually
> documented.

Indeed. I still haven't worked out if/how Xen actually *masks* the
corresponding MSI-X again. It can't do so when the evtchn is masked,
since that's just a bit in the shinfo page. So while the evtchn is
masked, the MSI can still be screaming into the void?

Perhaps it does so when the PIRQ is unbound from the evtchn?

> > FWIW the *guest* could potentlaly be smarter here and elect not to use
> > PIRQs when hardware assisted vAPIC is present. Aren't there some bits
> > in the CPUID that Xen advertises, which indicate that? 
> 
> Yes, it's in leaf 0x40000x04. FWIW, I would also be fine with removing
> XENFEAT_hvm_pirqs, as I don't think diverging from the hardware
> specifications gives us much benefit. We avoid a couple of vm exits
> for sure, but the cost of having all those modifications in guest
> OSes is not worth it.

These days with posted interrupts, it doesn't even save us any vmexits;
it's all that additional guest complexity just to give us *more*
vmexits than we would have had :)

Attachment: smime.p7s
Description: S/MIME cryptographic signature