Re: [PATCH] x86/hvm: don't expose XENFEAT_hvm_pirqs by default

[Xenia Ragiadakou <xenia.ragiadakou@xxxxxxx>]

Hi Roger,

On 10/1/24 17:21, Roger Pau Monné wrote:
> On Wed, Jan 10, 2024 at 03:47:12PM +0200, Xenia Ragiadakou wrote:
> > On 10/1/24 12:26, Jan Beulich wrote:
> > > On 10.01.2024 10:53, Roger Pau Monne wrote:
> > > > The HVM pirq feature allows routing interrupts from both physical and emulated
> > > > devices over event channels, this was done a performance improvement.  However
> > > > its usage is fully undocumented, and the only reference implementation is in
> > > > Linux.  It defeats the purpose of local APIC hardware virtualization, because
> > > > when using it interrupts avoid the usage of the local APIC altogether.
> > > So without sufficient APIC acceleration, isn't this arranging for degraded
> > > performance then? IOW should the new default perhaps be dependent on the
> > > degree of APIC acceleration?
> > >
> > > > It has also been reported to not work properly with certain devices, at least
> > > > when using some AMD GPUs Linux attempts to route interrupts over event
> > > > channels, but Xen doesn't correctly detect such routing, which leads to the
> > > > hypervisor complaining with:
> > > >
> > > > (XEN) d15v0: Unsupported MSI delivery mode 7 for Dom15
> > > >
> > > > When MSIs are attempted to be routed over event channels the entry delivery
> > > > mode is set to ExtINT, but Xen doesn't detect such routing and attempts to
> > > > inject the interrupt following the native MSI path, and the ExtINT delivery
> > > > mode is not supported.
> > > Shouldn't this be properly addressed nevertheless? The way it's described
> > > it sounds as if MSI wouldn't work at all this way; I can't spot why the
> > > issue would only be "with certain devices". Yet that in turn doesn't look
> > > to be very likely - pass-through use cases, in particular SR-IOV ones,
> > > would certainly have noticed.
> > The issue gets triggered when the guest performs save/restore of MSIs,
> > because PHYSDEVOP_map_pirq is not implemented for MSIs, and thus, QEMU
> > cannot remap the MSI to the event channel once unmapped.
> I'm kind of confused by this sentence, PHYSDEVOP_map_pirq does support
> MSIs, see xc_physdev_map_pirq_msi() helper in Xen code base.
Sorry I had to explain it better. For an HVM guest with 
XENFEAT_hvm_pirqs set, physdev_hvm_map_pirq() will be called, that has 
not support for MSI.
> > So, to fix this issue either would be needed to change QEMU to not unmap
> > pirq-emulated MSIs or to implement PHYSDEVOP_map_pirq for MSIs.
> >
> > But still, even when no device has been passed-through, scheduling latencies
> > (of hundreds of ms), were observed in the guest even when running a simple
> > loop application, that disappear once the flag is disabled. We did not have
> > the chance to root cause it further.
> So XENFEAT_hvm_pirqs is causing such latency issues?  That I certainly
> didn't notice.
We 've seen that in a setup with an HVM guest using emulated MSIs. We 
were running an application, like the one below, pinned on an isolated 
and pinned vcpu.
int main()
{
    struct timeval cur_time = {0}, prev_time = {0};
    int diff;

    gettimeofday(&prev_time, NULL);
    do {
        gettimeofday(&cur_time, NULL);

        diff = (((cur_time.tv_sec - prev_time.tv_sec)*1000) + ((cur_time
.tv_usec - prev_time.tv_usec)/1000));
        if (diff>10)
           printf("app scheduled after: %d msec\n",diff);

        gettimeofday(&prev_time, NULL);
    } while(1);
}

And we 're getting values of hundreds of ms like:
app scheduled after: 985 msec

> Regards, Roger.