Re: [Xen-devel] [PATCH RFC] pass-through: sync pir to irr after msix vector been updated

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Fri, Sep 13, 2019 at 09:50:34AM -0700, Joe Jin wrote:
> On 9/13/19 3:33 AM, Roger Pau Monné wrote:
> > On Thu, Sep 12, 2019 at 11:03:14AM -0700, Joe Jin wrote:
> >> With below testcase, guest kernel reported "No irq handler for vector":
> >>   1). Passthrough mlx ib VF to 2 pvhvm guests.
> >>   2). Start rds-stress between 2 guests.
> >>   3). Scale down 2 guests vcpu from 32 to 6 at the same time.
> >>
> >> Repeat above test several iteration, guest kernel reported "No irq handler
> >> for vector", and IB traffic downed to zero which caused by interrupt lost.
> >>
> >> When vcpu offline, kernel disabled local IRQ, migrate IRQ to other cpu,
> >> update MSI-X table, enable IRQ. If any new interrupt arrived after
> >> local IRQ disabled also before MSI-X table been updated, interrupt still 
> >> used old vector and dest cpu info, and when local IRQ enabled again, 
> >> interrupt been sent to wrong cpu and vector.
> > 
> > Yes, but that's something Linux shoulkd be able to handle, according
> > to your description there's a window where interrupts can be delivered
> > to the old CPU, but that's something expected.
> 
> Actually, kernel will check APIC IRR when do migration, if any pending
> IRQ, will retrigger IRQ to new destination, but Xen does not set the
> bit.

Right, because the interrupt is pending in the PIRR posted descriptor
field, it has not yet reached the vlapic IRR.

> > 
> >>
> >> Looks sync PIR to IRR after MSI-X been updated is help for this issue.
> > 
> > AFAICT the sync that you do is still using the old vcpu id, as
> > pirq_dpci->gmsi.dest_vcpu_id gets updated a little bit below. I'm
> > unsure about why does this help, I would expect the sync between pir
> > and irr to happen anyway, and hence I'm not sure why is this helping.
> 
> As my above update, IRQ retriggered by old cpu, so Xen need to set IRR
> for old cpu but not of new.

AFAICT you are draining any pending data from the posted interrupt
PIRR field into the IRR vlapic field, so that no stale interrupts are
left behind after the MSI fields have been updated by the guest. I
think this is correct, I wonder however whether you also need to
trigger a vcpu re-scheduling (pause/unpause the vpcu), so that pending
interrupts in IRR are injected by vmx_intr_assist.

Also, I think you should do this syncing after the pi_update_irte
call, or else there's still a window (albeit small) where you can
still get posted interrupts delivered to the old vcpu.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel