Re: [PATCH v3 06/11] x86/hvm: allowing registering EOI callbacks for GSIs

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Wed, Apr 07, 2021 at 05:51:14PM +0200, Jan Beulich wrote:
> On 31.03.2021 12:32, Roger Pau Monne wrote:
> > --- a/xen/arch/x86/hvm/irq.c
> > +++ b/xen/arch/x86/hvm/irq.c
> > +void hvm_gsi_execute_callbacks(unsigned int gsi)
> > +{
> > +    struct hvm_irq *hvm_irq = hvm_domain_irq(current->domain);
> > +    struct hvm_gsi_eoi_callback *cb;
> > +
> > +    read_lock(&hvm_irq->gsi_callbacks_lock);
> > +    list_for_each_entry ( cb, &hvm_irq->gsi_callbacks[gsi], list )
> > +        cb->callback(gsi, cb->data);
> > +    read_unlock(&hvm_irq->gsi_callbacks_lock);
> > +}
> 
> Just as an observation (for now at least) - holding the lock here
> means the callbacks cannot re-register themselves.

Well, re-registering would be weird, as the callback is not
unregistered after execution. What is likely more relevant is that the
callback cannot unregister itself. I haven't found a need for this so
far, so I think it's fine.

> > +bool hvm_gsi_has_callbacks(const struct domain *d, unsigned int gsi)
> > +{
> > +    struct hvm_irq *hvm_irq = hvm_domain_irq(d);
> > +    bool has_callbacks;
> > +
> > +    read_lock(&hvm_irq->gsi_callbacks_lock);
> > +    has_callbacks = !list_empty(&hvm_irq->gsi_callbacks[gsi]);
> > +    read_unlock(&hvm_irq->gsi_callbacks_lock);
> > +
> > +    return has_callbacks;
> > +}
> 
> What use is this function? Its result is stale by the time the
> caller can look at it, as you've dropped the lock.

Right, that function is only used to decide whether the vIOAPIC needs
to register an EOI callback when injecting a vector to the vlapic. The
workflow is to first register a callback with the vIOAPIC and
afterwards inject an interrupt which will trigger the callback
logic.

Playing with the callback registration while interrupts can be
injected will likely result in a malfunction of the device that relies
on those callbacks, but that's to be expected anyway when playing such
games.

That said multiple users sharing a vIOAPIC pin should be fine as long
as they follow the logic above: always register a callback before
attempting to inject an interrupt.

> > @@ -421,13 +423,25 @@ static void eoi_callback(unsigned int vector, void 
> > *data)
> >              if ( is_iommu_enabled(d) )
> >              {
> >                  spin_unlock(&d->arch.hvm.irq_lock);
> > -                hvm_dpci_eoi(vioapic->base_gsi + pin);
> > +                hvm_dpci_eoi(gsi);
> >                  spin_lock(&d->arch.hvm.irq_lock);
> >              }
> >  
> > +            /*
> > +             * Callbacks don't expect to be executed with any lock held, so
> > +             * drop the lock that protects the vIO-APIC fields from 
> > changing.
> > +             *
> > +             * Note that the redirection entry itself cannot go away, so 
> > upon
> > +             * retaking the lock we only need to avoid making assumptions 
> > on
> > +             * redirection entry field values (ie: recheck the IRR field).
> > +             */
> > +            spin_unlock(&d->arch.hvm.irq_lock);
> > +            hvm_gsi_execute_callbacks(gsi);
> > +            spin_lock(&d->arch.hvm.irq_lock);
> 
> The two pairs of unlock / re-lock want folding, I think - there's
> no point causing extra contention on the lock here.

The chunk above will go away on the next patch - there's no need to
fold it as it makes the following patch less clear.

> > @@ -443,7 +457,8 @@ static void ioapic_inj_irq(
> >      struct vlapic *target,
> >      uint8_t vector,
> >      uint8_t trig_mode,
> > -    uint8_t delivery_mode)
> > +    uint8_t delivery_mode,
> > +    bool callback)
> >  {
> >      HVM_DBG_LOG(DBG_LEVEL_IOAPIC, "irq %d trig %d deliv %d",
> >                  vector, trig_mode, delivery_mode);
> > @@ -452,7 +467,7 @@ static void ioapic_inj_irq(
> >             (delivery_mode == dest_LowestPrio));
> >  
> >      vlapic_set_irq_callback(target, vector, trig_mode,
> > -                            trig_mode ? eoi_callback : NULL, NULL);
> > +                            callback ? eoi_callback : NULL, NULL);
> 
> I think you'd better use trig_mode || callback here and ...
> 
> > @@ -466,6 +481,7 @@ static void vioapic_deliver(struct hvm_vioapic 
> > *vioapic, unsigned int pin)
> >      struct vlapic *target;
> >      struct vcpu *v;
> >      unsigned int irq = vioapic->base_gsi + pin;
> > +    bool callback = trig_mode || hvm_gsi_has_callbacks(d, irq);
> >  
> >      ASSERT(spin_is_locked(&d->arch.hvm.irq_lock));
> >  
> > @@ -492,7 +508,8 @@ static void vioapic_deliver(struct hvm_vioapic 
> > *vioapic, unsigned int pin)
> >              target = vlapic_lowest_prio(d, NULL, 0, dest, dest_mode);
> >          if ( target != NULL )
> >          {
> > -            ioapic_inj_irq(vioapic, target, vector, trig_mode, 
> > delivery_mode);
> > +            ioapic_inj_irq(vioapic, target, vector, trig_mode, 
> > delivery_mode,
> > +                           callback);
> 
> ... invoke hvm_gsi_has_callbacks() right here and ...
> 
> > @@ -507,7 +524,7 @@ static void vioapic_deliver(struct hvm_vioapic 
> > *vioapic, unsigned int pin)
> >          for_each_vcpu ( d, v )
> >              if ( vlapic_match_dest(vcpu_vlapic(v), NULL, 0, dest, 
> > dest_mode) )
> >                  ioapic_inj_irq(vioapic, vcpu_vlapic(v), vector, trig_mode,
> > -                               delivery_mode);
> > +                               delivery_mode, callback);
> 
> ... here, avoiding to call the function when you don't need the
> result.

I think there's a slim chance of not needing to use the callback local
variable, and hence didn't consider limiting it. I can do, but I'm
unsure this will bring any real benefit while making the code more
complex IMO.

Thanks, Roger.