Re: [Xen-devel] [RFC] ARM PCI Passthrough design document

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Thu, Jul 06, 2017 at 03:55:28PM -0500, Vikram Sethi wrote:
> > > > AER: Will PCIe non-fatal and fatal errors (secondary bus reset for 
> > > > fatal) 
> > > > be
> > recoverable in Xen?
> > > > Will drivers in doms be notified about fatal errors so they can be 
> > > > quiesced
> > before doing secondary bus reset in Xen?
> > > > Will Xen support Firmware First Error handling for AER? i.e When
> > > > platform does Firmware first error handling for AER and/or filtering of 
> > > > AER,
> > sends associated ACPI HEST logs to Xen How will AER notification and logs be
> > propagated to the doms: injected ACPI HEST?
> >
> > Hm, I'm not sure I follow here, I don't see AER tied to ACPI. AER is a PCIe
> > capability, and according to the spec can be setup completely independent to
> > ACPI.
> >
> True, it can be independent if not using firmware first AER handling (FFH). 
> But 
> Firmware tells the OS whether firmware first is in use.
> If FFH is in use, the AER interrupt goes to firmware and then firmware 
> processes 

I'm sorry, but how is the firmware supposed to know which interrupt is
AER using? That's AFAIK setup in the PCI AER capabilities, and
depends on whether the OS configures the device to use MSI or MSI-X.

Is there some kind of side-band mechanism that delivers the AER
interrupt using a different method?

> the AER logs, filters errors, and sends a ACPI HEST log with the filtered AER 
> regs to OS along with an ACPI event/interrupt. Kernel is not supposed to 
> touch 
> the AER registers directly in this case, but act on the register values in 
> the 
> HEST log.
> http://elixir.free-electrons.com/linux/latest/source/drivers/pci/pcie/aer/aerdrv_acpi.c#L94

That's not a problem IMHO, Xen could even mask the AER capability from
the Dom0/guest completely if needed.

> If Firmware is using FFH, Xen will get a HEST log with AER registers, and 
> must 
> parse those registers instead of reading AER config space.

Xen will not get an event, it's going to be delivered to Dom0 because
when using ACPI Dom0 is the OSPM (not Xen). I assume this event is
going to be notified by triggering an interrupt from the ACPI SCI?

> After the AER registers have been parsed (either from HEST log or native Xen 
> AER 
> interrupt handler), at least for fatal errors, Xen needs to send notification 
> to 
> the DOM with the device passthrough so that it's driver(s) can be quiesced 
> (via 
> callbacks to dev->driver->err_handler->error_detected for linux) before hot 
> reset/secondary bus reset.

I don't think this is relevant/true given the statement above (Dom0
being OSPM and receiving the event).

> Whether FFH is in use or not, Xen has 2 choices in how to present the error 
> to 
> doms for quiescing before secondary bus reset:

How is this secondary bus reset performed?

Is it something specific to each bridge or it's a standard
interface?

Can it be done directly by Dom0, or should it be done by Xen?

> a. Send a HEST log and ACPI interrupt/event to dom if it booted ACPI dom and 
> linux dom calls aer_recover_queue from ACPI ghes path 
> http://elixir.free-electrons.com/linux/latest/source/drivers/pci/pcie/aer/aerdrv_core.c#L592b.
>  Present a Root port wired interrupt source in dom ACPI/DT, and inject that 
> irq in the GIC LR registers. When dom kernel processes the interrupt and 
> queries 

You lost me here, I have no knowledge of ARM, and I don't know what
GIC LR is at all.

> config space AER, Xen emulates the AER values it wants the dom to see (in FFH 
> case based on register values in HEST), and if FFH was in use, not actually 
> allow the dom to clear out the AER registers.
> 
> Option b is probably better/easier since it works for ACPI/DT dom.

So as I understand it, the flow is the following:

1. Hardware generates an error.
2. This error triggers an interrupt that's delivered to Dom0 (either
   using an ACPI SCI or a specific AER MSI vector)
3. *Someone* has to do a secondary bus reset.

My question would be, who (either Xen or Dom0) should perform the bus
reset? (and why).

> In my view this is the basic AER error handling leaving the devices 
> inaccessible.
> To recover/resume the devices, the owning dom would need to signal Xen once 
> all 
> its driver(s) have quiesced, letting Xen know it is ok to do the secondary 
> bus 
> reset (for AER fatal errors). The best way to signal this would be to let the 
> dom try to hit SBR in the Root port bridge control register in config space, 
> and 
> Xen traps that and actually does the BCR.SBR write.
>
> Since Xen controls the ECAM config space access in Julien's proposed design, 
> I 
> don't see any fundamental issues with the above flow fitting into the design.

I think it's very hard for me (or Julien) to know exactly how all the
PCI capabilities behave and interact with other components (like
ACPI).

You seem to have a good amount of knowledge about this stuff, would
you mind writing your proposal as a diff to Julien's original
proposal, so that it can be properly reviewed and merged into the
design document?

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel