Re: [Xen-devel] [RFC] ARM PCI Passthrough design document

["Vikram Sethi" <vikrams@xxxxxxxxxxxxxx>]

Hi Roger,
Thanks for your comments. My responses inline.


> -----Original Message-----
> From: Xen-devel [mailto:xen-devel-bounces@xxxxxxxxxxxxx] On Behalf Of
> roger.pau@xxxxxxxxxx
> Sent: Tuesday, July 4, 2017 3:31 AM
> To: Julien Grall <julien.grall@xxxxxxxxxx>
> Cc: edgar.iglesias@xxxxxxxxxx; Stefano Stabellini <sstabellini@xxxxxxxxxx>; 
> Wei
> Chen <Wei.Chen@xxxxxxx>; Steve Capper <Steve.Capper@xxxxxxx>; Andre
> Przywara <andre.przywara@xxxxxxx>; manish.jaggi@xxxxxxxxxxxxxxxxxx;
> punit.agrawal@xxxxxxx; Vikram Sethi <vikrams@xxxxxxxxxxxxxxxx>; Sinan
> Kaya <okaya@xxxxxxxxxxxxxxxx>; Sameer Goel <sgoel@xxxxxxxxxxxxxxxx>;
> xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>; Dave P Martin
> <Dave.Martin@xxxxxxx>; Vijaya Kumar K
> <Vijaya.Kumar@xxxxxxxxxxxxxxxxxx>
> Subject: Re: [Xen-devel] [RFC] ARM PCI Passthrough design document
>
> Hello,
>
> My 2cents on what are the plans on PVH/x86.
>
> On Wed, Jun 28, 2017 at 04:22:48PM +0100, Julien Grall wrote:
> >
> >
> > On 20/06/17 01:19, Vikram Sethi wrote:
> > > Hi Julien,
> >
> > Hi Vikram,
> >
> > Thank you for your feedbacks.
> >
> > > Thanks for posting this. I think some additional topics need to be 
> > > covered 
> > > in
> the design document, under 3 main topics:
> >
> > I wanted to limit the scope of the PCI passthrough work to the strict
> > minimum. I didn't consider hotplug and AER in the scope because it is
> > optional feature.
> >
> > >
> > > Hotplug: how will Xen support hotplug? Many rootports may require
> firmware hooks such as ACPI ASL to take care of platform specific MMIO
> initialization on hotplug. Normally firmware (UEFI) would have done that
> platform specific setup at boot.
> >
> > We don't have ASL support in Xen. So I would expect the hotplug to be
> > handled by the hardware domain and then report it to Xen.
> >
> > This would also fit quite well to the current design as the hardware
> > domain will scan PCI devices at boot and then register them to Xen via
> > an hypercall.
>
> Hotplug will be done using an hypercall. We already have them in place for PV,
> and this is simply going to be reused:
>
> Hotplug PCI devices:
> PHYSDEVOP_manage_pci_add{_ext}
>
> hotplug MMCFG (ECAM) regions:
> PHYSDEVOP_pci_mmcfg_reserved
>
> > >
> > > AER: Will PCIe non-fatal and fatal errors (secondary bus reset for fatal) 
> > > be
> recoverable in Xen?
> > > Will drivers in doms be notified about fatal errors so they can be 
> > > quiesced
> before doing secondary bus reset in Xen?
> > > Will Xen support Firmware First Error handling for AER? i.e When
> > > platform does Firmware first error handling for AER and/or filtering of 
> > > AER,
> sends associated ACPI HEST logs to Xen How will AER notification and logs be
> propagated to the doms: injected ACPI HEST?
>
> Hm, I'm not sure I follow here, I don't see AER tied to ACPI. AER is a PCIe
> capability, and according to the spec can be setup completely independent to
> ACPI.
>
True, it can be independent if not using firmware first AER handling (FFH). But 
Firmware tells the OS whether firmware first is in use.
If FFH is in use, the AER interrupt goes to firmware and then firmware 
processes 
the AER logs, filters errors, and sends a ACPI HEST log with the filtered AER 
regs to OS along with an ACPI event/interrupt. Kernel is not supposed to touch 
the AER registers directly in this case, but act on the register values in the 
HEST log.
http://elixir.free-electrons.com/linux/latest/source/drivers/pci/pcie/aer/aerdrv_acpi.c#L94
If Firmware is using FFH, Xen will get a HEST log with AER registers, and must 
parse those registers instead of reading AER config space.
After the AER registers have been parsed (either from HEST log or native Xen 
AER 
interrupt handler), at least for fatal errors, Xen needs to send notification 
to 
the DOM with the device passthrough so that it's driver(s) can be quiesced (via 
callbacks to dev->driver->err_handler->error_detected for linux) before hot 
reset/secondary bus reset.

Whether FFH is in use or not, Xen has 2 choices in how to present the error to 
doms for quiescing before secondary bus reset:
a. Send a HEST log and ACPI interrupt/event to dom if it booted ACPI dom and 
linux dom calls aer_recover_queue from ACPI ghes path 
http://elixir.free-electrons.com/linux/latest/source/drivers/pci/pcie/aer/aerdrv_core.c#L592b.
 Present a Root port wired interrupt source in dom ACPI/DT, and inject that 
irq in the GIC LR registers. When dom kernel processes the interrupt and 
queries 
config space AER, Xen emulates the AER values it wants the dom to see (in FFH 
case based on register values in HEST), and if FFH was in use, not actually 
allow the dom to clear out the AER registers.

Option b is probably better/easier since it works for ACPI/DT dom.

In my view this is the basic AER error handling leaving the devices 
inaccessible.
To recover/resume the devices, the owning dom would need to signal Xen once all 
its driver(s) have quiesced, letting Xen know it is ok to do the secondary bus 
reset (for AER fatal errors). The best way to signal this would be to let the 
dom try to hit SBR in the Root port bridge control register in config space, 
and 
Xen traps that and actually does the BCR.SBR write.

Since Xen controls the ECAM config space access in Julien's proposed design, I 
don't see any fundamental issues with the above flow fitting into the design.

> In any case, Xen can trap or hide the capability from guests, Xen could 
> possibly
> even emulate AER somehow if that's more suitable (ie:
> guest sets up AER, Xen traps accesses to this capability and filters the 
> errors
> Xen wants to handle itself vs the errors that should be propagated to the
> guest).
>
> The biggest issue I see with AER (and DPC) is that it requires an interrupt. 
> So
> Xen would have to stole one (or more) interrupts from the guest in order to
> make use of those capabilities if they are to be exclusively managed by Xen.
> This could be done by simply telling the guest the device has less MSI/MSI-X
> interrupts than it really has.
>
> > > PCIe DPC (Downstream Port Containment): will it be supported in Xen, and
> Xen will register for DPC interrupt? When Xen brings the link back up will it 
> send
> a simulated hotplug to dom0 to show link back up?
> >
> > I don't feel it is necessary to look at AER for the first work of PCI
> > passthrough. I consider it as a separate feature that could probably
> > come with the RAS story.
> >
> > At the moment, I don't know who is going to handle the error and even
> > how they will be reported to the guest. But I don't think this will
> > have any impact on our design choice here.
> >
> > Let me know if you think it may have an impact.
>
> As Julien said, I think that you probably know more about AER/DPC than we do,
> so it would be good if you could go over the design document and mare sure
> that the current approach can work with the way you intend to use AER/DPC.
>

I think what I wrote above supplements the design, and I don't see any 
fundamental issue.
Let me know if you have any questions or concerns with proposed flow.

> Thanks, Roger.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> https://lists.xen.org/xen-devel


Thanks,
Vikram
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm 
Technologies, 
Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux 
Foundation Collaborative Project.



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel