Re: PCI devices passthrough on Arm design proposal

[Rahul Singh <Rahul.Singh@xxxxxxx>]

> On 18 Jul 2020, at 12:14 pm, Julien Grall <julien@xxxxxxx> wrote:
> 
> 
> 
> On 18/07/2020 10:55, Bertrand Marquis wrote:
>>> On 17 Jul 2020, at 18:05, Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
>>> 
>>> On Fri, Jul 17, 2020 at 03:47:25PM +0000, Bertrand Marquis wrote:
>>>>> On 17 Jul 2020, at 17:26, Julien Grall <julien@xxxxxxx> wrote:
>>>>> On 17/07/2020 15:47, Bertrand Marquis wrote:
>>>>>>>>> * Dom0Less implementation will require to have the capacity inside 
>>>>>>>>> Xen to discover the PCI devices (without depending on Dom0 to declare 
>>>>>>>>> them to Xen).
>>>>>>>>> 
>>>>>>>>> # Enable the existing x86 virtual PCI support for ARM:
>>>>>>>>> 
>>>>>>>>> The existing VPCI support available for X86 is adapted for Arm. When 
>>>>>>>>> the device is added to XEN via the hyper call 
>>>>>>>>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space access 
>>>>>>>>> is added to the PCI device to emulate the PCI devices.
>>>>>>>>> 
>>>>>>>>> A MMIO trap handler for the PCI ECAM space is registered in XEN so 
>>>>>>>>> that when guest is trying to access the PCI config space, XEN will 
>>>>>>>>> trap the access and emulate read/write using the VPCI and not the 
>>>>>>>>> real PCI hardware.
>>>>>>>>> 
>>>>>>>>> Limitation:
>>>>>>>>> * No handler is register for the MSI configuration.
>>>>>>>>> * Only legacy interrupt is supported and tested as of now, MSI is not 
>>>>>>>>> implemented and tested.
>>>>>>>> IIRC, legacy interrupt may be shared between two PCI devices. How do 
>>>>>>>> you plan to handle this on Arm?
>>>>>> We plan to fix this by adding proper support for MSI in the long term.
>>>>>> For the use case where MSI is not supported or not wanted we might have 
>>>>>> to find a way to forward the hardware interrupt to several guests to 
>>>>>> emulate some kind of shared interrupt.
>>>>> 
>>>>> Sharing interrupts are a bit pain because you couldn't take advantage of 
>>>>> the direct EOI in HW and have to be careful if one guest doesn't EOI in 
>>>>> timely maneer.
>>>>> 
>>>>> This is something I would rather avoid unless there is a real use case 
>>>>> for it.
>>>> 
>>>> I would expect that most recent hardware will support MSI and this
>>>> will not be needed.
>>> 
>>> Well, PCI Express mandates MSI support, so while this is just a spec,
>>> I would expect most (if not all) devices to support MSI (or MSI-X), as
>>> Arm platforms haven't implemented legacy PCI anyway.
>> Yes that’s our assumption to. But we have to start somewhere so MSI is
>> planned but in a future step. I would think that supporting non MSI if not
>> impossible will be a lot more complex due to the interrupt sharing.
>> I do think that not supporting non MSI should be ok on Arm.
>>> 
>>>> When MSI is not used, the only solution would be to enforce that
>>>> devices assigned to different guest are using different interrupts
>>>> which would limit the number of domains being able to use PCI
>>>> devices on a bus to 4 (if the enumeration can be modified correctly
>>>> to assign the interrupts properly).
>>>> 
>>>> If we all agree that this is an acceptable limitation then we would
>>>> not need the “interrupt sharing”.
>>> 
>>> I might be easier to start by just supporting devices that have MSI
>>> (or MSI-X) and then move to legacy interrupts if required?
>> MSI support requires also some support in the interrupt controller part
>> on arm. So there is some work to achieve that.
>>> 
>>> You should have most of the pieces you require already implemented
>>> since that's what x86 uses, and hence could reuse almost all of it?
>> Inside PCI probably but the GIC part will require some work.
> 
> We already have an ITS implementation in Xen. This is required in order to 
> use PCI devices in DOM0 on thunder-x (there is no legacy interrupts 
> supported).
> 
> It wasn't yet exposed to the guest because we didn't fully investigate the 
> security aspect of the implementation. However, for a tech preview this 
> should be sufficient.
> 

Ok We will have a look for the ITS implementation once we will start working on 
the MSI support. Thanks for the pointer.
> 
> -- 
> Julien Grall
>