Re: SR-IOV: do we need to virtualize in Xen or rely on Dom0?

[Jan Beulich <jbeulich@xxxxxxxx>]

On 10.06.2021 17:33, Oleksandr Andrushchenko wrote:
> On 10.06.21 17:10, Roger Pau Monné wrote:
>> On Thu, Jun 10, 2021 at 10:01:16AM +0000, Oleksandr Andrushchenko wrote:
>>> On 10.06.21 10:54, Roger Pau Monné wrote:
>>>> On Fri, Jun 04, 2021 at 06:37:27AM +0000, Oleksandr Andrushchenko wrote:
>>>>> Hi, all!
>>>>>
>>>>> While working on PCI SR-IOV support for ARM I started porting [1] on top
>>>>> of current PCI on ARM support [2]. The question I have for this series
>>>>> is if we really need emulating SR-IOV code in Xen?
>>>>>
>>>>> I have implemented a PoC for SR-IOV on ARM [3] (please see the top 2
>>>>> patches)
>>>>> and it "works for me": MSI support is still WIP, but I was able to see 
>>>>> that
>>>>> VFs are properly seen in the guest and BARs are properly programmed in 
>>>>> p2m.
>>>>>
>>>>> What I can't fully understand is if we can live with this approach or 
>>>>> there
>>>>> are use-cases I can't see.
>>>>>
>>>>> Previously I've been told that this approach might not work on FreeBSD
>>>>> running
>>>>> as Domain-0, but is seems that "PCI Passthrough is not supported
>>>>> (Xen/FreeBSD)"
>>>>> anyways [4].
>>>> PCI passthorgh is not supported on FreeBSD dom0 because PCI
>>>> passthrough is not supported by Xen itself when using a PVH dom0, and
>>>> that's the only mode FreeBSD dom0 can use.
>>> So, it is still not clear to me: how and if PCI passthrough is supported
>>>
>>> on FreeBSD, what are the scenarios and requirements for that?
>>>
>>>> PHYSDEVOP_pci_device_add can be added to FreeBSD, so it could be made
>>>> to work. I however think this is not the proper way to implement
>>>> SR-IOV support.
>>> I was not able to find any support for PHYSDEVOP_XXX in FreeBSD code,
>>>
>>> could you please point me to where are these used?
>> Those are not used on FreeBSD, because x86 PVHv2 dom0 doesn't
>> implement them anymore. They are implemented on Linux for x86 PV dom0,
>> AFAIK Arm doesn't use them either.
> 
> Well, ARM didn't until we started implementing PCI passthrough [1].
> 
> It was previously discussed [2], "# Discovering PCI devices:" and proposed
> 
> to use PHYSDEVOP_pci_device_add.
> 
> Long story short, it is not easy for ARM to enumerate PCI devices in Xen as 
> there is
> 
> no unified way of doing so: different platforms implement different PCI host 
> bridges
> 
> which require complex initialization including clocks, power domains etc.

Just for my own understanding: If this isn't done by firmware, doesn't
this mean you can't boot an Arm system from e.g. a disk connected through
a PCI-based controller? Host bridge setup is definitely firmware's job on
x86 ...

Jan