SR-IOV: do we need to virtualize in Xen or rely on Dom0?

[Oleksandr Andrushchenko <Oleksandr_Andrushchenko@xxxxxxxx>]

Hi, all!

While working on PCI SR-IOV support for ARM I started porting [1] on top
of current PCI on ARM support [2]. The question I have for this series
is if we really need emulating SR-IOV code in Xen?

I have implemented a PoC for SR-IOV on ARM [3] (please see the top 2 
patches)
and it "works for me": MSI support is still WIP, but I was able to see that
VFs are properly seen in the guest and BARs are properly programmed in p2m.

What I can't fully understand is if we can live with this approach or there
are use-cases I can't see.

Previously I've been told that this approach might not work on FreeBSD 
running
as Domain-0, but is seems that "PCI Passthrough is not supported 
(Xen/FreeBSD)"
anyways [4].

I also see ACRN hypervisor [5] implements SR-IOV inside it which makes 
me think I
miss some important use-case on x86 though.

I would like to ask for any advise with SR-IOV in hypervisor respect, 
any pointers
to documentation or any other source which might be handy in deciding if 
we do
need SR-IOV complexity in Xen.

And it does bring complexity if you compare [1] and [3])...

A bit of technical details on the approach implemented [3]:
1. We rely on PHYSDEVOP_pci_device_add
2. We rely on Domain-0 SR-IOV drivers to instantiate VFs
3. BARs are programmed in p2m implementing guest view on those (we have 
extended
vPCI code for that and this path is used for both "normal" devices and 
VFs the same way)
4. No need to trap PCI_SRIOV_CTRL
5. No need to wait 100ms in Xen before attempting to access VF registers 
when
enabling virtual functions on the PF - this is handled by Domain-0 itself.

Thank you in advance,
Oleksandr

[1] 
https://lists.xenproject.org/archives/html/xen-devel/2018-07/msg01494.html
[2] 
https://gitlab.com/xen-project/fusa/xen-integration/-/tree/integration/pci-passthrough
[3] https://github.com/xen-troops/xen/commits/pci_phase2
[4] https://wiki.freebsd.org/Xen
[5] https://projectacrn.github.io/latest/tutorials/sriov_virtualization.html