[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: SR-IOV: do we need to virtualize in Xen or rely on Dom0?

  • To: Roger Pau Monné <roger.pau@xxxxxxxxxx>
  • From: Oleksandr Andrushchenko <Oleksandr_Andrushchenko@xxxxxxxx>
  • Date: Thu, 10 Jun 2021 10:01:16 +0000
  • Accept-language: en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xyG66UZuq8U1OUofAufPfAzsja+kZ1kdTElr5MgJ/Ro=; b=bVNt+Rbxkmcu3EDZndQ4wNj71Ecl3nSWeOA5ZD0vPlIbhO9zNZxWgCtJhD5sT5C0P+qq/hJgqJDyCW78rH6pIZ3OYI3eoAT1dbnGbCsOGREHfZwBrfPr3fbnAClTm5hgaQbp/b6a9ulEp6STsbELf6Sv6/ZIGnvluMIh+wlbzP8EQ0XlcwPrDFcqwqFi2UhFe6vFHTJQdNiw7JwpKUWDAXEcCgX7J4VOIeMAISn1A5LKpOvcoxt8ePodu9npsEa5xwSI25BLDDb6E8uXq1Uu4hldtdZmaYsaRLmiJDQpGKvVFRLicL5pEdaWCi75lLgB0YWZsvjIlTKDuBLXHwolBw==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oUuKFS68aWIdFI6Tyq14vHp5+H1TTWYUYKzevIt07H7I7aE7lBhp/FPUc3OImhzZCeBS6PFGdvPcc5oZH0j20sfLsntwmNvmSVNl3HqHGt7cEahykAlpUD06bRwKTy1ppSovQwSQT0VmAPpMEXmjS7JQkNMvvrWob80L2mfId8wiEsSkEuPZvAnIC8g490aK0SnlMfPOjDBKYag5VY3YlhxK2cvuMkjtmYHV1Bobn5YD6cUxtAeT+isGXFhmPqrDXvvi9YIf/gQUbSvDOf2d2KU8ymacK6G8QFvTq9bDrYpCwWqPytYQMuHMOewQRbI4hQ+koncITKspqmd7Mh4tJg==
  • Authentication-results: citrix.com; dkim=none (message not signed) header.d=none;citrix.com; dmarc=none action=none header.from=epam.com;
  • Cc: "xen-devel@xxxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxxx>, Jan Beulich <jbeulich@xxxxxxxx>
  • Delivery-date: Thu, 10 Jun 2021 10:01:39 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Thread-index: AQHXWQwW/O3obtZgqkmpGRvt7ZNDkasM6aMAgAAjXoA=
  • Thread-topic: SR-IOV: do we need to virtualize in Xen or rely on Dom0?

Hello, Roger!

On 10.06.21 10:54, Roger Pau Monné wrote:
> On Fri, Jun 04, 2021 at 06:37:27AM +0000, Oleksandr Andrushchenko wrote:
>> Hi, all!
>> While working on PCI SR-IOV support for ARM I started porting [1] on top
>> of current PCI on ARM support [2]. The question I have for this series
>> is if we really need emulating SR-IOV code in Xen?
>> I have implemented a PoC for SR-IOV on ARM [3] (please see the top 2
>> patches)
>> and it "works for me": MSI support is still WIP, but I was able to see that
>> VFs are properly seen in the guest and BARs are properly programmed in p2m.
>> What I can't fully understand is if we can live with this approach or there
>> are use-cases I can't see.
>> Previously I've been told that this approach might not work on FreeBSD
>> running
>> as Domain-0, but is seems that "PCI Passthrough is not supported
>> (Xen/FreeBSD)"
>> anyways [4].
> PCI passthorgh is not supported on FreeBSD dom0 because PCI
> passthrough is not supported by Xen itself when using a PVH dom0, and
> that's the only mode FreeBSD dom0 can use.

So, it is still not clear to me: how and if PCI passthrough is supported

on FreeBSD, what are the scenarios and requirements for that?

> PHYSDEVOP_pci_device_add can be added to FreeBSD, so it could be made
> to work. I however think this is not the proper way to implement
> SR-IOV support.

I was not able to find any support for PHYSDEVOP_XXX in FreeBSD code,

could you please point me to where are these used?

If they are not, so how Xen under FreeBSD knows about PCI devices?

I am trying to extrapolate my knowledge of how Linux does that

(during PCI enumeration in Domain-0 we use hypercalls)

>> I also see ACRN hypervisor [5] implements SR-IOV inside it which makes
>> me think I
>> miss some important use-case on x86 though.
>> I would like to ask for any advise with SR-IOV in hypervisor respect,
>> any pointers
>> to documentation or any other source which might be handy in deciding if
>> we do
>> need SR-IOV complexity in Xen.
>> And it does bring complexity if you compare [1] and [3])...
>> A bit of technical details on the approach implemented [3]:
>> 1. We rely on PHYSDEVOP_pci_device_add
>> 2. We rely on Domain-0 SR-IOV drivers to instantiate VFs
>> 3. BARs are programmed in p2m implementing guest view on those (we have
>> extended
>> vPCI code for that and this path is used for both "normal" devices and
>> VFs the same way)
>> 4. No need to trap PCI_SRIOV_CTRL
>> 5. No need to wait 100ms in Xen before attempting to access VF registers
>> when
>> enabling virtual functions on the PF - this is handled by Domain-0 itself.
> I think the SR-IOV capability should be handled like any other PCI
> capability, ie: like we currently handle MSI or MSI-X in vPCI.
> It's likely that using some kind of hypercall in order to deal with
> SR-IOV could make this easier to implement in Xen, but that just adds
> more code to all OSes that want to run as the hardware domain.

I didn't introduce any new, but PHYSDEVOP_pci_device_add was enough.

The rest I did in Xen itself wrt SR-IOV.

> OTOH if we properly trap accesses to the SR-IOV capability (like it
> was proposed in [1] from your references) we won't have to modify OSes
> that want to run as hardware domains in order to handle SR-IOV devices.

Out of curiosity, could you please name a few? I do understand that

we do want to support unmodified OSes and this is indeed important.

But, still what are the other OSes which do support Xen + PCI passthrough?

> IMO going for the hypercall option seems easier now, but adds a burden
> to all OSes that want to manage SR-IOV devices that will hurt us long
> term.

Again, I was able to make it somewhat work with PHYSDEVOP_pci_device_add only.

> Thanks, Roger.

Thank you for your valuable comments,


>> Thank you in advance,
>> Oleksandr
>> [1]
>> https://urldefense.com/v3/__https://lists.xenproject.org/archives/html/xen-devel/2018-07/msg01494.html__;!!GF_29dbcQIUBPA!mIpLEvBmxRoFWUgYc7MFAF032kDGd897NP9t6d1fsc9uZoLHSW97GX8xAWKhvlFF8CezmvSZGg$
>>  [lists[.]xenproject[.]org]
>> [2]
>> https://urldefense.com/v3/__https://gitlab.com/xen-project/fusa/xen-integration/-/tree/integration/pci-passthrough__;!!GF_29dbcQIUBPA!mIpLEvBmxRoFWUgYc7MFAF032kDGd897NP9t6d1fsc9uZoLHSW97GX8xAWKhvlFF8Cf3Z8Yk7w$
>>  [gitlab[.]com]
>> [3] 
>> https://urldefense.com/v3/__https://github.com/xen-troops/xen/commits/pci_phase2__;!!GF_29dbcQIUBPA!mIpLEvBmxRoFWUgYc7MFAF032kDGd897NP9t6d1fsc9uZoLHSW97GX8xAWKhvlFF8CfncIdgvw$
>>  [github[.]com]
>> [4] 
>> https://urldefense.com/v3/__https://wiki.freebsd.org/Xen__;!!GF_29dbcQIUBPA!mIpLEvBmxRoFWUgYc7MFAF032kDGd897NP9t6d1fsc9uZoLHSW97GX8xAWKhvlFF8CceLOPdyg$
>>  [wiki[.]freebsd[.]org]
>> [5] 
>> https://urldefense.com/v3/__https://projectacrn.github.io/latest/tutorials/sriov_virtualization.html__;!!GF_29dbcQIUBPA!mIpLEvBmxRoFWUgYc7MFAF032kDGd897NP9t6d1fsc9uZoLHSW97GX8xAWKhvlFF8CdjqHPxZQ$
>>  [projectacrn[.]github[.]io]



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.