[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: SR-IOV: do we need to virtualize in Xen or rely on Dom0?

  • To: Oleksandr Andrushchenko <Oleksandr_Andrushchenko@xxxxxxxx>
  • From: Roger Pau Monné <roger.pau@xxxxxxxxxx>
  • Date: Thu, 10 Jun 2021 09:54:40 +0200
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KUP7Nqu6Fmge9xIhhwPZjy2IirUsrJcuon4/g2rpM/Y=; b=YqpMcUBPLYW1kvACF8ARtfAxKQa9vIUDbcJZ/JgqhvPxo45/hCDZXkNe3hHldoMnw6xWNPjLJPh7Uv4fPnJuA78zAQMgBolg8InM4CNo9LTsfMpRlGXnWYkxjg6fYGhVCoqBJcjU9jMr1FJrOGp4MMgP8mE1lnbHI0eIc+t+Ng9xLungtXxxmBX4tisaSr07pFIK+GucM7Le0z4CxFq4imCER8QU+EqxCGhuVbAu0RMtmjaxlvgyA0+q8vBPRu1Z3iiENf8sDxVfYJfIN3/X9S0iu7Aj0Y6YIhubeVJpVtSdL589uEXu77HSijgvOQFCVJy9bNFc+uWSzdT0Qh7wYA==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jNoufSonGXXLycwBc1ZJkC7ydbBl1lSAQie0zpoNIyItfyAHsXdpqO9hoDhqfgkXRmF5jfX5PRGj5SQ8EYdC6QVMGPjm66C+ePAnJIfgEoeWgbeLxZleTUkHiv9awmkkls7TGI55p2MbBjC9G+2n0Ds/sqDqy7pmciaUTSUZrjszrRYJxWVeqhJnVxeLbRVr7hcn60oPIe7DyqC/Z9r8f5R9ESfBUMbMKcdFX5HiNHow+ps8zB1BuSepWAWVZSP1Z8BUYyzS+qTk9QezHvK45bWSqL6JCvOukPOwIxgtLuZJ02BnNwJFeyTggjOg3LobVSZcuIkvl2W7nGPixrEhwA==
  • Authentication-results: esa2.hc3370-68.iphmx.com; dkim=pass (signature verified) header.i=@citrix.onmicrosoft.com
  • Cc: "xen-devel@xxxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxxx>, "Jan Beulich" <jbeulich@xxxxxxxx>
  • Delivery-date: Thu, 10 Jun 2021 07:55:01 +0000
  • Ironport-hdrordr: A9a23:MoZUe6jrPn1T/hVPW3KE28GLpHBQXyt13DAbv31ZSRFFG/FwyP rBoB1L73DJYWgqNE3I+erhBEGBKUmsk6KdxbNhQItKPTOWwldASbsC0WKM+UyEJ8STzJ846U 4kSdkDNDSSNykLsS+Z2njBLz9I+rDum8rE9ISurQYecegpUdAa0+4QMHfrLqQcfng+OXNWLu v62iIRzADQB0j/I/7LS0UtbqzmnZnmhZjmaRkJC1oO7xSPtyqh7PrfHwKD1hkTfjtTyfN6mF K13DDR1+GGibWW2xXc32jc49B/n8bg8MJKAIiphtIOIjvhpw60bMBKWqGEvhoyvOazgWxa3O XkklMFBYBe+nnRdma6rV/GwA/7ygsj7Hfk1BuxnWbjidaRfkN7N+NxwaZiNjfJ4Uspu99xlI hR2XiCipZRBRTc2Azg+tnzUQ1wnEbcmwtirQcqtQ0cbWIiUs4VkWRGl3klVKvoXRiKprzPKd MeT/01v51tABSnhxmzhBgd/DSuNk5DVituDHJy/PB96AIm6EyR+XFojfD3rk1wga7VdKM0kN gsEp4Y342mHfVmJ56UOo86ML2K4zv2MG3x2SSpUA3aKJ0=
  • Ironport-sdr: qBJcfkdp4OF5pEqEX7mx58Jdfr2jbIHnpM1hONdPUMYRbWP5+a2Px++ryVdBB2zGuk9931Q2pY GAdQ7mmPHzg9x7oCoV8K6Sj0QbeEB55P6JRoxUnTrLZPfE6eAj+cwXzyi4tuNiM/AWq6eKPFl0 Qr7IJcRS2s0VnHmAHbvlviQXMShpFiURQvGiTLWHo3U+9Ym/sHJBQ4HbreB83pswUzhDNNmVeS en4q5BkSfFdSb5JFHNPYHvTYFasIUQuuVb0EAZ0b/iHbV6WOPV/omrf2QoEK06YoYP6QudeJ+O sYY=
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On Fri, Jun 04, 2021 at 06:37:27AM +0000, Oleksandr Andrushchenko wrote:
> Hi, all!
> While working on PCI SR-IOV support for ARM I started porting [1] on top
> of current PCI on ARM support [2]. The question I have for this series
> is if we really need emulating SR-IOV code in Xen?
> I have implemented a PoC for SR-IOV on ARM [3] (please see the top 2 
> patches)
> and it "works for me": MSI support is still WIP, but I was able to see that
> VFs are properly seen in the guest and BARs are properly programmed in p2m.
> What I can't fully understand is if we can live with this approach or there
> are use-cases I can't see.
> Previously I've been told that this approach might not work on FreeBSD 
> running
> as Domain-0, but is seems that "PCI Passthrough is not supported 
> (Xen/FreeBSD)"
> anyways [4].

PCI passthorgh is not supported on FreeBSD dom0 because PCI
passthrough is not supported by Xen itself when using a PVH dom0, and
that's the only mode FreeBSD dom0 can use.

PHYSDEVOP_pci_device_add can be added to FreeBSD, so it could be made
to work. I however think this is not the proper way to implement
SR-IOV support.

> I also see ACRN hypervisor [5] implements SR-IOV inside it which makes 
> me think I
> miss some important use-case on x86 though.
> I would like to ask for any advise with SR-IOV in hypervisor respect, 
> any pointers
> to documentation or any other source which might be handy in deciding if 
> we do
> need SR-IOV complexity in Xen.
> And it does bring complexity if you compare [1] and [3])...
> A bit of technical details on the approach implemented [3]:
> 1. We rely on PHYSDEVOP_pci_device_add
> 2. We rely on Domain-0 SR-IOV drivers to instantiate VFs
> 3. BARs are programmed in p2m implementing guest view on those (we have 
> extended
> vPCI code for that and this path is used for both "normal" devices and 
> VFs the same way)
> 4. No need to trap PCI_SRIOV_CTRL
> 5. No need to wait 100ms in Xen before attempting to access VF registers 
> when
> enabling virtual functions on the PF - this is handled by Domain-0 itself.

I think the SR-IOV capability should be handled like any other PCI
capability, ie: like we currently handle MSI or MSI-X in vPCI.

It's likely that using some kind of hypercall in order to deal with
SR-IOV could make this easier to implement in Xen, but that just adds
more code to all OSes that want to run as the hardware domain.

OTOH if we properly trap accesses to the SR-IOV capability (like it
was proposed in [1] from your references) we won't have to modify OSes
that want to run as hardware domains in order to handle SR-IOV devices.

IMO going for the hypercall option seems easier now, but adds a burden
to all OSes that want to manage SR-IOV devices that will hurt us long

Thanks, Roger.

> Thank you in advance,
> Oleksandr
> [1] 
> https://lists.xenproject.org/archives/html/xen-devel/2018-07/msg01494.html
> [2] 
> https://gitlab.com/xen-project/fusa/xen-integration/-/tree/integration/pci-passthrough
> [3] https://github.com/xen-troops/xen/commits/pci_phase2
> [4] https://wiki.freebsd.org/Xen
> [5] https://projectacrn.github.io/latest/tutorials/sriov_virtualization.html



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.