Re: [Xen-devel] [early RFC] ARM PCI Passthrough design document

[Roger Pau Monne <roger.pau@xxxxxxxxxx>]

On Fri, Feb 10, 2017 at 10:11:53AM +0000, Paul Durrant wrote:
> > -----Original Message-----
> > From: Roger Pau Monne
> > Sent: 10 February 2017 09:49
> > To: Stefano Stabellini <sstabellini@xxxxxxxxxx>
> > Cc: Julien Grall <julien.grall@xxxxxxxxxx>; xen-devel <xen-
> > devel@xxxxxxxxxxxxxxxxxxxx>; Edgar Iglesias (edgar.iglesias@xxxxxxxxxx)
> > <edgar.iglesias@xxxxxxxxxx>; Steve Capper <Steve.Capper@xxxxxxx>; Punit
> > Agrawal <punit.agrawal@xxxxxxx>; Wei Chen <Wei.Chen@xxxxxxx>;
> > Campbell Sean <scampbel@xxxxxxxxxxxxxx>; Shanker Donthineni
> > <shankerd@xxxxxxxxxxxxxx>; Jiandi An <anjiandi@xxxxxxxxxxxxxx>;
> > manish.jaggi@xxxxxxxxxxxxxxxxxx; alistair.francis@xxxxxxxxxx; Andrew
> > Cooper <Andrew.Cooper3@xxxxxxxxxx>; Anshul Makkar
> > <anshul.makkar@xxxxxxxxxx>; Paul Durrant <Paul.Durrant@xxxxxxxxxx>
> > Subject: Re: [early RFC] ARM PCI Passthrough design document
> > 
> > On Wed, Feb 01, 2017 at 10:50:49AM -0800, Stefano Stabellini wrote:
> > > On Wed, 1 Feb 2017, Roger Pau Monné wrote:
> > > > On Wed, Jan 25, 2017 at 06:53:20PM +0000, Julien Grall wrote:
> > > > > Hi Stefano,
> > > > >
> > > > > On 24/01/17 20:07, Stefano Stabellini wrote:
> > > > > > On Tue, 24 Jan 2017, Julien Grall wrote:
> > > > > When using ECAM like host bridge, I don't think it will be an issue to
> > have
> > > > > both DOM0 and Xen accessing configuration space at the same time.
> > Although,
> > > > > we need to define who is doing what. In general case, DOM0 should
> > not
> > > > > touched an assigned PCI device. The only possible interaction would be
> > > > > resetting a device (see my answer below).
> > > >
> > > > Iff Xen is really going to perform the reset of passthrough devices, 
> > > > then I
> > > > don't see any reason to expose those devices to Dom0 at all, IMHO you
> > should
> > > > hide them from ACPI and ideally prevent Dom0 from interacting with
> > them using
> > > > the PCI configuration space (although that would require trapping on
> > accesses
> > > > to the PCI config space, which AFAIK you would like to avoid).
> > >
> > > Right! A much cleaner solution! If we are going to have Xen handle ECAM
> > > and emulating PCI host bridges, then we should go all the way and have
> > > Xen do everything about PCI.
> > 
> > Replying here because this thread has become so long that's hard to find a
> > good
> > place to put this information.
> > 
> > I've recently been told (f2f), that more complex passthrough (like Nvidia
> > vGPU
> > or Intel XenGT) work in a slightly different way, which seems to be a bit
> > incompatible with what we are proposing. I've been told that Nvidia vGPU
> > passthrough requires a driver in Dom0 (closed-source Nvidia code AFAIK),
> > and
> > that upon loading this driver a bunch of virtual functions appear out of the
> > blue in the PCI bus.
> > 
> > Now, if we completely hide passed-through devices from Dom0, it would be
> > impossible to load this driver, and thus to make the virtual functions 
> > appear.
> > I would like someone that's more familiar with this to comment, so I'm
> > adding
> > Paul and Anshul to the conversation.
> > 
> > To give some context to them, we were currently discussing to completely
> > hide
> > passthrough PCI devices from Dom0, and have Xen perform the reset of the
> > device. This would apply to PVH and ARM. Can you comment on whether
> > such
> > approach would work with things like vGPU passthrough?
> 
> Neither NVIDIA vGPU nor Intel GVT-g are pass-through. They both use emulation 
> to synthesize GPU devices for guests and then use the actual GPU to service 
> the commands sent by the guest driver to the virtual GPU. So, I think they 
> fall outside the discussion here.

So in this case those devices would simply be assigned to Dom0, and everything
would be trapped/emulated there? (by QEMU or whatever dm we are using)

> AMD MxGPU is somewhat different in that it is an almost-SRIOV solution. I say 
> 'almost' because the VF's are not truly independent and so some interception 
> of accesses to certain registers is required, so that arbitration can be 
> applied, or they can be blocked. In this case a dedicated driver in dom0 is 
> required, and I believe it needs access to both the PF and all the VFs to 
> function correctly. However, once initial set-up is done, I think the VFs 
> could then be hidden from dom0. The PF is never passed-through and so there 
> should be no issue in leaving it visible to dom0.

The approach we where thinking of is hiding everything from Dom0 when it
boots, so that Dom0 would never really see those devices. This would be done by
Xen scanning the PCI bus and any ECAM areas. DEvices that first need to be
assigned to Dom0 and then hidden where not part of the approach here.

> There is a further complication with GVT-d (Intel's term for GPU 
> pass-through) also because I believe there is also some initial set-up 
> required and some supporting emulation (e.g. Intel's guest driver expects 
> there to be an ISA bridge along with the GPU) which may need access to the 
> real GPU. It is also possible that, once this set-up is done, the GPU can 
> then be hidden from dom0 but I'm not sure because I was not involved with 
> that code.

And then I guess some MMIO regions are assigned to the guest, and some dm
performs the trapping of the accesses to the configuration space?

> Full pass-through of NVIDIA and AMD GPUs does not involve access from dom0 at 
> all though, so I don't think there should be any complication there.

Yes, in that case they would be treated as regular PCI devices, no involvement
from Dom0 would be needed. I'm more worried about this mixed cases, where some
Dom0 interaction is needed in order to perform the passthrough.

> Does that all make sense?

I guess, could you please keep an eye on further design documents? Just to
make sure that what's described here would work for the more complex
passthrough scenarios that XenServer supports.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel