Re: [Xen-devel] [early RFC] ARM PCI Passthrough design document

[Paul Durrant <Paul.Durrant@xxxxxxxxxx>]

> -----Original Message-----
> From: Roger Pau Monne
> Sent: 10 February 2017 09:49
> To: Stefano Stabellini <sstabellini@xxxxxxxxxx>
> Cc: Julien Grall <julien.grall@xxxxxxxxxx>; xen-devel <xen-
> devel@xxxxxxxxxxxxxxxxxxxx>; Edgar Iglesias (edgar.iglesias@xxxxxxxxxx)
> <edgar.iglesias@xxxxxxxxxx>; Steve Capper <Steve.Capper@xxxxxxx>; Punit
> Agrawal <punit.agrawal@xxxxxxx>; Wei Chen <Wei.Chen@xxxxxxx>;
> Campbell Sean <scampbel@xxxxxxxxxxxxxx>; Shanker Donthineni
> <shankerd@xxxxxxxxxxxxxx>; Jiandi An <anjiandi@xxxxxxxxxxxxxx>;
> manish.jaggi@xxxxxxxxxxxxxxxxxx; alistair.francis@xxxxxxxxxx; Andrew
> Cooper <Andrew.Cooper3@xxxxxxxxxx>; Anshul Makkar
> <anshul.makkar@xxxxxxxxxx>; Paul Durrant <Paul.Durrant@xxxxxxxxxx>
> Subject: Re: [early RFC] ARM PCI Passthrough design document
> 
> On Wed, Feb 01, 2017 at 10:50:49AM -0800, Stefano Stabellini wrote:
> > On Wed, 1 Feb 2017, Roger Pau Monné wrote:
> > > On Wed, Jan 25, 2017 at 06:53:20PM +0000, Julien Grall wrote:
> > > > Hi Stefano,
> > > >
> > > > On 24/01/17 20:07, Stefano Stabellini wrote:
> > > > > On Tue, 24 Jan 2017, Julien Grall wrote:
> > > > When using ECAM like host bridge, I don't think it will be an issue to
> have
> > > > both DOM0 and Xen accessing configuration space at the same time.
> Although,
> > > > we need to define who is doing what. In general case, DOM0 should
> not
> > > > touched an assigned PCI device. The only possible interaction would be
> > > > resetting a device (see my answer below).
> > >
> > > Iff Xen is really going to perform the reset of passthrough devices, then 
> > > I
> > > don't see any reason to expose those devices to Dom0 at all, IMHO you
> should
> > > hide them from ACPI and ideally prevent Dom0 from interacting with
> them using
> > > the PCI configuration space (although that would require trapping on
> accesses
> > > to the PCI config space, which AFAIK you would like to avoid).
> >
> > Right! A much cleaner solution! If we are going to have Xen handle ECAM
> > and emulating PCI host bridges, then we should go all the way and have
> > Xen do everything about PCI.
> 
> Replying here because this thread has become so long that's hard to find a
> good
> place to put this information.
> 
> I've recently been told (f2f), that more complex passthrough (like Nvidia
> vGPU
> or Intel XenGT) work in a slightly different way, which seems to be a bit
> incompatible with what we are proposing. I've been told that Nvidia vGPU
> passthrough requires a driver in Dom0 (closed-source Nvidia code AFAIK),
> and
> that upon loading this driver a bunch of virtual functions appear out of the
> blue in the PCI bus.
> 
> Now, if we completely hide passed-through devices from Dom0, it would be
> impossible to load this driver, and thus to make the virtual functions appear.
> I would like someone that's more familiar with this to comment, so I'm
> adding
> Paul and Anshul to the conversation.
> 
> To give some context to them, we were currently discussing to completely
> hide
> passthrough PCI devices from Dom0, and have Xen perform the reset of the
> device. This would apply to PVH and ARM. Can you comment on whether
> such
> approach would work with things like vGPU passthrough?

Neither NVIDIA vGPU nor Intel GVT-g are pass-through. They both use emulation 
to synthesize GPU devices for guests and then use the actual GPU to service the 
commands sent by the guest driver to the virtual GPU. So, I think they fall 
outside the discussion here.
AMD MxGPU is somewhat different in that it is an almost-SRIOV solution. I say 
'almost' because the VF's are not truly independent and so some interception of 
accesses to certain registers is required, so that arbitration can be applied, 
or they can be blocked. In this case a dedicated driver in dom0 is required, 
and I believe it needs access to both the PF and all the VFs to function 
correctly. However, once initial set-up is done, I think the VFs could then be 
hidden from dom0. The PF is never passed-through and so there should be no 
issue in leaving it visible to dom0.

There is a further complication with GVT-d (Intel's term for GPU pass-through) 
also because I believe there is also some initial set-up required and some 
supporting emulation (e.g. Intel's guest driver expects there to be an ISA 
bridge along with the GPU) which may need access to the real GPU. It is also 
possible that, once this set-up is done, the GPU can then be hidden from dom0 
but I'm not sure because I was not involved with that code.

Full pass-through of NVIDIA and AMD GPUs does not involve access from dom0 at 
all though, so I don't think there should be any complication there.

Does that all make sense?

  Paul

> 
> Roger.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel