Re: [PATCH v10 13/17] vpci: add initial support for virtual PCI bus topology

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Thu, 16 Nov 2023, Julien Grall wrote:
> IIUC, this means that Xen will allocate the BDF. I think this will become a
> problem quite quickly as some of the PCI may need to be assigned at a specific
> vBDF (I have the intel graphic card in mind).
> 
> Also, xl allows you to specificy the slot (e.g. <bdf>@<vslot>) which would not
> work with this approach.
> 
> For dom0less passthrough, I feel the virtual BDF should always be specified in
> device-tree. When a domain is created after boot, then I think you want to
> support <bdf>@<vslot> where <vslot> is optional.

Hi Julien,

I also think there should be a way to specify the virtual BDF, but if
possible (meaning: it is not super difficult to implement) I think it
would be very convenient if we could let Xen pick whatever virtual BDF
Xen wants when the user doesn't specify the virtual BDF. That's
because it would make it easier to specify the configuration for the
user. Typically the user doesn't care about the virtual BDF, only to
expose a specific host device to the VM. There are exceptions of course
and that's why I think we should also have a way for the user to
request a specific virtual BDF. One of these exceptions are integrated
GPUs: the OS drivers used to have hardcoded BDFs. So it wouldn't work if
the device shows up at a different virtual BDF compared to the host.

Thinking more about this, one way to simplify the problem would be if we
always reuse the physical BDF as virtual BDF for passthrough devices. I
think that would solve the problem and makes it much more unlikely to
run into drivers bugs.

And we allocate a "special" virtual BDF space for emulated devices, with
the Root Complex still emulated in Xen. For instance, we could reserve
ff:xx:xx and in case of clashes we could refuse to continue. Or we could
allocate the first free virtual BDF, after all the pasthrough devices.

Example:
- the user wants to assign physical 00:11.5 and b3:00.1 to the guest
- Xen create virtual BDFs 00:11.5 and b3:00.1 for the passthrough devices
- Xen allocates the next virtual BDF for emulated devices: b4:xx.x
- If more virtual BDFs are needed for emulated devices, Xen allocates
  b5:xx.x

I still think, no matter the BDF allocation scheme, that we should try
to avoid as much as possible to have two different PCI Root Complex
emulators. Ideally we would have only one PCI Root Complex emulated by
Xen. Having 2 PCI Root Complexes both of them emulated by Xen would be
tolerable but not ideal. The worst case I would like to avoid is to have
two PCI Root Complexes, one emulated by Xen and one emulated by QEMU.