Re: [RFC PATCH 2/6] xen/public: arch-arm: reserve resources for virtio-pci

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

+ Stewart, Vikram

On Wed, 15 Nov 2023, Oleksandr Tyshchenko wrote:
> On 15.11.23 14:33, Julien Grall wrote:
> > Thanks for adding support for virtio-pci in Xen. I have some questions.
> > 
> > On 15/11/2023 11:26, Sergiy Kibrik wrote:
> >> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@xxxxxxxx>
> >>
> >> In order to enable more use-cases such as having multiple
> >> device-models (Qemu) running in different backend domains which provide
> >> virtio-pci devices for the same guest, we allocate and expose one
> >> PCI host bridge for every virtio backend domain for that guest.
> > 
> > OOI, why do you need to expose one PCI host bridge for every stubdomain?
> > 
> > In fact looking at the next patch, it seems you are handling some of the 
> > hostbridge request in Xen. This is adds a bit more confusion.
> > 
> > I was expecting the virtual PCI device would be in the vPCI and each 
> > Device emulator would advertise which BDF they are covering.
> 
> 
> This patch series only covers use-cases where the device emulator 
> handles the *entire* PCI Host bridge and PCI (virtio-pci) devices behind 
> it (i.e. Qemu). Also this patch series doesn't touch vPCI/PCI 
> pass-through resources, handling, accounting, nothing. From the 
> hypervisor we only need a help to intercept the config space accesses 
> happen in a range [GUEST_VIRTIO_PCI_ECAM_BASE ... 
> GUEST_VIRTIO_PCI_ECAM_BASE + GUEST_VIRTIO_PCI_TOTAL_ECAM_SIZE] and 
> forward them to the linked device emulator (if any), that's all.
> 
> It is not possible (with current series) to run device emulators what
> emulate only separate PCI (virtio-pci) devices. For it to be possible, I 
> think, much more changes are required than current patch series does. 
> There at least should be special PCI Host bridge emulation in Xen (or 
> reuse vPCI) for the integration. Also Xen should be in charge of forming 
> resulting PCI interrupt based on each PCI device level signaling (if we 
> use legacy interrupts), some kind of x86's XEN_DMOP_set_pci_intx_level, 
> etc. Please note, I am not saying this is not possible in general, 
> likely it is possible, but initial patch series doesn't cover these 
> use-cases)
>
> We expose one PCI host bridge per virtio backend domain. This is a 
> separate PCI host bridge to combine all virtio-pci devices running in 
> the same backend domain (in the same device emulator currently).
> The examples:
> - if only one domain runs Qemu which servers virtio-blk, virtio-net, 
> virtio-console devices for DomU - only single PCI Host bridge will be 
> exposed for DomU
> - if we add another domain to run Qemu to serve additionally virtio-gpu, 
> virtio-input and virtio-snd for the *same* DomU - we expose second PCI 
> Host bridge for DomU
> 
> I am afraid, we cannot end up exposing only single PCI Host bridge with 
> current model (if we use device emulators running in different domains 
> that handles the *entire* PCI Host bridges), this won't work.
 

We were discussing the topic of vPCI and Virtio PCI just this morning
with Stewart and Vikram. We also intend to make them work well together
in the next couple of months (great timing!!)

However, our thinking is to go with the other approach Julien
suggested: a single PCI Root Complex emulated in Xen by vPCI. QEMU would
register individual PCI devices against it.

Vikram, Stewart, please comment. Our understanding is that it should be
possible to make QEMU virtio-pci work with vPCI with relatively minor
efforts and AMD volunteers to do the work in the next couple of months
on the vPCI side.


Although it should be possible to make both approaches work at the same
time, given that it would seem that EPAM and AMD have very similar
requirements, I suggest we work together and collaborate on a single
approach going forward that works best for everyone.


Let me start by saying that if we can get away with it, I think that a
single PCI Root Complex in Xen would be best because it requires less
complexity. Why emulate 2/3 PCI Root Complexes if we can emulate only
one?

Stewart, you are deep into vPCI, what's your thinking?