Re: [PATCH v1 2/6] xen/arm: Reserve resources for virtio-pci

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Tue, 24 Sep 2024, Julien Grall wrote:
> On 24/09/2024 18:11, Edgar E. Iglesias wrote:
> > On Tue, Sep 24, 2024 at 05:35:20PM +0100, Julien Grall wrote:
> > > Hi Edgar,
> > > 
> > > On 24/09/2024 17:23, Edgar E. Iglesias wrote:
> > > > From: "Edgar E. Iglesias" <edgar.iglesias@xxxxxxx>
> > > > 
> > > > Reserve memory ranges and interrupt lines for an externally
> > > > emulated PCI controller (e.g by QEMU) dedicated to hosting
> > > > Virtio devices and potentially other emulated devices.
> > > > 
> > > > Signed-off-by: Edgar E. Iglesias <edgar.iglesias@xxxxxxx>
> > > > ---
> > > >    xen/include/public/arch-arm.h | 17 +++++++++++++++++
> > > >    1 file changed, 17 insertions(+)
> > > > 
> > > > diff --git a/xen/include/public/arch-arm.h
> > > > b/xen/include/public/arch-arm.h
> > > > index e19f0251a6..654b827715 100644
> > > > --- a/xen/include/public/arch-arm.h
> > > > +++ b/xen/include/public/arch-arm.h
> > > > @@ -494,6 +494,20 @@ typedef uint64_t xen_callback_t;
> > > >    #define GUEST_RAM1_BASE   xen_mk_ullong(0x0200000000) /* 952GB of RAM
> > > > @ 8GB */
> > > >    #define GUEST_RAM1_SIZE   xen_mk_ullong(0xee00000000)
> > > > +/* Virtio PCI - Ordered by decreasing size to keep things aligned */
> > > > +#define GUEST_VIRTIO_PCI_PREFETCH_MEM_TYPE  xen_mk_ullong(0x43000000)
> > > > +#define GUEST_VIRTIO_PCI_PREFETCH_MEM_BASE
> > > > xen_mk_ullong(0x0f000000000)
> > > > +#define GUEST_VIRTIO_PCI_PREFETCH_MEM_SIZE  xen_mk_ullong(0x100000000)
> > > > +
> > > > +#define GUEST_VIRTIO_PCI_ECAM_BASE
> > > > (GUEST_VIRTIO_PCI_PREFETCH_MEM_BASE + \
> > > > +
> > > > GUEST_VIRTIO_PCI_PREFETCH_MEM_SIZE)
> > > > +#define GUEST_VIRTIO_PCI_ECAM_SIZE      xen_mk_ullong(0x10000000)
> > > > +
> > > > +#define GUEST_VIRTIO_PCI_MEM_TYPE         xen_mk_ullong(0x02000000)
> > > > +#define GUEST_VIRTIO_PCI_MEM_BASE         (GUEST_VIRTIO_PCI_ECAM_BASE +
> > > > \
> > > > +                                           GUEST_VIRTIO_PCI_ECAM_SIZE)
> > > > +#define GUEST_VIRTIO_PCI_MEM_SIZE         xen_mk_ullong(0x00002000000)
> > > 
> > > Why is this specific to virtio PCI? However, I am not entirely convinced
> > > we
> > > should have a second PCI hostbridge exposed to the guest for a few
> > > reasons:
> > >    1. This require to reserve yet another range in the address space
> > > (could
> > > be solved with a more dynamic layout)
> > >    2. From your instructions, the guest needs to explicitly do a PCI
> > > rescan.
> 
> Another big advantage I forgot to mention is disaggregation. In a world where
> the hostbridge is handled in Xen, you could have a PCI device emulated by dom0
> while the other is emulated by a stub domain.
> 
> > > 
> > > So rather than having a second hostbridge, have you considered to extend
> > > the
> > > existing hostbridge (implemented in Xen) to support a mix of physical PCI
> > > device and virtual one?
> > > 
> > 
> > Thanks Julien,
> > 
> > It's briefly come up in a couple of discussions but I haven't looked
> > carefully at it. It is a good idea and it's probably worth prototyping
> > to see what the gaps are in hypercall interfaces, QEMU support etc.
> 
> I also vaguely recall to discuss it on xen-devel. But I couldn't find the
> discussion... :(.
> 
> I think all the hypercalls should be there but will require some plumbing in
> Xen on Arm. QEMU should be able to request Xen to forward configuration access
> for a given PCI device (see XEN_DMOP_IO_RANGE_PCI). They will then be
> forwarded to QEMU using IOREQ_TYPE_PCI_CONFIG.
> 
> We also have an hypercall to be able to inject interrupts from QEMU (see
> XEN_DMOP_set_irq_level).

Hi Julien,

Yes, I remember a thread on xen-devel too about this topic when EPAM
suggested a similar two-hostbridges approach. I was one of the people
suggesting to use a single hostbridge at the time.

However, when we looked at the implementation more closely, the
two-hostbridge approach was easier to get up and running. It works
(almost) out of the box. Currently, we have the two-hostbridge solution
working on both ARM and x86 to enable virtio-pci to work alongside vPCI
in Dom0less/Hyperlaunch configurations.

While I think that a single hostbridge is better architecturally, it is
important to consider that virtio is moving toward a new transport
(virtio-msg, Bertrand is also involved) which does not require a
hostbridge. This new transport is key for all our use-cases as it
addresses safety requirements and supports AMP configurations without a
shared hypervisor between the frontend and backend. Edgar is one of the
top contributors to virtio-msg. Given this, I don't think it's
worthwhile to invest much effort in virtio-pci, as it will be replaced
soon in embedded applications.

Cheers,
Stefano