Re: Virtio in Xen on Arm (based on IOREQ concept)

[Julien Grall <julien@xxxxxxx>]

Hi Alex,

Thank you for your feedback!

On 21/07/2020 15:15, Alex Bennée wrote:
> Julien Grall <julien@xxxxxxx> writes:
>
> > (+ Andree for the vGIC).
> >
> > Hi Stefano,
> >
> > On 20/07/2020 21:38, Stefano Stabellini wrote:
> > > On Fri, 17 Jul 2020, Oleksandr wrote:
> > > > > > *A few word about solution:*
> > > > > > As it was mentioned at [1], in order to implement virtio-mmio Xen on Arm
> > > > > Any plans for virtio-pci? Arm seems to be moving to the PCI bus, and
> > > > > it would be very interesting from a x86 PoV, as I don't think
> > > > > virtio-mmio is something that you can easily use on x86 (or even use
> > > > > at all).
> > > > Being honest I didn't consider virtio-pci so far. Julien's PoC (we are based
> > > > on) provides support for the virtio-mmio transport
> > > >
> > > > which is enough to start working around VirtIO and is not as complex as
> > > > virtio-pci. But it doesn't mean there is no way for virtio-pci in Xen.
> > > >
> > > > I think, this could be added in next steps. But the nearest target is
> > > > virtio-mmio approach (of course if the community agrees on that).
> > > Aside from complexity and easy-of-development, are there any other
> > > architectural reasons for using virtio-mmio?
> <snip>
> > > For instance, what's your take on notifications with virtio-mmio? How
> > > are they modelled today?
> > The backend will notify the frontend using an SPI. The other way around
> > (frontend -> backend) is based on an MMIO write.
> >
> > We have an interface to allow the backend to control whether the
> > interrupt level (i.e. low, high). However, the "old" vGIC doesn't handle
> > properly level interrupts. So we would end up to treat level interrupts
> > as edge.
> >
> > Technically, the problem is already existing with HW interrupts, but the
> > HW should fire it again if the interrupt line is still asserted. Another
> > issue is the interrupt may fire even if the interrupt line was
> > deasserted (IIRC this caused some interesting problem with the Arch timer).
> >
> > I am a bit concerned that the issue will be more proeminent for virtual
> > interrupts. I know that we have some gross hack in the vpl011 to handle
> > a level interrupts. So maybe it is time to switch to the new vGIC?
> >
> > > Are they good enough or do we need MSIs?
> > I am not sure whether virtio-mmio supports MSIs. However for virtio-pci,
> > MSIs is going to be useful to improve performance. This may mean to
> > expose an ITS, so we would need to add support for guest.
> virtio-mmio doesn't support MSI's at the moment although there have been
> proposals to update the spec to allow them. At the moment the cost of
> reading the ISR value and then writing an ack in vm_interrupt:
>
>         /* Read and acknowledge interrupts */
>         status = readl(vm_dev->base + VIRTIO_MMIO_INTERRUPT_STATUS);
>         writel(status, vm_dev->base + VIRTIO_MMIO_INTERRUPT_ACK);
Hmmmm, the current way to handle MMIO is the following:
    * pause the vCPU
    * Forward the access to the backend domain
    * Schedule the backend domain
    * Wait for the access to be handled
    * unpause the vCPU

So the sequence is going to be fairly expensive on Xen.

> puts an extra vmexit cost to trap an emulate each exit. Getting an MSI
> via an exitless access to the GIC would be better I think.
> I'm not quite
> sure what the path to IRQs from Xen is.
vmexit on Xen on Arm is pretty cheap compare to KVM as we don't save a 
lot of things. In this situation, they handling an extra trap for the 
interrupt is likely to be meaningless compare to the sequence above.
I am assuming the sequence is also going to be used by the MSIs, right?

It feels to me that it would be worth spending time to investigate the 
cost of that sequence. It might be possible to optimize the ACK and 
avoid to wait for the backend to handle the access.
Cheers,

--
Julien Grall