Re: Virtio in Xen on Arm (based on IOREQ concept)

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Tue, 21 Jul 2020, Julien Grall wrote:
> Hi Alex,
> 
> Thank you for your feedback!
> 
> On 21/07/2020 15:15, Alex Bennée wrote:
> > Julien Grall <julien@xxxxxxx> writes:
> > 
> > > (+ Andree for the vGIC).
> > > 
> > > Hi Stefano,
> > > 
> > > On 20/07/2020 21:38, Stefano Stabellini wrote:
> > > > On Fri, 17 Jul 2020, Oleksandr wrote:
> > > > > > > *A few word about solution:*
> > > > > > > As it was mentioned at [1], in order to implement virtio-mmio Xen
> > > > > > > on Arm
> > > > > > Any plans for virtio-pci? Arm seems to be moving to the PCI bus, and
> > > > > > it would be very interesting from a x86 PoV, as I don't think
> > > > > > virtio-mmio is something that you can easily use on x86 (or even use
> > > > > > at all).
> > > > > 
> > > > > Being honest I didn't consider virtio-pci so far. Julien's PoC (we are
> > > > > based
> > > > > on) provides support for the virtio-mmio transport
> > > > > 
> > > > > which is enough to start working around VirtIO and is not as complex
> > > > > as
> > > > > virtio-pci. But it doesn't mean there is no way for virtio-pci in Xen.
> > > > > 
> > > > > I think, this could be added in next steps. But the nearest target is
> > > > > virtio-mmio approach (of course if the community agrees on that).
> > > 
> > > > Aside from complexity and easy-of-development, are there any other
> > > > architectural reasons for using virtio-mmio?
> > > 
> > <snip>
> > > > 
> > > > For instance, what's your take on notifications with virtio-mmio? How
> > > > are they modelled today?
> > > 
> > > The backend will notify the frontend using an SPI. The other way around
> > > (frontend -> backend) is based on an MMIO write.
> > > 
> > > We have an interface to allow the backend to control whether the
> > > interrupt level (i.e. low, high). However, the "old" vGIC doesn't handle
> > > properly level interrupts. So we would end up to treat level interrupts
> > > as edge.
> > > 
> > > Technically, the problem is already existing with HW interrupts, but the
> > > HW should fire it again if the interrupt line is still asserted. Another
> > > issue is the interrupt may fire even if the interrupt line was
> > > deasserted (IIRC this caused some interesting problem with the Arch
> > > timer).
> > > 
> > > I am a bit concerned that the issue will be more proeminent for virtual
> > > interrupts. I know that we have some gross hack in the vpl011 to handle
> > > a level interrupts. So maybe it is time to switch to the new vGIC?
> > > 
> > > > Are they good enough or do we need MSIs?
> > > 
> > > I am not sure whether virtio-mmio supports MSIs. However for virtio-pci,
> > > MSIs is going to be useful to improve performance. This may mean to
> > > expose an ITS, so we would need to add support for guest.
> > 
> > virtio-mmio doesn't support MSI's at the moment although there have been
> > proposals to update the spec to allow them. At the moment the cost of
> > reading the ISR value and then writing an ack in vm_interrupt:
> > 
> >     /* Read and acknowledge interrupts */
> >     status = readl(vm_dev->base + VIRTIO_MMIO_INTERRUPT_STATUS);
> >     writel(status, vm_dev->base + VIRTIO_MMIO_INTERRUPT_ACK);
> > 
> 
> Hmmmm, the current way to handle MMIO is the following:
>     * pause the vCPU
>     * Forward the access to the backend domain
>     * Schedule the backend domain
>     * Wait for the access to be handled
>     * unpause the vCPU
> 
> So the sequence is going to be fairly expensive on Xen.
> 
> > puts an extra vmexit cost to trap an emulate each exit. Getting an MSI
> > via an exitless access to the GIC would be better I think.
> > I'm not quite
> > sure what the path to IRQs from Xen is.
> 
> vmexit on Xen on Arm is pretty cheap compare to KVM as we don't save a lot of
> things. In this situation, they handling an extra trap for the interrupt is
> likely to be meaningless compare to the sequence above.

+1


> I am assuming the sequence is also going to be used by the MSIs, right?
> 
> It feels to me that it would be worth spending time to investigate the cost of
> that sequence. It might be possible to optimize the ACK and avoid to wait for
> the backend to handle the access.

+1