RE: Virtio Xen (WAS: Re: [Linux] [ARM] Granting memory obtained from the DMA API)

[Paul Durrant <xadimgnik@xxxxxxxxx>]

> -----Original Message-----
> From: Xen-devel <xen-devel-bounces@xxxxxxxxxxxxxxxxxxxx> On Behalf Of Julien 
> Grall
> Sent: 23 September 2020 10:04
> To: Simon Leiner <simon@xxxxxxxxx>
> Cc: xen-devel@xxxxxxxxxxxxxxxxxxxx; Stefano Stabellini 
> <sstabellini@xxxxxxxxxx>; Bertrand Marquis
> <Bertrand.Marquis@xxxxxxx>; Oleksandr Tyshchenko <olekstysh@xxxxxxxxx>
> Subject: Re: Virtio Xen (WAS: Re: [Linux] [ARM] Granting memory obtained from 
> the DMA API)
> 
> On 29/08/2020 11:38, Simon Leiner wrote:
> > Hi Julien,
> 
> Hi Simon,
> 
> Apologies for the late answer.
> 
> > On 25.08.20 15:02, Julien Grall wrote:
> >> May I ask why did you create a new transport rather than using the
> >> existing one?
> >
> > We wanted a mechanism for dynamically creating virtio devices at
> > runtime. I looked at virtio-mmio briefly and it seemed to me that a lot
> > of things would have to be known in advance (how many devices are
> > there? How much memory do they need? Where does the memory range for
> > virtio-mmio start on the device domain?). So after reading a bit about
> > Xen and how the classic split drivers work, I figured building a split
> > driver for virtio was the way to go. The basic principle is really
> > simple:
> >
> >   - Using grants to share memory for the virtqueues
> >   - Using event channels as a queue notification mechanism
> >   - All state handling and other information exchange (like number of
> >     queues, grant refs, event channel numbers etc.) is done through the
> >     Xenbus.
> >
> > On the Linux side, this is implemented as a kernel module. No patches
> > to the kernel itself (apart from the ones I sent in earlier) or to Xen
> > itself are required.
> >
> >> So far, there is an RFC to implement virtio-mmio for Xen on Arm
> >
> > I did not see that before. Also, I'm not familiar with the ioreq
> > mechanism. But from skimming the patch, it seems like it achieves the
> > same thing (dynamic creation of virtio devices at runtime). Is that
> > right?
> 
> I am not aware of any mechanism with virtio-mmio to notify a new device
> after the OS booted. But virtio-pci should allow you to do as as this is
> just a PCI device.
> 
> However, you will still need to size the PCI hostbridge I/O region
> correctly when the domain creation. Not sure if this would be an issue
> in your use case.
> 
> >
> >> But the idea of a Xen specific transport is discussed on the mailing
> >> list time to time. It would be more secure than existing transport,
> >> but I am worried about the adoption of the transport.
> >
> > What are the security issues with the existing transport mechanisms?
> In the Xen PV model, a backend cannot access the frontend memory unless
> the latter did explictly grant.
> 
> In the default virtio-{mmio, pci} model, a backend must have full access
> to the frontend memory. This can be an issue if you don't trust your
> backend or it can get compromised.

Is 'full access' required? The virtio device implementation would essentially 
be performing DMA so and vIOMMU implementation could restrict memory access on 
a per-PCI-device basis.

> 
> > I'm quite new to the Xen community, so I have no idea about adoption
> > rates.
> >
> >> A new transport requires to modify all the OSes so they can work on
> >> Xen.
> >
> > Just to be clear: They would need to be modified in order to support
> > that mechanism, but it changes nothing about the interface between
> > hypervisor and guest.
> 
> Right, this is the first bit I am more concerned about. IMHO, the main
> goal of virtio is to allow moving from one hypervisor to another without
> (or possibly limited) changing the guest OS code.
> 
> Adding a new transport open source OS for a new transport is fairly
> easy, but it may be harder to justify for proprietary OS if it only
> benefits Xen.
> 
> That said, if we can convince other hypervisor vendors to adopt it then
> most of my concern are moot :).
> 
> >
> > However, supporting the same use case with an already existing
> > transport mechanism is more elegant than building another transport
> > mechanism specifically for that case IMO. The only technical difference
> > between my implementation and the virtio-mmio approach in actually
> > running the virtio device is that I'm using event channels for queue
> > notification while virtio-mmio uses some bytes in memory for that. I do
> > not have any measurements indicating whether or not this makes a
> > difference.
> 
> The virtio-mmio notification is potentially going to be expensive on Xen
> because the guest because a vCPU will be blocked until the I/O has been
> handled by the IOREQ server.
> 
> The notification would look like:
>      1) Frontend write in notification address
>      2) Trap in Xen
>      3) Pause the vCPU and forward the I/O to the IOREQ server (e.g.
> your virtio backend)
>      4) Schedule the domain where the IOREQ server resides
>      5) Wait for the I/O completion
>      6) Unpause the vCPU
>      7) Return to guest
> 
> We may be able to optimize as there is no need to wait for the I/O to
> complete when we notify.

Perhaps take a look at the 'bufioreq' ring in the implementation?

  Paul

> 
> Cheers,
> 
> --
> Julien Grall