Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Mon, 11 Apr 2022, Julien Grall wrote:
> On 09/04/2022 02:00, Stefano Stabellini wrote:
> > On Wed, 23 Mar 2022, Rahul Singh wrote:
> > > in dom0less system. This patch introduce the new feature to support the
> > > signaling between two domUs in dom0less system.
> > > 
> > > Signed-off-by: Rahul Singh <rahul.singh@xxxxxxx>
> > > ---
> > >   docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
> > >   1 file changed, 96 insertions(+)
> > >   create mode 100644 docs/designs/dom0less-evtchn.md
> > > 
> > > diff --git a/docs/designs/dom0less-evtchn.md
> > > b/docs/designs/dom0less-evtchn.md
> > > new file mode 100644
> > > index 0000000000..6a1b7e8c22
> > > --- /dev/null
> > > +++ b/docs/designs/dom0less-evtchn.md
> > > @@ -0,0 +1,96 @@
> > > +# Signaling support between two domUs on dom0less system
> > > +
> > > +## Current state: Draft version
> > > +
> > > +## Proposer(s): Rahul Singh, Bertrand Marquis
> > > +
> > > +## Problem Statement:
> > > +
> > > +The goal of this work is to define a simple signaling system between Xen
> > > guests
> > > +in dom0less systems.
> > > +
> > > +In dom0less system, we cannot make use of xenbus and xenstore that are
> > > used in
> > > +normal systems with dynamic VMs to communicate between domains by
> > > providing a
> > > +bus abstraction for paravirtualized drivers.
> > > +
> > > +One possible solution to implement the signaling system between domUs is
> > > based
> > > +on event channels.
> > 
> > I suggest to reword this as follows:
> > 
> > ---
> > Dom0less guests would benefit from a statically-defined memory sharing
> > and signally system for communication. One that would be immediately
> > available at boot without any need for dynamic configurations.
> > 
> > In embedded a great variety of guest operating system kernels exist,
> > many of which don't have support for xenstore, grant table or other
> > complex drivers. Some of them are small kernel-space applications (often
> > called "baremetal", not to be confused with the term "baremetal" used in
> > datacenter which means "without hypervisors") or RTOSes.  Additionally,
> > for safety reasons, users often need to be able to configure the full
> > system statically so that it can be verified statically.
> > 
> > Event channels are very simple and can be added even to baremetal
> > applications. This proposal introduces a way to define them statically
> > to make them suitable to dom0less embedded deployments.
> > ---
> > 
> > 
> > > +## Proposal:
> > > +
> > > +Event channels are the basic primitive provided by Xen for event
> > > notifications.
> > > +An event channel is a logical connection between 2 domains (more
> > > specifically
> > > +between dom1,port1 and dom2,port2). They essentially store one bit of
> > > +information, the event of interest is signalled by transitioning this bit
> > > from
> > > +0 to 1. An event is an equivalent of a hardware interrupt.
> > > +
> > > +Notifications are received by a guest via an interrupt from Xen to the
> > > guest,
> > > +indicating when an event arrives (setting the bit). Further notifications
> > > are
> > > +masked until the bit is cleared again. When a domain wants to wait for
> > > data it
> > > +will block until an event arrives, and then send an event to signal that
> > > data
> > > +has been consumed. Events are delivered asynchronously to guests and are
> > > +enqueued when the guest is not running.
> > > +
> > > +Event channel communication will be established statically between two
> > > domU
> > > +guests before unpausing the domains after domain creation. Event channel
> > > +connection information between domUs will be passed to XEN via device
> > > tree
> > > +node.
> > > +
> > > +Under the /chosen node, there needs to be sub nodes with compatible
> > > +"xen,evtchn" that descibes the event channel connection between two
> > > domUs.
> > > +
> > > +The event channel sub-node has the following properties:
> > > +
> > > +- compatible
> > > +
> > > +    "xen,evtchn"
> > > +
> > > +- xen,evtchn
> > > +
> > > +    The property is four numbers of tuples of
> > > +    (local-port-domU1,domU1-phandle,local-port-domU2,domU2-phandle)
> > > where:
> > > +
> > > +    local-port-domU1 is an integer value that will be used to allocte
> > > local
> > > +    port for domU1 to send an event notification to the remote domain.
> > > +
> > > +    domU1-phandle is a single phandle to an domain to which
> > > local-port-domU1
> > > +    will be allocated.
> > > +
> > > +    local-port-domU2 is an integer value that will be used to allocte
> > > local
> > > +    port for domU2 to send an event notification to the remote domain.
> > > +
> > > +    domU2-phandle is a single phandle to an domain to which
> > > local-port-domU2
> > > +    will be allocated.
> > > +
> > > +Example:
> > > +
> > > +    chosen {
> > > +        ....
> > > +
> > > +        domU1: domU1 {
> > > +            ......
> > > +        };
> > > +
> > > +        domU2: domU2 {
> > > +            ......
> > > +        };
> > > +
> > > +        evtchn@1 {
> > > +            compatible = "xen,evtchn";
> > > +            xen,evtchn = <0xa &domU1 0xb &domU2>;
> > > +        };
> > > +
> > > +        evtchn@2 {
> > > +            compatible = "xen,evtchn";
> > > +            xen,evtchn = <0xc &domU1 0xd &domU2>;
> > > +        };
> > > +    };
> > 
> > There is no need to use two evtchn nodes for this given that in device
> > tree it is entirely normal to have multiple tuplets in a property. Also,
> > it would be good to have a version number in the compatible string so
> > that we can change version in the future.
> > 
> > 1)
> >      chosen {
> >          ....
> > 
> >          domU1: domU1 {
> >              ......
> >          };
> > 
> >          domU2: domU2 {
> >              ......
> >          };
> > 
> >          evtchn {
> >              compatible = "xen,evtchn-v1";
> >              xen,evtchn = <0xa &domU1 0xb &domU2 0xc &domU1 0xd &domU2>;
> >          };
> >      };
> > 
> > 
> > I should mention that it would be also possible to use sub-nodes to
> > express this information:
> > 
> > 2)
> >          domU1: domU1 {
> >              ...
> >              /* one sub-node per local event channel */
> >              ec1: evtchn@a {
> >                  compatible = "xen,evtchn-v1";
> >                  /* local-evtchn link-to-foreign-evtchn */
> >                  xen,evtchn = <0xa &ec3>
> >              };
> >              ec2: evtchn@c {
> >                  compatible = "xen,evtchn-v1";
> >                  xen,evtchn = <0xc &ec4>
> >              };
> >          };
> > 
> >          domU2: domU2 {
> >              ...
> >              ec3: evtchn@b {
> >                  compatible = "xen,evtchn-v1";
> >                  xen,evtchn = <0xb &ec1>
> >              };
> >              ec4: evtchn@d {
> >                  compatible = "xen,evtchn-v1";
> >                  xen,evtchn = <0xd &ec2>
> >              };
> >          };
> >      };
> 
> As for 1), you could combine all the ports in one node.

I thought about it but I couldn't come up with a way to do that which
retains the simplicity of this example. The problem is that in device
tree you can only link to nodes, not to individual properties. So I
think we would have to have separate nodes for each event channel so
that we could separately link to them. 

Otherwise, we would have to add the foreign event channel number in
addition to the link to be able to distinguish them. And that would
result in duplicated information. E.g.:

       domU1: domU1 {
           ...
           /* one sub-node per local event channel */
           ec1: evtchn@a {
               compatible = "xen,evtchn-v1";
               /* local-evtchn link-to-foreign foreign-evtchn */
               xen,evtchn = <0xa &ec2 0xa 0xc &ec2 0xd>
           };
       };

       domU2: domU2 {
           ...
           ec2: evtchn@b {
               compatible = "xen,evtchn-v1";
               xen,evtchn = <0xb &ec1 0xa 0xd &ec1 0xc>
           };
       };