Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Wed, 23 Mar 2022, Rahul Singh wrote:
> in dom0less system. This patch introduce the new feature to support the
> signaling between two domUs in dom0less system.
> 
> Signed-off-by: Rahul Singh <rahul.singh@xxxxxxx>
> ---
>  docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
>  1 file changed, 96 insertions(+)
>  create mode 100644 docs/designs/dom0less-evtchn.md
> 
> diff --git a/docs/designs/dom0less-evtchn.md b/docs/designs/dom0less-evtchn.md
> new file mode 100644
> index 0000000000..6a1b7e8c22
> --- /dev/null
> +++ b/docs/designs/dom0less-evtchn.md
> @@ -0,0 +1,96 @@
> +# Signaling support between two domUs on dom0less system
> +
> +## Current state: Draft version
> +
> +## Proposer(s): Rahul Singh, Bertrand Marquis
> +
> +## Problem Statement:
> +
> +The goal of this work is to define a simple signaling system between Xen 
> guests
> +in dom0less systems.
> +
> +In dom0less system, we cannot make use of xenbus and xenstore that are used 
> in
> +normal systems with dynamic VMs to communicate between domains by providing a
> +bus abstraction for paravirtualized drivers.
> +
> +One possible solution to implement the signaling system between domUs is 
> based
> +on event channels.

I suggest to reword this as follows:

---
Dom0less guests would benefit from a statically-defined memory sharing
and signally system for communication. One that would be immediately
available at boot without any need for dynamic configurations.

In embedded a great variety of guest operating system kernels exist,
many of which don't have support for xenstore, grant table or other
complex drivers. Some of them are small kernel-space applications (often
called "baremetal", not to be confused with the term "baremetal" used in
datacenter which means "without hypervisors") or RTOSes.  Additionally,
for safety reasons, users often need to be able to configure the full
system statically so that it can be verified statically.

Event channels are very simple and can be added even to baremetal
applications. This proposal introduces a way to define them statically
to make them suitable to dom0less embedded deployments.
---


> +## Proposal:
> +
> +Event channels are the basic primitive provided by Xen for event 
> notifications.
> +An event channel is a logical connection between 2 domains (more specifically
> +between dom1,port1 and dom2,port2). They essentially store one bit of
> +information, the event of interest is signalled by transitioning this bit 
> from
> +0 to 1. An event is an equivalent of a hardware interrupt.
> +
> +Notifications are received by a guest via an interrupt from Xen to the guest,
> +indicating when an event arrives (setting the bit). Further notifications are
> +masked until the bit is cleared again. When a domain wants to wait for data 
> it
> +will block until an event arrives, and then send an event to signal that data
> +has been consumed. Events are delivered asynchronously to guests and are
> +enqueued when the guest is not running.
> +
> +Event channel communication will be established statically between two domU
> +guests before unpausing the domains after domain creation. Event channel
> +connection information between domUs will be passed to XEN via device tree
> +node.
> +
> +Under the /chosen node, there needs to be sub nodes with compatible
> +"xen,evtchn" that descibes the event channel connection between two domUs.
> +
> +The event channel sub-node has the following properties:
> +
> +- compatible
> +
> +    "xen,evtchn"
> +
> +- xen,evtchn
> +
> +    The property is four numbers of tuples of
> +    (local-port-domU1,domU1-phandle,local-port-domU2,domU2-phandle) where:
> +
> +    local-port-domU1 is an integer value that will be used to allocte local
> +    port for domU1 to send an event notification to the remote domain.
> +
> +    domU1-phandle is a single phandle to an domain to which local-port-domU1
> +    will be allocated.
> +
> +    local-port-domU2 is an integer value that will be used to allocte local
> +    port for domU2 to send an event notification to the remote domain.
> +
> +    domU2-phandle is a single phandle to an domain to which local-port-domU2
> +    will be allocated.
> +
> +Example:
> +
> +    chosen {
> +        ....
> +
> +        domU1: domU1 {
> +            ......
> +        };
> +
> +        domU2: domU2 {
> +            ......
> +        };
> +
> +        evtchn@1 {
> +            compatible = "xen,evtchn";
> +            xen,evtchn = <0xa &domU1 0xb &domU2>;
> +        };
> +
> +        evtchn@2 {
> +            compatible = "xen,evtchn";
> +            xen,evtchn = <0xc &domU1 0xd &domU2>;
> +        };
> +    };

There is no need to use two evtchn nodes for this given that in device
tree it is entirely normal to have multiple tuplets in a property. Also,
it would be good to have a version number in the compatible string so
that we can change version in the future.

1)
    chosen {
        ....

        domU1: domU1 {
            ......
        };

        domU2: domU2 {
            ......
        };

        evtchn {
            compatible = "xen,evtchn-v1";
            xen,evtchn = <0xa &domU1 0xb &domU2 0xc &domU1 0xd &domU2>;
        };
    };


I should mention that it would be also possible to use sub-nodes to
express this information:

2)
        domU1: domU1 {
            ...
            /* one sub-node per local event channel */
            ec1: evtchn@a {
                compatible = "xen,evtchn-v1";
                /* local-evtchn link-to-foreign-evtchn */
                xen,evtchn = <0xa &ec3>
            };
            ec2: evtchn@c {
                compatible = "xen,evtchn-v1";
                xen,evtchn = <0xc &ec4>
            };
        };

        domU2: domU2 {
            ...
            ec3: evtchn@b {
                compatible = "xen,evtchn-v1";
                xen,evtchn = <0xb &ec1>
            };
            ec4: evtchn@d {
                compatible = "xen,evtchn-v1";
                xen,evtchn = <0xa &ec2>
            };
        };
    };

This format has the advantage that doesn't need a new top-level node
type under /chosen. That is desirable few a few reasons. Today we only
have domains (dom0 is legacy). In the future we might have nested
domains and non-Xen domains. With System Device Tree, domains are under
/domains instead of /chosen.

So normally I would argue to use the sub-node format because it doesn't
need a new top-level node under /chosen. However, in this case it looks
like the 1) format is simpler to write and also simpler to parse in Xen.

In 1), we would need to loop over xen,evtchn and for each tuplet we
would only need to fetch the foreign domid.

In 2), we would need to check the compatible string of every
"xen,evtchn-v1" node, and we would have to fetch from the phandle both
the remote event channel number but also the domain-id of the parent.

So it looks like 1) is better because it is much simpler to parse. Do
you agree?


> +In above example two event channel comunication will be established between
> +domU1 and domU2.
> +
> +    domU1 (port 0xa) <-----------------> domU2 (port 0xb)
> +    domU1 (port 0xc) <-----------------> domU2 (port 0xd)
> +
> +domU1 and domU2 can send the signal to remote domain via hypercall
> +EVTCHNOP_send(.) on local port.

I think this is fine in principle. Like Jan wrote, at some point we'll
need to specify the device tree binding to expose this information to
the guest.