Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Bertrand Marquis <Bertrand.Marquis@xxxxxxx>]

Hi Stefano,

> On 9 Apr 2022, at 02:44, Stefano Stabellini <sstabellini@xxxxxxxxxx> wrote:
> 
> On Fri, 8 Apr 2022, Stefano Stabellini wrote:
>> On Wed, 23 Mar 2022, Rahul Singh wrote:
>>> in dom0less system. This patch introduce the new feature to support the
>>> signaling between two domUs in dom0less system.
>>> 
>>> Signed-off-by: Rahul Singh <rahul.singh@xxxxxxx>
>>> ---
>>> docs/designs/dom0less-evtchn.md | 96 +++++++++++++++++++++++++++++++++
>>> 1 file changed, 96 insertions(+)
>>> create mode 100644 docs/designs/dom0less-evtchn.md
>>> 
>>> diff --git a/docs/designs/dom0less-evtchn.md 
>>> b/docs/designs/dom0less-evtchn.md
>>> new file mode 100644
>>> index 0000000000..6a1b7e8c22
>>> --- /dev/null
>>> +++ b/docs/designs/dom0less-evtchn.md
>>> @@ -0,0 +1,96 @@
>>> +# Signaling support between two domUs on dom0less system
>>> +
>>> +## Current state: Draft version
>>> +
>>> +## Proposer(s): Rahul Singh, Bertrand Marquis
>>> +
>>> +## Problem Statement:
>>> +
>>> +The goal of this work is to define a simple signaling system between Xen 
>>> guests
>>> +in dom0less systems.
>>> +
>>> +In dom0less system, we cannot make use of xenbus and xenstore that are 
>>> used in
>>> +normal systems with dynamic VMs to communicate between domains by 
>>> providing a
>>> +bus abstraction for paravirtualized drivers.
>>> +
>>> +One possible solution to implement the signaling system between domUs is 
>>> based
>>> +on event channels.
>> 
>> I suggest to reword this as follows:
>> 
>> ---
>> Dom0less guests would benefit from a statically-defined memory sharing
>> and signally system for communication. One that would be immediately
>> available at boot without any need for dynamic configurations.
>> 
>> In embedded a great variety of guest operating system kernels exist,
>> many of which don't have support for xenstore, grant table or other
>> complex drivers. Some of them are small kernel-space applications (often
>> called "baremetal", not to be confused with the term "baremetal" used in
>> datacenter which means "without hypervisors") or RTOSes.  Additionally,
>> for safety reasons, users often need to be able to configure the full
>> system statically so that it can be verified statically.
>> 
>> Event channels are very simple and can be added even to baremetal
>> applications. This proposal introduces a way to define them statically
>> to make them suitable to dom0less embedded deployments.
>> ---
>> 
>> 
>>> +## Proposal:
>>> +
>>> +Event channels are the basic primitive provided by Xen for event 
>>> notifications.
>>> +An event channel is a logical connection between 2 domains (more 
>>> specifically
>>> +between dom1,port1 and dom2,port2). They essentially store one bit of
>>> +information, the event of interest is signalled by transitioning this bit 
>>> from
>>> +0 to 1. An event is an equivalent of a hardware interrupt.
>>> +
>>> +Notifications are received by a guest via an interrupt from Xen to the 
>>> guest,
>>> +indicating when an event arrives (setting the bit). Further notifications 
>>> are
>>> +masked until the bit is cleared again. When a domain wants to wait for 
>>> data it
>>> +will block until an event arrives, and then send an event to signal that 
>>> data
>>> +has been consumed. Events are delivered asynchronously to guests and are
>>> +enqueued when the guest is not running.
>>> +
>>> +Event channel communication will be established statically between two domU
>>> +guests before unpausing the domains after domain creation. Event channel
>>> +connection information between domUs will be passed to XEN via device tree
>>> +node.
>>> +
>>> +Under the /chosen node, there needs to be sub nodes with compatible
>>> +"xen,evtchn" that descibes the event channel connection between two domUs.
>>> +
>>> +The event channel sub-node has the following properties:
>>> +
>>> +- compatible
>>> +
>>> +    "xen,evtchn"
>>> +
>>> +- xen,evtchn
>>> +
>>> +    The property is four numbers of tuples of
>>> +    (local-port-domU1,domU1-phandle,local-port-domU2,domU2-phandle) where:
>>> +
>>> +    local-port-domU1 is an integer value that will be used to allocte local
>>> +    port for domU1 to send an event notification to the remote domain.
>>> +
>>> +    domU1-phandle is a single phandle to an domain to which 
>>> local-port-domU1
>>> +    will be allocated.
>>> +
>>> +    local-port-domU2 is an integer value that will be used to allocte local
>>> +    port for domU2 to send an event notification to the remote domain.
>>> +
>>> +    domU2-phandle is a single phandle to an domain to which 
>>> local-port-domU2
>>> +    will be allocated.
>>> +
>>> +Example:
>>> +
>>> +    chosen {
>>> +        ....
>>> +
>>> +        domU1: domU1 {
>>> +            ......
>>> +        };
>>> +
>>> +        domU2: domU2 {
>>> +            ......
>>> +        };
>>> +
>>> +        evtchn@1 {
>>> +            compatible = "xen,evtchn";
>>> +            xen,evtchn = <0xa &domU1 0xb &domU2>;
>>> +        };
>>> +
>>> +        evtchn@2 {
>>> +            compatible = "xen,evtchn";
>>> +            xen,evtchn = <0xc &domU1 0xd &domU2>;
>>> +        };
>>> +    };
>> 
>> There is no need to use two evtchn nodes for this given that in device
>> tree it is entirely normal to have multiple tuplets in a property. Also,
>> it would be good to have a version number in the compatible string so
>> that we can change version in the future.
>> 
>> 1)
>>    chosen {
>>        ....
>> 
>>        domU1: domU1 {
>>            ......
>>        };
>> 
>>        domU2: domU2 {
>>            ......
>>        };
>> 
>>        evtchn {
>>            compatible = "xen,evtchn-v1";
>>            xen,evtchn = <0xa &domU1 0xb &domU2 0xc &domU1 0xd &domU2>;
>>        };
>>    };
>> 
>> 
>> I should mention that it would be also possible to use sub-nodes to
>> express this information:
>> 
>> 2)
>>        domU1: domU1 {
>>            ...
>>            /* one sub-node per local event channel */
>>            ec1: evtchn@a {
>>                compatible = "xen,evtchn-v1";
>>                /* local-evtchn link-to-foreign-evtchn */
>>                xen,evtchn = <0xa &ec3>
>>            };
>>            ec2: evtchn@c {
>>                compatible = "xen,evtchn-v1";
>>                xen,evtchn = <0xc &ec4>
>>            };
>>        };
>> 
>>        domU2: domU2 {
>>            ...
>>            ec3: evtchn@b {
>>                compatible = "xen,evtchn-v1";
>>                xen,evtchn = <0xb &ec1>
>>            };
>>            ec4: evtchn@d {
>>                compatible = "xen,evtchn-v1";
>>                xen,evtchn = <0xa &ec2>
>>            };
>>        };
>>    };
>> 
>> This format has the advantage that doesn't need a new top-level node
>> type under /chosen. That is desirable few a few reasons. Today we only
>> have domains (dom0 is legacy). In the future we might have nested
>> domains and non-Xen domains. With System Device Tree, domains are under
>> /domains instead of /chosen.
>> 
>> So normally I would argue to use the sub-node format because it doesn't
>> need a new top-level node under /chosen. However, in this case it looks
>> like the 1) format is simpler to write and also simpler to parse in Xen.
>> 
>> In 1), we would need to loop over xen,evtchn and for each tuplet we
>> would only need to fetch the foreign domid.
>> 
>> In 2), we would need to check the compatible string of every
>> "xen,evtchn-v1" node, and we would have to fetch from the phandle both
>> the remote event channel number but also the domain-id of the parent.
>> 
>> So it looks like 1) is better because it is much simpler to parse. Do
>> you agree?
> 
> [...]
>> 
>> I think this is fine in principle. Like Jan wrote, at some point we'll
>> need to specify the device tree binding to expose this information to
>> the guest.
> 
> Actually, thinking more about it, I think it is likely that the guest
> device tree bindings will include information about how it is supposed
> to be used. For instance, the domU device tree might pair an event
> channel with a shared memory region so that the domU knows that they are
> expected to be used together.
> 
> If Xen is to generate such a device tree for guests, then we need that
> information also on the host device tree too (the one given to Xen and
> discussed here.)
> 
> So, I think it would be a good idea to discuss the domU device tree
> bindings for this, as part of this design document, even if we don't
> implement it straight away.

What you mention here is actually combining 2 different solutions inside
Xen to build a custom communication solution.
My assumption here is that the user will actually create the device tree
nodes he wants to do that and we should not create guest node entries
as it would enforce some design.

If everything can be statically defined for Xen then the user can also
statically define node entries inside his guest to make use of the events
and the shared memories.

For example one might need more than one event to build a communication
system, or more than one shared memory or could build something
communicating with multiple guest thus requiring even more events and
shared memories.

Cheers
Bertrand