Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Bertrand Marquis <Bertrand.Marquis@xxxxxxx>]

Hi Stefano,

> On 14 Apr 2022, at 02:14, Stefano Stabellini <sstabellini@xxxxxxxxxx> wrote:
> 
> On Mon, 11 Apr 2022, Bertrand Marquis wrote:
>> What you mention here is actually combining 2 different solutions inside
>> Xen to build a custom communication solution.
>> My assumption here is that the user will actually create the device tree
>> nodes he wants to do that and we should not create guest node entries
>> as it would enforce some design.
>> 
>> If everything can be statically defined for Xen then the user can also
>> statically define node entries inside his guest to make use of the events
>> and the shared memories.
>> 
>> For example one might need more than one event to build a communication
>> system, or more than one shared memory or could build something
>> communicating with multiple guest thus requiring even more events and
>> shared memories.
> 
> Hi Bertrand, Rahul,
> 
> If the guests are allowed some level of dynamic discovery, this feature
> is not needed. They can discover the shared memory location from the
> domU device tree, then proceed to allocate evtchns as needed and tell
> the other end the evtchn numbers over shared memory. I already have an
> example of it here:
> 
> https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/2251030537/Xen+Shared+Memory+and+Interrupts+Between+VMs
> 
> What if the guest doesn't support device tree at runtime, like baremetal
> or Zephyr? The shared memory address can be hardcoded or generated from
> device tree at build time. That's no problem. Then, the event channels
> can still be allocated at runtime and passed to the other end over
> shared memory. That's what the example on the wikipage does.
> 
> 
> When are static event channels actually useful? When the application
> cannot allocate the event channels at runtime at all. The reason for the
> restriction could be related to safety (no dynamic allocations at
> runtime) or convenience (everything else is fully static, why should the
> event channel numbers be dynamic?)

An other use case here is dom0less: you cannot have dom0 create them.

> 
> Given the above, I can see why there is no need to describe the static
> event channel info in the domU device tree: static event channels are
> only useful in fully static configurations, and in those configurations
> the domU device tree dynamically generated by Xen is not needed. I can
> see where you are coming from.
> 
> 
> The workflow that we have been trying to enable with the System Device
> Tree effort (System Device Tree is similar to a normal Device Tree plus
> the xen,domains nodes) is the following:
> 
> S-DT ---[lopper]---> Linux DT
>                L--> Zephyr DT ---[Zephyr build]---> Zephyr .h files
> 
> S-DT contains all the needed information for both the regular Linux DT
> generation and also the Zephyr/RTOS/baremetal header files generation,
> that happens at build time.
> 
> S-DT is not the same as the Xen device tree, but so far it has been
> conceptually and practically similar. I always imagine that the bindings
> we have in Xen we'll also have corresponding bindings in System Device
> Tree.
> 
> For this workflow to work S-DT needs all the info so that both Linux DT
> and Zephyr DT and Zephyr .h files can be generated.
> 
> Does this proposal contain enough information so that Zephyr .h files
> could be statically generated with the event channel numbers and static
> shared memory regions addresses?
> 
> I am not sure. Maybe not?

Yes it should be possible to have all infos as the integrator will setup the
system and will decide upfront the address and the event(s) number(s).

> 
> 
> It is possible that the shared memory usage is so application specific
> that there is no point in even talking about it. But I think that
> introducing a simple bundle of both event channels and shared memory
> would help a lot.
> 
> Something like the following in the Xen device tree would be enough to
> specify an arbitrary number of event channels connected with the same
> domains sharing the memory region.
> 
> It looks like that if we did the below, we would carry a lot more useful
> information compared to the original proposal alone. We could add a
> similar xen,notificaiton property to the domU reserved-memory region in
> device tree generated by Xen for consistency, so that everything
> available to the domU is described fully in device tree.
> 
> 
>    domU1 {
>        compatible = "xen,domain";
> 
>        /* one sub-node per local event channel */
>        ec1: evtchn@1 {
>            compatible = "xen,evtchn-v1";
>            /* local-evtchn link-to-foreign-evtchn */
>            xen,evtchn = <0x1 &ec3>
>        };
>        ec2: evtchn@2 {
>            compatible = "xen,evtchn-v1";
>            xen,evtchn = <0x2 &ec4>
>        };
>        /*
>         * shared memory region between DomU1 and DomU2.
>         */
>        domU1-shared-mem@50000000 {
>            compatible = "xen,domain-shared-memory-v1";
>            xen,shm-id = <0x1>;
>            xen,shared-mem = <0x50000000 0x20000000 0x60000000>;
>            /* this is new */
>            xen,notification = <&ec1 &ec2>;
>        }
>    };
> 
>    domU2 {
>        compatible = "xen,domain";
> 
>        /* one sub-node per local event channel */
>        ec3: evtchn@3 {
>            compatible = "xen,evtchn-v1";
>            /* local-evtchn link-to-foreign-evtchn */
>            xen,evtchn = <0x3 &ec1>
>        };
>        ec4: evtchn@4 {
>            compatible = "xen,evtchn-v1";
>            xen,evtchn = <0x4 &ec2>
>        };
>        /*
>         * shared memory region between domU1 and domU2.
>         */
>        domU2-shared-mem@50000000 {
>            compatible = "xen,domain-shared-memory-v1";
>            xen,shm-id = <0x1>;
>            xen,shared-mem = <0x50000000 0x20000000 0x70000000>;
>            /* this is new */
>            xen,notification = <&ec3 &ec4>;
>        }
>    };

Few remarks/questions on this:
- this is not a shared memory anymore as you add a notification system to it
- what if someone wants to use only a shared memory, or an event, what should 
xen do ?
- in xen device tree, how do you associate the event with the shared memory ?

> 
> 
> 
> The good thing about this is that:
> 
> - it is very flexible
> - nothing to do in this series, except switching to the
>  one-subnode-per-evtchn model, which we called 2) in the previous email
> - there were good reasons to use the one-subnode-per-evtchn model anyway
> - the xen,notification property can be added later without issues, after 
> Penny's series
> 
> There are a couple of ways to implement the xen,notification property
> but we don't need to discuss them now.

I think there is something to do here but we need a bit more discussion and 
this can be done later.
Right now I am not quite sure we will not add something that will end up not 
being used.

> 
> 
> Short Summary
> ------------
> I think it is fine to only introduce the Xen device tree binding for
> static event channels without domU binding, but I prefer if we switched
> to using proposal 2) "one subnode per event channel".

I will let Rahul answer on that.

Cheers
Bertrand