Re: [PATCH] xen/evtchn: Add design for static event channel signaling for domUs..

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Mon, 11 Apr 2022, Bertrand Marquis wrote:
> What you mention here is actually combining 2 different solutions inside
> Xen to build a custom communication solution.
> My assumption here is that the user will actually create the device tree
> nodes he wants to do that and we should not create guest node entries
> as it would enforce some design.
> 
> If everything can be statically defined for Xen then the user can also
> statically define node entries inside his guest to make use of the events
> and the shared memories.
> 
> For example one might need more than one event to build a communication
> system, or more than one shared memory or could build something
> communicating with multiple guest thus requiring even more events and
> shared memories.

Hi Bertrand, Rahul,

If the guests are allowed some level of dynamic discovery, this feature
is not needed. They can discover the shared memory location from the
domU device tree, then proceed to allocate evtchns as needed and tell
the other end the evtchn numbers over shared memory. I already have an
example of it here:

https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/2251030537/Xen+Shared+Memory+and+Interrupts+Between+VMs

What if the guest doesn't support device tree at runtime, like baremetal
or Zephyr? The shared memory address can be hardcoded or generated from
device tree at build time. That's no problem. Then, the event channels
can still be allocated at runtime and passed to the other end over
shared memory. That's what the example on the wikipage does.


When are static event channels actually useful? When the application
cannot allocate the event channels at runtime at all. The reason for the
restriction could be related to safety (no dynamic allocations at
runtime) or convenience (everything else is fully static, why should the
event channel numbers be dynamic?)

Given the above, I can see why there is no need to describe the static
event channel info in the domU device tree: static event channels are
only useful in fully static configurations, and in those configurations
the domU device tree dynamically generated by Xen is not needed. I can
see where you are coming from.


The workflow that we have been trying to enable with the System Device
Tree effort (System Device Tree is similar to a normal Device Tree plus
the xen,domains nodes) is the following:

S-DT ---[lopper]---> Linux DT
                L--> Zephyr DT ---[Zephyr build]---> Zephyr .h files

S-DT contains all the needed information for both the regular Linux DT
generation and also the Zephyr/RTOS/baremetal header files generation,
that happens at build time.

S-DT is not the same as the Xen device tree, but so far it has been
conceptually and practically similar. I always imagine that the bindings
we have in Xen we'll also have corresponding bindings in System Device
Tree.

For this workflow to work S-DT needs all the info so that both Linux DT
and Zephyr DT and Zephyr .h files can be generated.

Does this proposal contain enough information so that Zephyr .h files
could be statically generated with the event channel numbers and static
shared memory regions addresses?

I am not sure. Maybe not?


It is possible that the shared memory usage is so application specific
that there is no point in even talking about it. But I think that
introducing a simple bundle of both event channels and shared memory
would help a lot.

Something like the following in the Xen device tree would be enough to
specify an arbitrary number of event channels connected with the same
domains sharing the memory region.

It looks like that if we did the below, we would carry a lot more useful
information compared to the original proposal alone. We could add a
similar xen,notificaiton property to the domU reserved-memory region in
device tree generated by Xen for consistency, so that everything
available to the domU is described fully in device tree.


    domU1 {
        compatible = "xen,domain";

        /* one sub-node per local event channel */
        ec1: evtchn@1 {
            compatible = "xen,evtchn-v1";
            /* local-evtchn link-to-foreign-evtchn */
            xen,evtchn = <0x1 &ec3>
        };
        ec2: evtchn@2 {
            compatible = "xen,evtchn-v1";
            xen,evtchn = <0x2 &ec4>
        };
        /*
         * shared memory region between DomU1 and DomU2.
         */
        domU1-shared-mem@50000000 {
            compatible = "xen,domain-shared-memory-v1";
            xen,shm-id = <0x1>;
            xen,shared-mem = <0x50000000 0x20000000 0x60000000>;
            /* this is new */
            xen,notification = <&ec1 &ec2>;
        }
    };

    domU2 {
        compatible = "xen,domain";

        /* one sub-node per local event channel */
        ec3: evtchn@3 {
            compatible = "xen,evtchn-v1";
            /* local-evtchn link-to-foreign-evtchn */
            xen,evtchn = <0x3 &ec1>
        };
        ec4: evtchn@4 {
            compatible = "xen,evtchn-v1";
            xen,evtchn = <0x4 &ec2>
        };
        /*
         * shared memory region between domU1 and domU2.
         */
        domU2-shared-mem@50000000 {
            compatible = "xen,domain-shared-memory-v1";
            xen,shm-id = <0x1>;
            xen,shared-mem = <0x50000000 0x20000000 0x70000000>;
            /* this is new */
            xen,notification = <&ec3 &ec4>;
        }
    };



The good thing about this is that:

- it is very flexible
- nothing to do in this series, except switching to the
  one-subnode-per-evtchn model, which we called 2) in the previous email
- there were good reasons to use the one-subnode-per-evtchn model anyway
- the xen,notification property can be added later without issues, after 
Penny's series

There are a couple of ways to implement the xen,notification property
but we don't need to discuss them now.


Short Summary
------------
I think it is fine to only introduce the Xen device tree binding for
static event channels without domU binding, but I prefer if we switched
to using proposal 2) "one subnode per event channel".