Re: [RFC PATCH] xen/arm: Rebranding dom0less feature

["Daniel P. Smith" <dpsmith@xxxxxxxxxxxxxxxxxxxx>]

On 7/8/23 14:08, Stefano Stabellini wrote:
> On Sat, 8 Jul 2023, Rich Persaud wrote:
>> On Jul 8, 2023, at 03:29, Luca Fancellu <luca.fancellu@xxxxxxx> wrote:
>>> 
>>>>>>
>>>>>> Instead, the use case configurations should themselves be 
describable.
>>>>>
>>>>> Thanks Christopher, Daniel and all!
>>>>>
>>>>> So if I understand correctly, you are in favor if renaming 
Dom0less to
>>>>> Hyperlaunch throughout the Xen codebase? And we need a 
clarification of
>>>>> the docs/, especially docs/features/dom0less.pandoc?
>>>>
>>>> Christopher wrote:
>>>>>> = Community resourcing
>>>>
>>>> Note the pre-requisite work items for upstream Xen, listed under 
"Community Resourcing", to merge code for Hyperlaunch common interfaces 
and test cases, with docs on configuration of Hyperlaunch to deliver 
functionality for dom0less use cases.
>>>
>>> Are you saying that before renaming the “dom0less” feature, we 
should wait for it to be ported to the common code?
>>
>> Why "wait"? In what timeframe do you expect dom0less to use 
Hyperlaunch code?
>>
>> Can kernel component foo adopt the name of kernel component bar 
without code change?
>>
>> Can dom0less stakeholders derive Hyperlaunch benefits without using 
Hyperlaunch code?
>
>
> I think Rich is saying that before using the same name we should make
> sure that the interfaces and features are actually comparable and maybe
> even "compatible". I think that is very reasonable. Rich, did I
> understand correctly?

Essentially, yes this is what is being sought here. This does not mean 
that the full capability has to be present for the adoption of the 
common name, but that it can be accomplished through a path of integration.
> The Hyperlaunch (x86) code is not yet upstream, but the design document
> that describes the device tree interface shows an interface that is very
> similar, almost compatible, with today's dom0less (ARM) device tree
> interface.

I would caution the use of the current in-tree document as it is today, 
it was posted under the design folder as it was only the design and not 
burdened with the realities of implementation. Along the path of v1 
implementation, recent PVH expansion, and roles update, each have 
required updates to the design which are not yet included in the in-tree 
docs.
To address, this the plan below starts with the documentation patch 
posted in v1 of the hyperlaunch series:

https://lists.xenproject.org/archives/html/xen-devel/2022-07/msg00353.html

and will contain updates for community feedback received:


https://lists.xenproject.org/archives/html/xen-devel/2022-07/msg01015.html

and development work that has been done for PVH and roles.

> The structure of the device tree information is the same. Going through
> it I could only spot only tiny differences:
> - top level node is "hypervisor" instead of "chosen"
> - "module-addr" instead of "reg"
> - "module,kernel" instead of "multiboot,kernel"
> - "module,ramdisk" instead of "multiboot,ramdisk"
>
> The rest is the same. If we sort out these small differences one way or
> the other then the resulting interface should actually be fully
> compatible and we could reuse the existing Dom0less (ARM) code to parse
> an HyperLaunch (x86) configuration.
>
> The top level node is not a problem. We could easily deal with both
> "hypervisor" and also "chosen". Or we could pick a third different name
> for both: "domains" which is the one used by System Device Tree.
>
> I think we should rename "module-addr" to "reg" in the hyperlaunch
> design document. I don't think it would have any effect on the existing
> hyperlaunch (x86) code and usage because direct addresses are typically
> not used on x86.
>
> "module,kernel" and "module,ramdisk": we could either get rid of them in
> favor of "multiboot,kernel" and "multiboot,ramdisk", or we could add
> "module,kernel" and "module,ramdisk" as alternative aliases in the
> existing dom0less (ARM) code. We already have "xen,linux-zimage" and
> "xen,linux-initrd" as aliases so it is not a problem.
>
>
> Also, I do think that Dom0less stakeholders would benefit from
> Hyperlaunch code such as Dom0's reduction of privilege. Things like
> "permissions" and "functions" of the Hyperlauch device tree interface
> design document.

The roles work takes that to its final conclusion, from which everyone 
will benefit.
> So, my opinion is that we should go ahead with dom0less->hyperlaunch
> rename but we should also try to make the two device tree interfaces
> compatible, sorting out the small differences above. That would help a
> lot in terms of documentation and tooling. It would be ideal if things
> like ImageBuilder worked equally well for Hyperlaunch (x86) and Dom0less
> (ARM).

Let me build on the plan laid out by Christopher, that detailed arriving 
at the completion for hyperlaunch, and provide a set of steps to arrive 
at a new short-term milestone to a larger roadmap. The intent being to 
arrive at the immediate desire of the community to see dom0less renamed. 
For the longer term, as Christopher was alluding, there is still a 
significant amount of work to be done to deliver one of the biggest 
market differentiating capabilities for Xen in some time.
As Stefano has acknowledged, hyperlaunch is a concept that is beyond 
domain construction and that is, or will be, embodied by a set of 
interfaces and capabilities. Considering dom0less as it is implemented 
today, does not meet the former and in spirit meets the latter. 
Compounding this is that dom0less is a supported feature today, with its 
own defined interface, and a user base that is using that interface, at 
least to some degree AIUI.
We have a responsibility to the dom0less user base and future 
hyperlaunch users with requirements based on hyperlaunch design docs and 
presentations.  As such, any action taken should be done so under a 
larger roadmap of adding the complete hyperlaunch capability to Xen.
With the initial funding by AMD, the first milestone was to be moving 
Xen on to a common representation of boot material provided to the 
hypervisor. As a result of this discussion, I would like to put forth a 
new nearer term milestone of incorporating dom0less under hyperlaunch.
The naming suggestions by the community are greatly appreciated, and I 
do not want to seem dismissive of clear offers of help and assistance. 
This area is something Christopher and I discussed at length during the 
drafting of the hyperlaunch design. Something we arrived at is that 
there is only a single top level feature, which is hyperlaunch. The 
hyperlaunch feature itself will provide multiple means to configure how 
a launch will occur, one could even consider them modes of operation.
Reviewing the design doc, you will see that an initial attempt to 
categorize them was done, splitting them into a dynamic mode and static 
mode. Under these modes were multiple use cases. With that, we did not 
mean to limit the hyperlaunch modes to strictly those two, and thus 
other modes are more than reasonable to add. As such, my suggestion 
would be the introduction of the dom0less compatibility mode for 
hyperlaunch. Its very definition is a mode of hyperlaunch that supports 
the "legacy" dom0less configuration interface. This approach allows 
dom0less to effectively become hyperlaunch, deconflicts the fact that 
hyperlaunch proper has a different interface than dom0less, and provides 
a clean roadmap for migration to existing dom0less users.
To provide a concrete, measurable set of steps to achieve the dom0less 
merging milestone, I will lay out the approach as three patch series 
that will be a collaboration by the community. Each will need to be 
submitted, reviewed and merged into the tree, with the end being the 
existing dom0less becoming hyperlaunch's dom0less compatibility mode.
== Patch Series 1 (Resourced by Apertus)

The goal of this series is to properly introduce the hyperlaunch 
"feature" in to Xen. This series would be submitted by myself and will 
consist of two patches derived from the original hyperlaunch v1 series. 
The first patch is the docs patch that updates the hyperlaunch device 
tree design to reflect review feedback and updates for PVH and roles, 
mentioned above.
The second patch will start with the v1 series patch that moved the fdt 
parsing helpers into common:
https://lists.xenproject.org/archives/html/xen-devel/2022-07/msg00352.html

The patch currently moves common FDT parsing to common/fdt.c, it will be 
updated to add this path to DEVICE TREE in MAINTAINERS. As part of 
updating MAINTAINERS, there will be the addition of HYPERLAUNCH, which 
will own common/domain-builder/ and doc/design/launch paths. As such, 
this will effectively be a declaration of the top level hyperlaunch 
feature, with this as an interface, and establish the maintainers of the 
feature.
== Patch Series 2 (Requesting resourcing by Arm)

The goal of this series would be to move the dom0less device tree 
parsing under hyperlaunch. We would respectfully request a member of the 
Arm community to volunteer to take ownership and steward the series 
through submission and review process.
The implementation of this series would see dom0less device tree parsing 
to use, and expand if necessary, the common/fdt parsing helpers. I 
personally would see this logic move under common/domain-builder using a 
file name that would not collide with the files from the hyperlaunch v1 
series, e.g.  a suggestion would be fdt-dom0less.c.
As for ownership of that file, I would suggest the addition of a 
HYPERLAUNCH DOM0LESS COMPATIBILITY be added to MAINTAINERS with the 
appropriate Arm community members, but understand and willing to 
consider the position that it falls under HYPERLAUNCH. The purpose of 
using HYPERLAUNCH DOM0LESS COMPATIBILITY will be to provide a means to 
signal the retirement of dom0less compatibility mode at some future point.
== Patch Series 3 (Resourced by Apertus)

The goal of this series will be to formalize hyperlaunch dom0less 
compatibility mode. The series would be submitted by myself and/or 
Christopher. It will consist of documentation patches that will add a 
doc/features/hyperlaunch.rst and a 
doc/features/hyperlaunch/dom0less-compatiblilty.rst. The hyperlaunch.rst 
path will fall under HYPERLAUNCH, while dom0less-compatiblilty.rst will 
fall under HYPERLAUNCH DOM0LESS COMPATIBILITY. This will also see 
DOM0LESS in SUPPORTED.md be renamed to HYPERLAUNCH.
In summary, I would like to reiterate points made by Rich and 
Christopher. The motivation and concept of the hypervisor 
differentiating capability that is hyperlaunch goes back to the 2012 
domain builder work that was a companion to the hardware domain work.
Since then, there have been a few of us in the OpenXT community that 
desired to make this an integral part of Xen. Internal OpenXT 
discussions in 2017-2018 along with the announcement of dom0less 
inspired confidence there were in fact other uses cases and interest in 
Xen gaining such an integrated capability.
A fact that should not go overlooked or undervalued is that hyperlauch 
would not exist today without the extremely generous support StarLab 
provided to sponsor Christopher and I to do the R&D, proof of viability, 
and a full working prototype that provided the basis for the v1 series. 
This was not a minor investment on their part, and taking the design to 
completion requires further substantial investment. Due to an 
acquisition imposed shift in market focus, StarLab has since stepped away.
Fortunately, and for which we are immensely appreciative, AMD has 
recently stepped up to fund some hyperlaunch work items. With a growing 
number of hyperlaunch use cases, I would be happy to help those with 
budgetary influence to communicate the business benefits of investment 
in open work items, targeting specific launch configurations, safety and 
security properties.
The amount of work to be done goes beyond just parallel domain 
construction, there are tangential capabilities that need updating and 
incorporation. For instance, a topic discussed during the summit, VPCI 
and device assignment at/during hypervisor startup. Anyone looking or 
willing to provide purely financial support, feel free to reach out 
directly to Christopher and me, as we have a few avenues to enable the 
flow of funds.
Attached are candidate work items for funding and resourcing of 
hyperlaunch integration, building upon the proposed Apertus and Arm 
patch series for dom0less rename and migration to hyperlaunch common 
interfaces.
Lastly, thank you to everyone who has taken the time to engage and 
collaborate on how to resolve the task at hand!
V/r,
Daniel P. Smith

===

Here are outlines for the next development items appropriate for funding 
for progress on hyperlaunch integration, following on from the three 
series described above.
== Work Item: initial x86 Hyperlaunch SUPPORT and launch of dom0 
(Resourcing TBD)
The goal is to enable initial hyperlaunch SUPPORT on x86, building upon 
from the work in the posted v3 series of hyperlaunch.
Development proceeds from patches 9-12 of the v1 hyperlaunch series, to 
add boot with minimal construction of a classic dom0 from a device tree 
configuration. The Hyperlaunch SUPPORT statement would add x86 and Xen 
nightly CI and release-gating test configurations would be extended to 
include hyperlaunch on x86.

== Work Item: Hyperlaunch XSM policy for guests (Resourcing TBD)

The goal is to ensure that security policies of guests started via 
hyperlaunch are accurately aligned with the configured functions of each 
guest, which would allow for reduction of privilege of the dom0 in 
dom0less systems.
A new XSM policy with granular permissions for domain functions, aligned 
with the roles work, would be integrated into the hypervisor.  It would 
define new domain security labels appropriate for assigning to domUs, 
matching privileges for the domain functions that can be assigned with 
hyperlaunch, and allowing for more expressive policy control than is 
expressible with dom0less.

== Work Item: Hyperlaunch of Arm guests SUPPORT (Resourcing TBD)

The goal of this effort is to enable hyperlaunch of Arm guests.

It would enable use of the architecture-common Hyperlaunch Device Tree 
format on Arm, providing the forward migration path from the Dom0less 
format, to the new format with additional flexibility for guest 
construction, including control over console privilege assignment and 
XSM label control.
It would achieve the objective of using common cross-architecture boot 
structures and community-maintained common code.
Development would proceed from patches 9-10 of the v1 hyperlaunch 
series, to add boot with construction of dom0 and domU guests from a 
hyperlaunch device tree configuration.
Xen nightly CI and release-gating test configurations would be extended 
to include hyperlaunch with guests on Arm.

== Work Item: Hyperlaunch of x86 PV and PVH guests (Resourcing TBD)

The goal of this effort is to enable Hyperlaunch of x86 PV and PVH guests.

This builds upon the initial x86 Hyperlaunch support to add domain 
construction of PV and PVH guests.
Xen nightly CI and release-gating test configurations would be extended 
to include hyperlaunch with guests on x86.

== Work Item: XSM-enforced FuSA (Resourcing TBD)

Hyperlaunch enables disaggregation, and XSM enables granular policy.

The goal of this work is to design and implement new modular, 
fine-grained policy to integrate into Xen for control for 
safety-critical VMs and isolation from non-safety-critical VMs.

== Work Item: Design for Hyperlaunch of x86 HVM guests (Resourcing TBD)

The goal of this effort is to research, prototype and produce design 
documentation for enabling hyperlaunch of x86 HVM guests, ie. with 
support for a device emulator.
This would build upon the x86 Hyperlaunch support for PVH guests.

Investigation to consider:
- launch alongside a classic dom0 domain
- launch on a static partitioned system without a dom0
- stub domains for device emulator isolation
- boot domain integration
- device assignment


-- Thanks for your consideration!