Re: [PATCH 2/2] arm/efi: Use dom0less configuration when using EFI boot

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Fri, 17 Sep 2021, Luca Fancellu wrote:
> > On 16 Sep 2021, at 21:16, Stefano Stabellini <sstabellini@xxxxxxxxxx> wrote:
> > 
> > On Thu, 16 Sep 2021, Jan Beulich wrote:
> >> On 16.09.2021 17:07, Luca Fancellu wrote:
> >>> I explain here my understanding on dom0less, this feature is used to 
> >>> start domUs at
> >>> Xen boot in parallel, the name is misleading but it doesn’t require dom0 
> >>> to be absent.
> >>> 
> >>> So if you have a dom0 kernel embed in the image, it's completely fine to 
> >>> start it and it 
> >>> doesn’t have to be skipped.
> >>> 
> >>> Here the possible user cases:
> >>> 1) start only dom0 [dom0 modules on xen.cfg or embedded in Xen image]
> >>> 2) start only domUs, true dom0less [no dom0 modules on xen.cfg or 
> >>> embedded in Xen image, domUs on DT]
> >>> 3) start dom0 and domUs [(dom0 modules on xen.cfg or embedded in Xen 
> >>> image) and domUs on DT]
> >> 
> >> If that's the intention - fine. Stefano?
> > 
> 
> Hi Stefano,
> 
> > What do you mean by dom0 modules embedded in the Xen image? I am not
> > familiar with it, but I imagine it doesn't involve any multiboot,module
> > nodes in device tree, right?
> > 
> > Putting aside "dom0 modules embedded in Xen image" that I didn't fully
> > understand, there are three ways to load Dom0:
> > 
> > - dom0 kernel (and ramdisk, optional) on xen.cfg
> > - dom0 kernel (and ramdisk, optional) on device tree using the "reg" 
> > property
> > - dom0 kernel (and ramdisk, optional) on device tree using the 
> > "uefi,binary" property
> 
> True for the #1 and #2, the last one is not implemented. The uefi,binary 
> property
> for now is only used to load domU modules.

Yeah, it is no problem that is not currently implemented, but from a
device tree binding / efi interface perspective it should be possible.

 
> > Then, the other use cases are:
> > 
> > - true dom0less, domUs on device tree using the "reg" property
> > - true dom0less, domUs on device tree using the "uefi,binary" property
> > 
> > And of course all the possible combinations between Dom0 and DomU
> > loading.
> > 
> > 
> > Currently, patch #1 checks for the presence of a Dom0 kernel node and, if
> > present, it decides not to proceed with xen.cfg. So if the Dom0 kernel
> > node is *not* present, efi_arch_use_config_file returns true.
> > 
> > However, this could be a true dom0less configuration without any Dom0
> > kernel. If so, you might not want to load xen.cfg at all because it is
> > not needed. In a true dom0less configuration, we probably want
> > efi_arch_use_config_file to return false.
> 
> In a true dom0less configuration we might need to read xen.cfg to retrieve the
> Xen command line, 

The Xen command line could also be on device tree
(/chosen/xen,xen-bootargs).


> but following the actual implementation of the common code
> there is more. I’m going to explain.
> 
> What efi_arch_use_config_file really does is not only choosing to read xen.cfg
> or not. Following the common code (xen/common/efi/boot.c) and what its result 
> activate
> along the path, it basically decides if the UEFI stub is used as a loader 
> from filesystem
> or not. We need the UEFI stub as a loader to be 100% UEFI and load our 
> modules.
>
> The original check basically says “if there are multiboot,module in the DT, 
> then some
> bootloader has loaded in memory the required modules so I’m not gonna load 
> anything
> from the filesystem because I assume it puts everything in place for me to 
> boot.”

OK, I am following. It looks like this is the source of the issue.

 
> >From misc/efi.txt:
> When booted as an EFI application, Xen requires a configuration file as 
> described below unless a bootloader,
> such as GRUB, has loaded the modules and describes them in the device tree 
> provided to Xen. If a bootloader
> provides a device tree containing modules then any configuration files are 
> ignored, and the bootloader is
> responsible for populating all relevant device tree nodes.
> 
> What I’m doing in patch #1 is restricting that check to just the 
> multiboot,module that are
> Dom0 module, why? Because with the introduction of dom0less we need to specify
> multiboot,modules for domUs, but the presence or not of dom0 modules is the 
> only
> Information we need to understand if the user decided to start Xen with 
> everything
> in places (modules in memory, xen command line, dtb) or if the job is 
> demanded to the
> UEFI stub and its configuration file.

I don't think so. Imagine a case where the user has everything in device
tree, doesn't need xen.cfg, but dom0 and domUs are specified as
uefi,binary.

We don't want xen.cfg but we do want to be able to load files from the
filesystem. This might not be currently implemented but from an bindings
perspective it should be possible.


> By the configuration file you can also load in memory the Xen dtb, so Xen can
> be started as an EFI application without the DTB and then load it using the 
> EFI stub.

This can be very useful but it would follow the !fdt check and return
true from efi_arch_use_config_file. So it doesn't really conflict with
anything we would around multiboot,module and xen,cfg-loading.


> I’m not against this new property “xen,cfg-loading”, I just think it is not 
> needed because
> we have all the information we need without it and in any case we need to 
> read the
> configuration file because otherwise we won’t have access to the Xen command 
> line.

We don't necessarely need to read the Xen command line from xen.cfg :-)


> Now I’m going to show you examples of all use cases that are valid with the 
> introduction
> of this serie:
> 
> 1) Start Xen as EFI application and load only Dom0
> 
>     Xen.cfg:
>     [global]
>     default=xen_dom0
> 
>     [xen_dom0]
>     options=<Xen command line>
>     kernel=vmlinuz-3.0.31-0.4-xen [domain 0 command line options]
>     ramdisk=initrd-3.0.31-0.4-xen
>     dtb=<xen DTB>
> 
>     DT:
>     {no modification}
> 
> 2) Start Xen as EFI application to load a true dom0less setup
> 
>     Xen.cfg:
>     [global]
>     default=xen_true_dom0less
> 
>     [xen_true_dom0less]
>     options=<Xen command line>
>     dtb=<xen DTB>
> 
>     DT:
>     chosen {
>         #size-cells = <0x1>;
>       #address-cells = <0x1>;
> 
>       domU1 {
>             #size-cells = <0x1>; 
>             #address-cells = <0x1>;
>             compatible = "xen,domain”;
>             cpus = <1>;
>             memory = <0 0xC0000>;
>            
>             module@1 {
>                 compatible = "multiboot,kernel", "multiboot,module”;
>                 bootargs = "console=ttyAMA0 root=/dev/ram0 rw”;
>                 uefi,binary = “domU_kernel1.bin"
>             };
> 
>             module@2 {
>                 compatible = “multiboot,ramdisk", "multiboot,module”;
>                 uefi,binary = “domU_ramdisk1.bin"
>             };
> 
>             module@3 {
>                 compatible = "multiboot,device-tree", "multiboot,module”;
>                 uefi,binary = “domU_passthrough1.dtb"
>             }; 
>         };
>         
>         domU2 { <as above> };
>     }
> 
> 3) Start Xen as EFI application to load Dom0 and DomUs
> 
>     Xen.cfg:
>     [global]
>     default=xen_dom0_domUs
> 
>     [xen_dom0_domUs]
>     options=<Xen command line>
>     kernel=vmlinuz-3.0.31-0.4-xen [domain 0 command line options]
>     ramdisk=initrd-3.0.31-0.4-xen
>     dtb=<xen DTB>
> 
>     DT:
>     chosen {
>         #size-cells = <0x1>;
>       #address-cells = <0x1>;
> 
>       domU1 {
>             #size-cells = <0x1>; 
>             #address-cells = <0x1>;
>             compatible = "xen,domain”;
>             cpus = <1>;
>             memory = <0 0xC0000>;
>            
>             module@1 {
>                 compatible = "multiboot,kernel", "multiboot,module”;
>                 bootargs = "console=ttyAMA0 root=/dev/ram0 rw”;
>                 uefi,binary = “domU_kernel1.bin"
>             };
> 
>             module@2 {
>                 compatible = “multiboot,ramdisk", "multiboot,module”;
>                 uefi,binary = “domU_ramdisk1.bin"
>             };
> 
>             module@3 {
>                 compatible = "multiboot,device-tree", "multiboot,module”;
>                 uefi,binary = “domU_passthrough1.dtb"
>             }; 
>         };
>         
>         domU2 { <as above> };
>     }
> 
> So as you see every case is covered without the introduction of the
> property.
> 
> Please let me know what do you think.

I think that from an interface perspective (not a code perspective) we
need to be able to account for cases like:

4) Start Xen as EFI application and load only Dom0
no Xen.cfg
DT:
  xen,xen-bootargs
  dom0/uefi,binary
  domUs/uefi,binary


But in any case, even disregarding this case, past experience has taught
me that it is always better to have an explicit flag to trigger a new
behavior, rather than relying on "guesswork". If we introduce
"xen,cfg-loading", we are going to be a lot more future-proof that if we
don't introduce it in terms of backward and forward compatibility in
case we need to change anything.