Re: [PATCH] xen/device-tree: Allow exact match for overlapping regions

[Grygorii Strashko <grygorii_strashko@xxxxxxxx>]

On 14.11.24 15:09, Julien Grall wrote:
> On 14/11/2024 12:22, Michal Orzel wrote:
> > On 14/11/2024 13:04, Julien Grall wrote:
> > > Hi Michal,
> > >
> > > On 14/11/2024 11:48, Michal Orzel wrote:
> > > > On 14/11/2024 11:31, Julien Grall wrote:
> > > > > Looking at the code, I think /memreserve/ and /reserved-memory are not
> > > > > mapped in Xen and everything in /reserved-memory is mapped to dom0.
> > > > Why do we forward /reserved-memory to dom0 fdt but /memreserve/ not?
> > > I was wondering the same. The main issue I can think of with
> > > /memreserve/ is some of the regions will likely be for Xen own usage. So
> > Can you give example of regions defined as reserved for Xen usage (other than 
> > static-mem)?
> The spin table to bring-up CPUs.
>
> > > we would need to have a way to exclude them from dom0.
> > >
> > >   >  From the discussion> we're having it seems like we should treat them
> > > equally. Also, looking at Luca patch,
> > > > we seem to special case /memreserve/ and only allow for overlap /memresrve/ 
> > > > with boot modules
> > > > and not /reserved-memory/ with boot modules. If we are going to claim that all 
> > > > the boot modules
> > > > can be marked as reserved by the bootloader, then I think we should treat them 
> > > > equally providing
> > > > the same experience to dom0.
> > > In my mind, /memreserved/ and /reserved-memory/ are different. The
> > > former doesn't say what the region is for, but the latter will indicate it.
> > In the context of this patch, I don't agree. We're discussing overlap, and if a 
> > region A
> > from /memreserve/ overlaps fully with a module A, we know what is the purpose 
> > of it.
>  > Today it's initrd, but as you say we cannot rule out other modules as well.
>
> To give a concrete example, the /reserved-region/ can be used to reserve space 
> for the VGA buffer. It would be odd that someone would put the boot module in 
> the middle of the VGA buffer... If Xen ends up to use the VGA buffer (not the 
> case today), then it would be a problem. Xen would need to be reworked to move 
> all boot modules outside of the memory because it can access the VGA (or any 
> other reserved regions).
>
> The problem is slightly different for /memreserve/ because we don't exactly 
> know what the regions are for until we parse the rest of the DT. Yes 
> technically, a user could put the initrd in the wrong place. So the problem is 
> the same. But this is a relexation I am more willing to accept over 
> /reserved-region/ right now.
>
> > > So I am not 100% sure how the bootmodule could end up in
> > > /reserved-memory/ because they are described as part of the multiboot
> > > modules. Do you have a scenario?
> > I don't same as I don't have scenario for /memreserve/ overlapping with sth 
> > else than initrd.
> > All of these comes from my validation of u-boot, grub, barebox code. I have a 
> > feeling that due to
> > U-Boot trick that is not present in any other *known* bootloader, we are trying 
> > to over-engineer the problem :)
> > But as Stefano and you wrote, we should follow the spec and for me we should 
> > therefore treat them equally.
> See above why I don't think we should treat them equally today. We might be 
> able in the future if we can categorize all the regions in /memreserve/.
>
> [...]
>
> > > > Last thing I wanted to ask (for my clarity) is that if bootloader loads initrd 
> > > > at region A and marks
> > > > it as reserved (either in /memreserve/ or /reserved-memory/), and later on Xen 
> > > > copies initrd from region
> > > > A to B, shouldn't the reserved memory region be updated to cover new region for 
> > > > initrd?
> > > If we mark the region has a reserved, then we are telling the OS it
> > > can't use the region. But I am not sure why it would be needed as Xen
> > Well, in the context of initrd, kernel uses it even though it is reserved. This 
> > is because
> > of the second part of the spec where other bindings come into play.
> >
> > > doesn't care how the regions is going to be used by the domain. From a
> > > domain side, do you see any reason why we would want to mark again the
> > > region as reserved?
> > TBH I don't same as I still don't know why U-Boot does that trick. It comes 
> > from a very
> > old code and my initial understanding is that it is done purely for U-boot 
> > bookkeeping.
> /memreserve/ (and now) /reserved-regions/ are an easy way to find the regions 
> that should be excluded from the RAM. Without that, you will need to have 
> special case (such as for initrd, and the various xen boot moudles). I suspect 
> that Linux have a special case and hence why it hasn't been a problem that Xen 
> doesn't reserve the region.
My be it will help in this discussion - some investigation results.

At boot time (only ARM64, but other arches looks similar):

- Determines if initrd present from DT : 
early_init_dt_scan()->setup_machine_fdt()
or by checking bootargs "initrd/initrdmem=".

- [1] In arm64_memblock_init() it adds initrd in reserved memory
https://github.com/torvalds/linux/blob/master/arch/arm64/mm/init.c#L274
therefore Linux doesn't depends on any kind of "DT reserved memory" ranges for 
initrd.

- The Linux processes "reserved-memory" nodes first.
https://github.com/torvalds/linux/blob/master/drivers/of/fdt.c#L504

- After this FDT memory reserve map is processed:
https://github.com/torvalds/linux/blob/master/drivers/of/fdt.c#L496

The Linux doesn't perform any special processing of FDT memory reserve map -
it just adds it to the reserved memory list (splice allowed), which is 
reasonable, as
Linux doesn't know what is the purpose of each reserved memory block in FDT 
memory reserve map.

Later based on DT (or other meaning) some reserved memory can be handled in a 
special way.

After early_init_fdt_scan_reserved_mem() finishes - the result is the list of 
reserved memory
ranges which are excluded from generic memory management process.


In case of Xen, initrd is not intended to be used by Xen directly, but should 
be passed to Dom0.
My current, understanding - it will be copied, so from the Xen point of view 
the memory occupied by Initrd is
not permanently reserved and can be even freed once Initrd successfully passed 
in Dom0.
Also, due to the point [1] above no need to pass initrd memory as reserved 
region to Dom0 Linux based domain.

Honestly, it looks like nothing from FDT memory reserve map make sense to 
blindly pass to Dom0/hwdom
- initrd (described above)
- FDT - it will be generated
NOTE. From u-boot fdt_shrink_to_minimum() code it can be seen that FDT region 
also could be in /memreserve/
https://github.com/u-boot/u-boot/blob/master/boot/fdt_support.c#L766
- spin-table : Xen owns it, I assume.
- any kind of secure/firmware memory - domains do not care unless specified explicitly in 
"reserved-memory" (or by other means).
  Xen will guarantee that such ranges are not touched/allocated, i assume.

So in my opinion.
- native Linux (or other OS) need to know about all reserved memory to avoid 
clash with generic memory management routines.
- virtual Linux (OS) does not need to know - as Xen should exclude memory, 
defined as reserved in System device tree,
  when allocating memory for domain.
  if domain need to know about some reserved memory - it is expected be defined 
in DT explicitly (or by other means).

The Xen performs the primary memory management role in virtualized environment,
so it is the main user of any kind of information about reserved memory.

Do not shot me please I've tried to be helpful...and issue fixed.
BR,
-grygorii