Re: [PATCH] xen/arm: Increase DOM0_FDT_EXTRA_SIZE to support max reserved memory banks

[Oleksandr Tyshchenko <Oleksandr_Tyshchenko@xxxxxxxx>]

On 3/27/26 09:30, Orzel, Michal wrote:

Hello Michal

> 
> 
> On 26/03/2026 20:03, Oleksandr Tyshchenko wrote:
>>
>>
>> On 3/26/26 18:50, Orzel, Michal wrote:
>>
>> Hello Michal
>>
>>>
>>>
>>> On 26/03/2026 14:15, Oleksandr Tyshchenko wrote:
>>>> Xen fails to construct the hardware domain's device tree with
>>>> FDT_ERR_NOSPACE (-3) when the host memory map is highly fragmented
>>>> (e.g., numerous reserved memory regions).
>>>>
>>>> This occurs because DOM0_FDT_EXTRA_SIZE underestimates the space
>>>> required for the generated extra /memory node. make_memory_node()
>>> Where does this extra /memory node come from? If this is for normal reserved
>>> memory regions, they should be present in the host dtb and therefore 
>>> accounted
>>> by fdt_totalsize (the host dtb should have reserved regions described in 
>>> /memory
>>> and /reserved-memory. Are you trying to account for static shm regions?
>>
>>
>> I might have misunderstood something, but here is my analysis:
>>
>> The extra /memory node is generated by Xen itself in handle_node() ->
>> make_memory_node() (please refer to the if ( reserved_mem->nr_banks > 0
>> ) check).
>>
>> Even though the normal reserved memory regions are present in the host
>> DTB (and thus accounted for in fdt_totalsize), Xen generates a new
>> /memory node specifically for the hardware domain to describe these
>> regions as reserved but present in the memory map. And since this node
>> is generated at runtime (it is not a direct copy from the host DTB),
>> its size must be covered by DOM0_FDT_EXTRA_SIZE.
> Yes, but the original DTB should also have these reserved regions described in
> /memory nodes, thus taking up some space that is already accounted in
> fdt_totalsize. Are you trying to say that in host DTB, these reserved ranges 
> fit
> nicely into e.g. a single /memory node range (i.e. a single reg pair covering
> most of the RAM)?

yes


  I can see that it might be possible but the commit msg needs
> to be clear about it. As of now, it reads as if the problem occured always 
> when
> there are multiple reserved memory regions. That's not true if a host DTB
> generates one /memory per one /reserved.

Yes, you are correct that the total size depends on how the host DTB is 
structured compared to how Xen regenerates it at runtime. So, the issue 
can arise if host DTB represents RAM using a single, large reg entry or 
a few entries.

***

I will update the commit message to clarify that, something like below:

Xen fails to construct the hardware domain's device tree with
FDT_ERR_NOSPACE (-3) when the host memory map is highly fragmented
(e.g., numerous reserved memory regions) and the host DTB represents
RAM compactly (e.g., a single reg pair or just a few).

This occurs because DOM0_FDT_EXTRA_SIZE underestimates the space
required for the generated extra /memory node. While the host DTB
might represent RAM compactly, make_memory_node() aggregates
all reserved regions into a single reg property.
With NR_MEM_BANKS (256) and 64-bit address/size cells, this property
can grow up to 4KB (256 * 16), easily exceeding the space originally
occupied by the host DTB's nodes plus the current padding, thereby
overflowing the allocated buffer.


> 
> Another issue is with the static shm nodes. User specifies the regions in the
> domain configuration and Xen creates *additional* nodes under /reserved and
> /memory that afaict we don't account for.

Yes, you are right.

Since these SHM sub-nodes and properties are generated purely from the 
Xen domain configuration and are not present in the host DTB, they have 
zero space allocated for them in fdt_totalsize.

So we need to redefine the macro. I propose the following formula that 
separates the range data (16 bytes per bank in /memory) from the node 
overhead (160 bytes per SHM region):

#define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry) + \
                             (NR_MEM_BANKS * 16) +                    \
                             (NR_SHMEM_BANKS * 160))

***
I will append the following text for the commit message:

Additionally, SHM regions require space for discrete sub-nodes under 
/reserved-memory. Each of the up to NR_SHMEM_BANKS (32) regions triggers 
the creation of a sub-node with properties (compatible, reg, xen,id, and 
xen,offset). These runtime-generated sub-nodes require approximately 160 
bytes each, further consuming the allocated buffer space.

Fix this by increasing DOM0_FDT_EXTRA_SIZE to account for both
fragmented reg properties (NR_MEM_BANKS * 16) under extra /memory
node and the injection of discrete SHM sub-nodes under /reserved-memory 
node (NR_SHMEM_BANKS * 160).

Please tell me, does it look ok?


> 
>>
>> For the instance, 10 reserved regions:
>>
>> (XEN) RAM: 0000000040000000 - 000000007fffffff
>> (XEN)
>> (XEN) MODULE[0]: 0000000043200000 - 000000004330afff Xen
>> (XEN) MODULE[1]: 0000000043400000 - 0000000043402fff Device Tree
>> (XEN) MODULE[2]: 0000000042e00000 - 000000004316907f Ramdisk
>> (XEN) MODULE[3]: 0000000040400000 - 0000000042d2ffff Kernel
>> (XEN)  RESVD[0]: 0000000040009000 - 0000000040009fff
>> (XEN)  RESVD[1]: 0000000040008000 - 0000000040008fff
>> (XEN)  RESVD[2]: 0000000040007000 - 0000000040007fff
>> (XEN)  RESVD[3]: 0000000040006000 - 0000000040006fff
>> (XEN)  RESVD[4]: 0000000040005000 - 0000000040005fff
>> (XEN)  RESVD[5]: 0000000040004000 - 0000000040004fff
>> (XEN)  RESVD[6]: 0000000040003000 - 0000000040003fff
>> (XEN)  RESVD[7]: 0000000040002000 - 0000000040002fff
>> (XEN)  RESVD[8]: 0000000040001000 - 0000000040001fff
>> (XEN)  RESVD[9]: 0000000040000000 - 0000000040000fff
>> ...
>>
>>   From make_memory_node():
>>
>> (XEN) Create memory node
>> (XEN)   Bank 0: 0x50000000->0x70000000
>> (XEN) (reg size 4, nr cells 4)
>>
>>
>>
>> (XEN) Create memory node
>> (XEN)   Bank 0: 0x40009000->0x4000a000
>> (XEN)   Bank 1: 0x40008000->0x40009000
>> (XEN)   Bank 2: 0x40007000->0x40008000
>> (XEN)   Bank 3: 0x40006000->0x40007000
>> (XEN)   Bank 4: 0x40005000->0x40006000
>> (XEN)   Bank 5: 0x40004000->0x40005000
>> (XEN)   Bank 6: 0x40003000->0x40004000
>> (XEN)   Bank 7: 0x40002000->0x40003000
>> (XEN)   Bank 8: 0x40001000->0x40002000
>> (XEN)   Bank 9: 0x40000000->0x40001000
>> (XEN) (reg size 4, nr cells 40)
>>
>>>
>>>> aggregates all reserved regions into a single reg property. With
>>>> NR_MEM_BANKS (256) and 64-bit address/size cells, this property
>>>> can grow up to 4KB (256 * 16), easily overflowing the allocated
>>>> buffer.
>>>>
>>>> Fix this by increasing DOM0_FDT_EXTRA_SIZE to account for
>>>> the worst-case size: NR_MEM_BANKS * 16 bytes.
>>>>
>>>> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@xxxxxxxx>
>>>> ---
>>>> Just to be clear, I have not seen a real-world issue with this.
>>>> The issue was observed during testing of limit conditions.
>>>> With this patch applied, Xen successfully boots the hardware domain,
>>>> exposing 256 reserved memory regions to it (using a synthetically
>>>> generated configuration).
>>>> ---
>>>> ---
>>>>    xen/arch/arm/domain_build.c | 6 ++++--
>>>>    1 file changed, 4 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>>> index e8795745dd..7f9f0f5510 100644
>>>> --- a/xen/arch/arm/domain_build.c
>>>> +++ b/xen/arch/arm/domain_build.c
>>>> @@ -100,9 +100,11 @@ int __init parse_arch_dom0_param(const char *s, const 
>>>> char *e)
>>>>    /*
>>>>     * Amount of extra space required to dom0's device tree.  No new nodes
>>> This comment would want to be updated because since its introduction things 
>>> have
>>> changed. Even the 128 came up as a result of adding /hypervisor node.
>>
>> You are right. I suggest the following wording:
>>
>> Amount of extra space required to dom0's device tree.
>> This covers nodes generated by Xen, which are not directly copied
>> from the host DTB. It is calculated as:
>>    - Space for /hypervisor node (128 bytes).
>>    - One terminating reserve map entry (16 bytes).
>>    - Space for a generated memory node covering all possible reserved
>>      memory regions (NR_MEM_BANKS * 16 bytes).
>>
>>
>>>
>>>>     * are added (yet) but one terminating reserve map entry (16 bytes) is
>>>> - * added.
>>>> + * added. Plus space for an extra memory node to cover all possible 
>>>> reserved
>>>> + * memory regions (2 addr cells + 2 size cells).
>>>>     */
>>>> -#define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry))
>>>> +#define DOM0_FDT_EXTRA_SIZE (128 + sizeof(struct fdt_reserve_entry) + \
>>>> +    (NR_MEM_BANKS * 16))
>>>>    
>>>>    unsigned int __init dom0_max_vcpus(void)
>>>>    {
>>>
>>> ~Michal
>>>
> 
> ~Michal
> 
>