Re: E820 memory allocation issue on Threadripper platforms

[Jan Beulich <jbeulich@xxxxxxxx>]

On 16.01.2024 01:22, Patrick Plenefisch wrote:
> I managed to set up serial access and saved the output with the requested
> flags as the attached logs

Thanks. While you didn't ...

> On Thu, Jan 11, 2024 at 5:13 AM Juergen Gross <jgross@xxxxxxxx> wrote:
>> On 11.01.24 09:37, Jan Beulich wrote:
>>> On 11.01.2024 03:29, Patrick Plenefisch wrote:
>>>> I ran into a memory allocation issue, I think. It is the same as
>>>> https://github.com/QubesOS/qubes-issues/issues/8791 and I saw at the
>> end it
>>>> was recommended (by marmarek) that the issue reporter forward the issue
>> to
>>>> this list. I searched the list, but as I didn't see it in the list
>> already,
>>>> I'm doing that now.
>>>>
>>>> Hardware:
>>>> I have an AMD Threadripper 7960X on a ASRock TRX50 WS motherboard. The
>>>> Qubes reporter had a Threadripper 3970X on an ASUS Prime TRX40-Pro
>>>> Motherboard. I saw a 3rd issue report of a similar issue on another
>>>> Threadripper, so I think this may be Threadripper-specific.
>>>>
>>>> Setup:
>>>> The QuebesOS reporter was using Qubes Installer.
>>>> My install was that I had a fresh install of Debian 12 (no gui), and
>> then
>>>> did `apt install xen-system-amd64` and rebooted.
>>>>
>>>> The issue:
>>>> Any boot of Xen on the hardware results in a halted machine. When
>>>> monitoring the logs with `vga=,keep`, we get:
>>>>
>>>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch
>> input)
>>>> (XEN) Freed 644kB init memory
>>>> mapping kernel into physical memory
>>>> about to get started…
>>>> xen hypervisor allocated kernel memory conflicts with E820
>>>
>>> So first of all (the title doesn't say it) this is a Linux Dom0 issue.
>>> Whether or not needing addressing in Xen is unknown at this point.
>>>
>>>> (XEN) Hardware Dom0 halted: halting machine
>>>>
>>>> None of the settings I or the Qubes reporter have tried have been able
>> to
>>>> get past this failure.
>>>>
>>>> I am happy to provide debugging support.
>>>
>>> Well, the crucial piece of data initially is going to be: What's the
>>> E820 map Xen gets to see, what's the E820 map Dom0 gets to see, and
>>> what address range is the conflict detected for? The first question
>>> is possible to answer by supplying a serial log. The second question
>>> likely means adding some debugging code to either Xen or Linux. The
>>> answer to third question may be possible to infer from the other
>>> data, but would likely be better to obtain explicitly by adjusting /
>>> amending the message Linux emits.

... fiddle with the Linux message,  ...

>> The needed information should all be in the hypervisor messages.
>> 
>> The hypervisor is initially presenting a memory map to dom0 which is not
>> the
>> same as the native memory map. Dom0 tries to rearrange its memory layout to
>> be compatible with the native memory map.
>>
>> The seen message ("xen hypervisor allocated kernel memory conflicts with
>> E820")
>> tells us that the kernel position is conflicting with the native memory map
>> (at least one guest pfn occupied by the kernel would be at a non-RAM
>> populated
>> location after rearrangement of memory).
>>
>> In theory it would be possible to cover this case, too, but it would be
>> quite
>> cumbersome. Right now only the initrd is allowed to conflict with the
>> memory map
>> (it will be moved in this case), kernel and initial page table conflicts
>> are not
>> handled.
>>
>> When I added the conflict handling nearly 10 years ago, there was no
>> hardware
>> known to have memory holes at addresses which would conflict with Xen's
>> initial
>> idea of dom0 memory layout.

... as per

(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x4a00000

there's an overlap with not exactly a hole, but with an
EfiACPIMemoryNVS region:

(XEN)  0000000100000-0000003159fff type=2 attr=000000000000000f
(XEN)  000000315a000-0000003ffffff type=7 attr=000000000000000f
(XEN)  0000004000000-0000004045fff type=10 attr=000000000000000f
(XEN)  0000004046000-0000009afefff type=7 attr=000000000000000f

(the 3rd of the 4 lines). Considering there's another region higher
up:

(XEN)  00000a747f000-00000a947efff type=10 attr=000000000000000f

I'm inclined to say it is poor firmware (or, far less likely, boot
loader) behavior to clobber a rather low and entirely arbitrary RAM
range, rather than consolidating all such regions near the top of
RAM below 4Gb. There are further such odd regions, btw:

(XEN)  0000009aff000-0000009ffffff type=0 attr=000000000000000f
...
(XEN)  000000b000000-000000b020fff type=0 attr=000000000000000f

If the kernel image was sufficiently much larger, these could become
a problem as well. Otoh if the kernel wasn't built with
CONFIG_PHYSICAL_START=0x1000000, i.e. to start at 16Mb, but at, say,
2Mb, things should apparently work even with this unusual memory
layout (until the kernel would grow enough to again run into that
very region).

It remains to be seen in how far it is reasonably possible to work
around this in the kernel. While (sadly) still unsupported, in the
meantime you may want to consider running Dom0 in PVH mode.

Jan