Re: E820 memory allocation issue on Threadripper platforms

[Juergen Gross <jgross@xxxxxxxx>]

On 11.01.24 09:37, Jan Beulich wrote:
> On 11.01.2024 03:29, Patrick Plenefisch wrote:
> > Hi,
> >
> > I ran into a memory allocation issue, I think. It is the same as
> > https://github.com/QubesOS/qubes-issues/issues/8791 and I saw at the end it
> > was recommended (by marmarek) that the issue reporter forward the issue to
> > this list. I searched the list, but as I didn't see it in the list already,
> > I'm doing that now.
> >
> > Hardware:
> > I have an AMD Threadripper 7960X on a ASRock TRX50 WS motherboard. The
> > Qubes reporter had a Threadripper 3970X on an ASUS Prime TRX40-Pro
> > Motherboard. I saw a 3rd issue report of a similar issue on another
> > Threadripper, so I think this may be Threadripper-specific.
> >
> > Setup:
> > The QuebesOS reporter was using Qubes Installer.
> > My install was that I had a fresh install of Debian 12 (no gui), and then
> > did `apt install xen-system-amd64` and rebooted.
> >
> > The issue:
> > Any boot of Xen on the hardware results in a halted machine. When
> > monitoring the logs with `vga=,keep`, we get:
> >
> > (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
> > (XEN) Freed 644kB init memory
> > mapping kernel into physical memory
> > about to get started…
> > xen hypervisor allocated kernel memory conflicts with E820
> So first of all (the title doesn't say it) this is a Linux Dom0 issue.
> Whether or not needing addressing in Xen is unknown at this point.
>
> > (XEN) Hardware Dom0 halted: halting machine
> >
> > None of the settings I or the Qubes reporter have tried have been able to
> > get past this failure.
> >
> > I am happy to provide debugging support.
> Well, the crucial piece of data initially is going to be: What's the
> E820 map Xen gets to see, what's the E820 map Dom0 gets to see, and
> what address range is the conflict detected for? The first question
> is possible to answer by supplying a serial log. The second question
> likely means adding some debugging code to either Xen or Linux. The
> answer to third question may be possible to infer from the other
> data, but would likely be better to obtain explicitly by adjusting /
> amending the message Linux emits.
The needed information should all be in the hypervisor messages.

The hypervisor is initially presenting a memory map to dom0 which is not the
same as the native memory map. Dom0 tries to rearrange its memory layout to
be compatible with the native memory map.

The seen message ("xen hypervisor allocated kernel memory conflicts with E820")
tells us that the kernel position is conflicting with the native memory map
(at least one guest pfn occupied by the kernel would be at a non-RAM populated
location after rearrangement of memory).

In theory it would be possible to cover this case, too, but it would be quite
cumbersome. Right now only the initrd is allowed to conflict with the memory map
(it will be moved in this case), kernel and initial page table conflicts are not
handled.

When I added the conflict handling nearly 10 years ago, there was no hardware
known to have memory holes at addresses which would conflict with Xen's initial
idea of dom0 memory layout.

I can look into this later, but right now I'm just about to go offline probably
until end of January.


Juergen

Attachment: OpenPGP_0xB0DE9DD628BF132F.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature