Re: E820 memory allocation issue on Threadripper platforms

[Xenia Ragiadakou <xenia.ragiadakou@xxxxxxx>]

On 11/1/24 10:37, Jan Beulich wrote:
> On 11.01.2024 03:29, Patrick Plenefisch wrote:
> > Hi,
> >
> > I ran into a memory allocation issue, I think. It is the same as
> > https://github.com/QubesOS/qubes-issues/issues/8791 and I saw at the end it
> > was recommended (by marmarek) that the issue reporter forward the issue to
> > this list. I searched the list, but as I didn't see it in the list already,
> > I'm doing that now.
> >
> > Hardware:
> > I have an AMD Threadripper 7960X on a ASRock TRX50 WS motherboard. The
> > Qubes reporter had a Threadripper 3970X on an ASUS Prime TRX40-Pro
> > Motherboard. I saw a 3rd issue report of a similar issue on another
> > Threadripper, so I think this may be Threadripper-specific.
> >
> > Setup:
> > The QuebesOS reporter was using Qubes Installer.
> > My install was that I had a fresh install of Debian 12 (no gui), and then
> > did `apt install xen-system-amd64` and rebooted.
> >
> > The issue:
> > Any boot of Xen on the hardware results in a halted machine. When
> > monitoring the logs with `vga=,keep`, we get:
> >
> > (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
> > (XEN) Freed 644kB init memory
> > mapping kernel into physical memory
> > about to get started…
> > xen hypervisor allocated kernel memory conflicts with E820
> So first of all (the title doesn't say it) this is a Linux Dom0 issue.
> Whether or not needing addressing in Xen is unknown at this point.
>
> > (XEN) Hardware Dom0 halted: halting machine
> >
> > None of the settings I or the Qubes reporter have tried have been able to
> > get past this failure.
> >
> > I am happy to provide debugging support.
> Well, the crucial piece of data initially is going to be: What's the
> E820 map Xen gets to see, what's the E820 map Dom0 gets to see, and
> what address range is the conflict detected for? The first question
> is possible to answer by supplying a serial log. The second question
> likely means adding some debugging code to either Xen or Linux. The
> answer to third question may be possible to infer from the other
> data, but would likely be better to obtain explicitly by adjusting /
> amending the message Linux emits.
We 've already hit similar issue because xen doesn't take into account 
the reserved memory regions when loading the dom0 kernel (even if it is 
relocatable). It can be worked around by changing accordingly 
CONFIG_PHYSICAL_START in kernel config.
Let me provide more details on how to get the info Jan requested:

1) in the xen cmdline add: e820-verbose=true console_to_ring

2) in the dom0 kernel cmdline add: earlyprintk=xen

3) change the xen log message emitted by the linux kernel to print the 
conflicting address, like below
diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index cfa99e8f054b..ad88b700d58e 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -717,7 +717,7 @@ static void __init xen_reserve_xen_mfnlist(void)
        xen_relocate_p2m();
        memblock_phys_free(start, size);
}
-
+void xen_raw_printk(const char *fmt, ...);
/**
  * xen_memory_setup - Hook for machine specific memory setup.
  **/
@@ -853,7 +853,8 @@ char * __init xen_memory_setup(void)
         */
        if (xen_is_e820_reserved(__pa_symbol(_text),
                        __pa_symbol(__bss_stop) - __pa_symbol(_text))) {
-               xen_raw_console_write("Xen hypervisor allocated kernel 
memory conflicts with E820 map\n");
+               xen_raw_printk("Xen hypervisor allocated kernel memory 
conflicts with E820 map: %#lx - %#lx\n",
+                              __pa_symbol(_text), 
__pa_symbol(__bss_stop));
                BUG();
        }

> Jan