Re: Problem loading linux 5.19 as PV dom0

[Juergen Gross <jgross@xxxxxxxx>]

On 23.06.22 11:08, Jan Beulich wrote:
> On 23.06.2022 11:01, Juergen Gross wrote:
> > On 23.06.22 10:47, Jan Beulich wrote:
> > > On 23.06.2022 10:06, Juergen Gross wrote:
> > > > On 23.06.22 09:55, Jan Beulich wrote:
> > > > > On 22.06.2022 18:06, Juergen Gross wrote:
> > > > > > A Linux kernel 5.19 can only be loaded as dom0, if it has been
> > > > > > built with CONFIG_AMD_MEM_ENCRYPT enabled. This is due to the
> > > > > > fact that otherwise the (relevant) last section of the built
> > > > > > kernel has the NOLOAD flag set (it is still marked with
> > > > > > SHF_ALLOC).
> > > > > >
> > > > > > I think at least the hypervisor needs to be changed to support
> > > > > > this layout. Otherwise it will put the initial page tables for
> > > > > > dom0 at the same position as this last section, leading to
> > > > > > early crashes.
> > > > > Isn't Xen using the bzImage header there, rather than any ELF
> > > > > one? In which case it would matter how the NOLOAD section is
> > > > For a PV kernel? No, I don't think so.
> > > Actually it's a mix (and the same for PV and PVH) - the bzImage
> > > header is parsed to get at the embedded ELF header. XenoLinux was
> > > what would/could be loaded as plain ELF.
> > >
> > > > > actually represented in that header. Can you provide a dump (or
> > > > > binary representation) of both headers?
> > > > Program Header:
> > > >        LOAD off    0x0000000000200000 vaddr 0xffffffff81000000 paddr
> > > > 0x0000000001000000 align 2**21
> > > >             filesz 0x000000000145e114 memsz 0x000000000145e114 flags r-x
> > > >        LOAD off    0x0000000001800000 vaddr 0xffffffff82600000 paddr
> > > > 0x0000000002600000 align 2**21
> > > >             filesz 0x00000000006b7000 memsz 0x00000000006b7000 flags rw-
> > > >        LOAD off    0x0000000002000000 vaddr 0x0000000000000000 paddr
> > > > 0x0000000002cb7000 align 2**21
> > > >             filesz 0x00000000000312a8 memsz 0x00000000000312a8 flags rw-
> > > >        LOAD off    0x00000000020e9000 vaddr 0xffffffff82ce9000 paddr
> > > > 0x0000000002ce9000 align 2**21
> > > >             filesz 0x00000000001fd000 memsz 0x0000000000317000 flags rwx
> > > 20e9000 + 317000 = 240000
> > >
> > > >        NOTE off    0x000000000165df10 vaddr 0xffffffff8245df10 paddr
> > > > 0x000000000245df10 align 2**2
> > > >             filesz 0x0000000000000204 memsz 0x0000000000000204 flags ---
> > > >
> > > >
> > > > Sections:
> > > > Idx Name          Size      VMA               LMA               File off  Algn
> > > > ...
> > > >     30 .smp_locks    00009000  ffffffff82edc000  0000000002edc000  022dc000  
> > > > 2**2
> > > >                      CONTENTS, ALLOC, LOAD, READONLY, DATA
> > > >     31 .data_nosave  00001000  ffffffff82ee5000  0000000002ee5000  022e5000  
> > > > 2**2
> > > >                      CONTENTS, ALLOC, LOAD, DATA
> > > >     32 .bss          0011a000  ffffffff82ee6000  0000000002ee6000  022e6000  
> > > > 2**12
> > > >                      ALLOC
> > > 2ee6000 + 11a000 = 240000
> > >
> > > >     33 .brk          00026000  ffffffff83000000  ffffffff83000000  00000000  
> > > > 2**0
> > > >                      ALLOC
> > > This space isn't covered by any program header. Which in turn may be a
> > > result of its LMA matching its VMA, unlike for all other sections.
> > > Looks like a linker script or linker issue to me: While ...
> > >
> > > > And the related linker script part:
> > > >
> > > >            __end_of_kernel_reserve = .;
> > > >
> > > >            . = ALIGN(PAGE_SIZE);
> > > >            .brk (NOLOAD) : AT(ADDR(.brk) - LOAD_OFFSET) {
> > > ... this AT() looks correct to me, I'm uncertain of the use of NOLOAD.
> > > Note that .bss doesn't have NOLOAD, matching the vast majority of the
> > > linker scripts ld itself has.
> > Yeah, but the filesz and memsz values of the .bss related program header
> > differ a lot (basically by the .bss size plus some alignment),
> That's the very nature of .bss - no data to be loaded from the file.
>
> > and the
> > .bss section flags clearly say that its attributes match those of .brk.
> >
> > I'm not sure why the linker wouldn't add .brk to the same pgrogram
> > header entry as .bss, but maybe that is some .bss special handling.
> I don't know either, but I suspect this to be an effect of using NOLOAD
> (without meaning to decide yet whether it's a wrong use of the
> attribute or bad handling of it in ld).
I just did a test: dropping the "(NOLOAD)" for .brk in the linker script
will result in the .brk section to be included in the same program header
as the .bss section.

I'll start an upstream discussion to drop it (I could imagine problems
e.g. when using grub, as grub might place the initrd overlapping the .brk
section when not using a compressed kernel).


Juergen

Attachment: OpenPGP_0xB0DE9DD628BF132F.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature
Description: OpenPGP digital signature