Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM

[Julien Grall <julien@xxxxxxx>]

Hi,

On 18/12/2019 17:09, Roman Shaposhnik wrote:
> Hi,
>
> On Wed, Dec 18, 2019 at 4:56 AM Julien Grall <julien@xxxxxxx> wrote:
> > > So that is, in fact, my first question -- why is Xen not showing
> > > available memory in xl info?
> > I am not entirely sure what exact information you want.
> >
> > The output you dumped above contain the available memory for the memory
> > (see "free_memory").
> >
> > Are you looking from something different?
> Just to be clear: I was giving 2G via devicetrees (the same device
> trees that would
> make Linux detect 2G of RAM) hence I was expecting xl info to show that. Instead
> I only got 1120M shown by xl info.
>
> > On 18/12/2019 00:04, Roman Shaposhnik wrote:
> > >           memory {
> > >                   device_type = "memory";
> > >                   reg = <0x0 0x0 0x0 0x5e00000 0x0 0x5f00000 0x0 0x1000
> > > 0x0 0x5f02000 0x0 0xefd000 0x0 0x6e00000 0x0 0x60f000 0x0 0x7410000
> > > 0x0 0x1aaf0000 0x0 0x21f00000 0x0 0x100000 0x0 0x22000000 0x0
> > > 0x1c000000>;
> > >           };
> > >
> > >           reserved-memory {
> > >                   ranges;
> > >                   #size-cells = <0x2>;
> > >                   #address-cells = <0x2>;
> > >
> > >                   ramoops@21f00000 {
> > >                           ftrace-size = <0x20000>;
> > >                           console-size = <0x20000>;
> > >                           reg = <0x0 0x21f00000 0x0 0x100000>;
> > >                           record-size = <0x20000>;
> > >                           compatible = "ramoops";
> > >                   };
> > >
> > >                   linux,cma {
> > >                           linux,cma-default;
> > >                           reusable;
> > >                           size = <0x0 0x8000000>;
> > >                           compatible = "shared-dma-pool";
> > >                   };
> > >           };
> > >
> > > If you look at the REG -- it does now add up to 2Gb, but booting Xen
> > > with it has exactly the
> > > same effect as booting it with: reg = <0x0 0x0 0x0 0x80000000>;\
> > If you boot Xen using EFI, the memory information wil come from EFI and
> > the DT node will be ignored. So unless UEFI is able to pick up the
> > modification of the DT memory node, modifying the DT is not going to
> > affect anything.
> That's a good point, but given that I always go through GRUB, I was
> expecting devicetree command to completely overshadow whatever
> information UEFI may have. Am I wrong?
GRUB will load Xen/Linux as an EFI application. Both of them will ignore 
the memory nodes when booting using EFI. For more details, see the 
answer I wrote separately.

> > > I am attaching a full log, and I see the following in the logs:
> > >
> > > (XEN) Allocating 1:1 mappings totalling 720MB for dom0:
> > > (XEN) BANK[0] 0x00000008000000-0x0000001c000000 (320MB)
> > > (XEN) BANK[1] 0x00000040000000-0x00000058000000 (384MB)
> > > (XEN) BANK[2] 0x0000007b000000-0x0000007c000000 (16MB)
> > >
> > > Which sort of makes sense, I guess -- but I still don't understand
> > > where all these ranges
> > > are coming from and how come Xen doesn't see the full 2Gb even with various
> > > devicetrees I tried.
> > The range aboves describe the memory range given to Dom0. For all the
> > memory given to Xen,m you want to look at the top of your log:
> >
> > (XEN) Checking for initrd in /chosen
> > (XEN) RAM: 0000000000000000 - 0000000005dfffff
> > (XEN) RAM: 0000000005f00000 - 0000000006dfefff
> > (XEN) RAM: 0000000006e00000 - 000000000740efff
> > (XEN) RAM: 0000000007410000 - 000000001db8dfff
> > (XEN) RAM: 00000000350f0000 - 000000003dbd2fff
> > (XEN) RAM: 000000003dbd3000 - 000000003dffffff
> > (XEN) RAM: 0000000040000000 - 000000005a653fff
> > (XEN) RAM: 000000007ada0000 - 000000007ada3fff
> > (XEN) RAM: 000000007aea8000 - 000000007afa9fff
> > (XEN) RAM: 000000007afaa000 - 000000007ec73fff
> > (XEN) RAM: 000000007ec74000 - 000000007fdddfff
> > (XEN) RAM: 000000007fdde000 - 000000007fea5fff
> > (XEN) RAM: 000000007fea6000 - 000000007ff6dfff
> > (XEN) RAM: 000000007ffff000 - 000000007fffffff
> >
> > Looking at the differences with the Linux logs, there is indeed some
> > memory not detected by Xen.
> >
> > On Xen, we only consider usuable memory any EFI description with
> > EfiConventionalMemory, EfiBootServicesCode and EfiBootServicesData.
> >
> > Linux include more type here, so this may explain why we see a difference.
> >
> > While Looking at it, I have also noticed that we don't seem to care
> > about the memory attribute. I suspect this could be another latent issue
> > in Xen if the attribute does not match.
> Anything I can do to help debug this? I can run any kind of debug builds, etc.
> if needed.
Thank you for the offer, I think I have a good understanding of the 
problem now. So debug should not be necessary.
However, I would appreciate if anyone could help to write a patch for it.

> I mean -- at this point it would be really great to get HiKey back to the status
> of Xen-on-ARM developer board.
>
> > > Any ideas here would be greatly apprecaited!
> > >
> > > Thanks,
> > > Roman.
> > >
> > > P.S. Any guess at what these mean?
> > >
> > > (XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x00ffff87355558
> > > gva=0xffff872f2000 gpa=0x000000000f0000
> > > (XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x00ffffb734e558
> > > gva=0xffffb72eb000 gpa=0x000000000f0000
> > > (XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x00ffff8f9d2558
> > > gva=0xffff8f96f000 gpa=0x000000000f0000
> > It means that Linux has tried to access something that has not been
> > mapped in stage-2. As Dom0 is mapped 1:1, the GPA also give you the host
> > physical address. In this case, it is trying to access 0xf0000.
> >
> > This seems to belong to the RAM, but this part has not been allocated to
> > Dom0.
> Got it! Thank you! Am I correct in guessing that this can only come from
> a driver of some sorts trying to tickle the hardware? IOW, I should be
> looking for some abnormalities in my linux kernel messages to try and
> see what this could be.
Xen will inject a data abort to the guest. So you should get a stack 
trace from Linux.
Cheers,

--
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel