Re: [Xen-users] FW: Dom0 bootup fail on ARM : Unable to allocate first memory bank

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

On Fri, 2015-09-04 at 02:21 +0000, Tom Ting[äéå] wrote:
> Please refer to the bottom
> 
> > -----Original Message-----
> > From: Ian Campbell [mailto:ian.campbell@xxxxxxxxxx] 
> > Sent: Friday, September 04, 2015 12:41 AM
> > To: Tom Ting[äéå]; xen-users@xxxxxxxxxxxxx
> > Subject: Re: [Xen-users] Dom0 bootup fail on ARM : Unable to allocate 
> > first memory bank
> > 
> > On Thu, 2015-09-03 at 11:29 +0000, Tom Ting[äéå] wrote:
> > > Hi All
> > >  
> > > I am currently working on bringing Xen(latest mainline 4.6 RC) alive 
> > > on an ARM Cortex-A53 platform with 512 MB RAM.
> > > Xen is complaining about it couldn't allocate 128 MB from DOM0 and 
> > > stopped.
> > > I tried to use pagealloc_info() to dump memory status, finding 
> > > there's 
> > > only small amount of memory in HEAP.
> > > Does any one know what is going on or where to check for this 
> > > problem??
> > > Really appreciate for your help.
> > 
> > With 4.6 RC you should have ended up with 32MB of Xenheap and the rest 
> > of free memory in domheap, which ought to be plenty for a 128M dom0 
> > really.
> > 
> > > (XEN) Checking for initrd in /chosen
> > > (XEN) RAM: 0000000000000000 - 000000001ffe0fff
> > > (XEN)
> > > (XEN) MODULE[0]: 0000000001ff2000 - 0000000001ff3000 Device Tree
> > > (XEN) MODULE[1]: 0000000003000000 - 0000000003600000 Kernel 
> > > earlyprintk console=ttyS0,115200 cma=16m@64m
> > > (XEN) RESVD[0]: 0000000001ff2000 - 0000000001ff3000
> > > (XEN)
> > > (XEN) Command line: console=ttyS0,115200 earlyprintk loglevel=4
> > 
> > Aside: Those look like Linux kernel options, not Xen hypervisor 
> > options.
> > 
> > > (XEN) Placing Xen at 0x000000001fc00000-0x000000001fe00000
> > > (XEN) Update BOOTMOD_XEN from 0000000000200000-0000000000302d81 =>
> > > 000000001fc00000-000000001fd02d81
> > > (XEN) Domain heap initialised
> > > (XEN) Physical memory information:
> > > (XEN) Xen heap: 0kB free
> > > (XEN) heap[01]: 8kB free
> > > (XEN) heap[02]: 8kB free
> > > (XEN) heap[03]: 16kB free
> > > (XEN) heap[04]: 32kB free
> > > (XEN) heap[05]: 64kB free
> > > (XEN) heap[06]: 128kB free
> > > (XEN) heap[07]: 256kB free
> > > (XEN) heap[08]: 512kB free
> > > (XEN) heap[09]: 1024kB free
> > > (XEN) heap[10]: 2048kB free
> > > (XEN) heap[11]: 4096kB free
> > > (XEN) heap[12]: 8192kB free
> > > (XEN) heap[13]: 16328kB free
> > > (XEN) Dom heap: 32712kB free
> > 
> > This is not as much RAM as expected I think, it only sums to ~64M. I 
> > wonder where the rest has gone!
> > 
> > Inferring from the logs your memory layout seems to be:
> > 
> > RAM:        0000000000000000 - 000000001ffe0fff 511M
> > - Free 0000000000000000 - 0000000000200000 2M
> > - Xen  0000000000200000 - 0000000000302d81 1.01M    (initial
> > placement)
> > - DTB       0000000001ff2000 - 0000000001ff3000 4K
> > - Free 0000000001ff3000 - 000000001fc00000 476M
> > - Xen  000000001fc00000 - 000000001fd02d81 1.01M    (relocated)
> > 
> > So there is a pretty big 476M free area there. Why is it not in the 
> > heaps then I wonder!
> > 
> > Probably the next step would be to instrument 
> > xen/arch/arm/setup.c:setup_mm (the 64 bit one, there's an #ifdef), to 
> > log which regions it is adding as heap and why it is rejecting those 
> > which it isn't adding...
> > 
> > I think you are using u-boot? Please could you "fdt chosen ; fdt print 
> > /chosen" right before you do the boot. It might also be useful to see 
> > your fdt
> > 
> > > (XEN)
> > > (XEN) Bad console= option 'ttyS0'
> > > (XEN) Bad console= option '115200'
> > > Xen 4.6.0-rc
> > > (XEN) Xen version 4.6.0-rc (tom_ting@(none)) (aarch64-linux-xgcc 
> > > (Realtek
> > > ASDK64-4.9.3 Build 2180) 4.9.3 20150413 (prerelease)) debug=y Wed Sep 
> > > 
> > > 2
> > > 20:16:44 CST 2015
> > > (XEN) Latest ChangeSet: Wed Sep 2 17:15:27 2015 +0800 
> > > git:3cad003-dirty
> > 
> > What is commit 3cad003, doesn't appear in my tree? and what are the 
> > changes which added the -dirty? IOW what is your exact code?
> > 
> > Ian.
> > 
> (remove the reset of log)
> 
> Hi Ian
> 
> My base commit on mainline is 801ab48e5556cb54f67e3cb57f077f47e8663ced
> 
> Here's my settings:
> - commit 3cad003
>  -> this is my local commit which add in new 
> xen/arch/arm/platforms/<platform.c>, but there's actually nothing inside 
> but only device-tree compatible info.

If this is only compat strings then you don't actually need it, the generic
fallbacks will work fine.

>     The other thing in this commit is early-printk-config in 
> arch/arm/Rules.mk

OK.

>  and arch/arm/arm64/head.S which defines load-address/size at the start 
> of the image(sync kernel format).

I'm not sure what this entails exactly, so please post these changes

>     For dirty part, it's the debug code i'm putting on.
>     
> - FDT config, I haven't correct the bootargs for DOM0 yet, since my 
> platform/kernel is still developing. Currently I just put a dummy at 
> 0x03000000 but not expecting it will boot up.
> chosen {
>         #address-cells = <0x00000001>;
>         #size-cells = <0x00000001>;
>         bootargs = "console=dtuart dtuart=serial0";
>         module@0 {
>                 compatible = "xen,linux-zimage", "xen,multiboot-module";
>                 reg = <0x03000000 0x00600000>;
>                 bootargs = "earlyprintk console=ttyS0,115200 cma=16m@64m"
> ;  <== This bootargs will be correct accordingly
>         };
> };

Is this a fixed version based on my feedback regarding hypervisor vs kernel
args?

> I put extra debug message at setup_mm->end_boot_allocator to check 
> memory_regions/total_avail_pages and have following info :
> 
> It seems that the bootmem_region_list seems to be corrupt after calling 
> init_heap_pages(bootmem_region_list[0])??

This could be read as "free(bootmem_region_list[0]). IOW using
bootmem_region_list[0] after this point is a use after free bug.

Ian.


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users