[Xen-users] FW: Dom0 bootup fail on ARM : Unable to allocate first memory bank

[Tom Ting[äéå] <tom_ting@xxxxxxxxxxx>]

Please refer to the bottom

>-----Original Message-----
>From: Ian Campbell [mailto:ian.campbell@xxxxxxxxxx] 
>Sent: Friday, September 04, 2015 12:41 AM
>To: Tom Ting[äéå]; xen-users@xxxxxxxxxxxxx
>Subject: Re: [Xen-users] Dom0 bootup fail on ARM : Unable to allocate first 
>memory bank
>
>On Thu, 2015-09-03 at 11:29 +0000, Tom Ting[äéå] wrote:
>> Hi All
>>  
>> I am currently working on bringing Xen(latest mainline 4.6 RC) alive 
>> on an ARM Cortex-A53 platform with 512 MB RAM.
>> Xen is complaining about it couldn't allocate 128 MB from DOM0 and 
>> stopped.
>> I tried to use pagealloc_info() to dump memory status, finding there's 
>> only small amount of memory in HEAP.
>> Does any one know what is going on or where to check for this problem??
>> Really appreciate for your help.
>
>With 4.6 RC you should have ended up with 32MB of Xenheap and the rest of free 
>memory in domheap, which ought to be plenty for a 128M dom0 really.
>
>> (XEN) Checking for initrd in /chosen
>> (XEN) RAM: 0000000000000000 - 000000001ffe0fff
>> (XEN)
>> (XEN) MODULE[0]: 0000000001ff2000 - 0000000001ff3000 Device Tree
>> (XEN) MODULE[1]: 0000000003000000 - 0000000003600000 Kernel 
>> earlyprintk console=ttyS0,115200 cma=16m@64m
>> (XEN) RESVD[0]: 0000000001ff2000 - 0000000001ff3000
>> (XEN)
>> (XEN) Command line: console=ttyS0,115200 earlyprintk loglevel=4
>
>Aside: Those look like Linux kernel options, not Xen hypervisor options.
>
>> (XEN) Placing Xen at 0x000000001fc00000-0x000000001fe00000
>> (XEN) Update BOOTMOD_XEN from 0000000000200000-0000000000302d81 =>
>> 000000001fc00000-000000001fd02d81
>> (XEN) Domain heap initialised
>> (XEN) Physical memory information:
>> (XEN) Xen heap: 0kB free
>> (XEN) heap[01]: 8kB free
>> (XEN) heap[02]: 8kB free
>> (XEN) heap[03]: 16kB free
>> (XEN) heap[04]: 32kB free
>> (XEN) heap[05]: 64kB free
>> (XEN) heap[06]: 128kB free
>> (XEN) heap[07]: 256kB free
>> (XEN) heap[08]: 512kB free
>> (XEN) heap[09]: 1024kB free
>> (XEN) heap[10]: 2048kB free
>> (XEN) heap[11]: 4096kB free
>> (XEN) heap[12]: 8192kB free
>> (XEN) heap[13]: 16328kB free
>> (XEN) Dom heap: 32712kB free
>
>This is not as much RAM as expected I think, it only sums to ~64M. I wonder 
>where the rest has gone!
>
>Inferring from the logs your memory layout seems to be:
>
>RAM:   0000000000000000 - 000000001ffe0fff 511M
> - Free 0000000000000000 - 0000000000200000 2M
> - Xen  0000000000200000 - 0000000000302d81 1.01M      (initial
>placement)
> - DTB 0000000001ff2000 - 0000000001ff3000 4K
> - Free 0000000001ff3000 - 000000001fc00000 476M
> - Xen  000000001fc00000 - 000000001fd02d81 1.01M      (relocated)
>
>So there is a pretty big 476M free area there. Why is it not in the heaps then 
>I wonder!
>
>Probably the next step would be to instrument xen/arch/arm/setup.c:setup_mm 
>(the 64 bit one, there's an #ifdef), to log which regions it is adding as heap 
>and why it is rejecting those which it isn't adding...
>
>I think you are using u-boot? Please could you "fdt chosen ; fdt print 
>/chosen" right before you do the boot. It might also be useful to see your fdt
>
>> (XEN)
>> (XEN) Bad console= option 'ttyS0'
>> (XEN) Bad console= option '115200'
>> Xen 4.6.0-rc
>> (XEN) Xen version 4.6.0-rc (tom_ting@(none)) (aarch64-linux-xgcc 
>> (Realtek
>> ASDK64-4.9.3 Build 2180) 4.9.3 20150413 (prerelease)) debug=y Wed Sep 
>> 2
>> 20:16:44 CST 2015
>> (XEN) Latest ChangeSet: Wed Sep 2 17:15:27 2015 +0800 
>> git:3cad003-dirty
>
>What is commit 3cad003, doesn't appear in my tree? and what are the changes 
>which added the -dirty? IOW what is your exact code?
>
>Ian.
>
(remove the reset of log)

Hi Ian

My base commit on mainline is 801ab48e5556cb54f67e3cb57f077f47e8663ced

Here's my settings:
- commit 3cad003
 -> this is my local commit which add in new 
xen/arch/arm/platforms/<platform.c>, but there's actually nothing inside but 
only device-tree compatible info.
    The other thing in this commit is early-printk-config in arch/arm/Rules.mk 
and arch/arm/arm64/head.S which defines load-address/size at the start of the 
image(sync kernel format).
    For dirty part, it's the debug code i'm putting on.
    
- FDT config, I haven't correct the bootargs for DOM0 yet, since my 
platform/kernel is still developing. Currently I just put a dummy at 0x03000000 
but not expecting it will boot up.
chosen {
        #address-cells = <0x00000001>;
        #size-cells = <0x00000001>;
        bootargs = "console=dtuart dtuart=serial0";
        module@0 {
                compatible = "xen,linux-zimage", "xen,multiboot-module";
                reg = <0x03000000 0x00600000>;
                bootargs = "earlyprintk console=ttyS0,115200 cma=16m@64m";  <== 
This bootargs will be correct accordingly
        };
};

I put extra debug message at setup_mm->end_boot_allocator to check 
memory_regions/total_avail_pages and have following info :

It seems that the bootmem_region_list seems to be corrupt after calling 
init_heap_pages(bootmem_region_list[0])??

Log :
== Dump bootmem_region_list at the start of end_boot_allocator() ==
(XEN) DEBUG bootmem_region_list[0], s:00001000, e:01ff2000
(XEN) DEBUG bootmem_region_list[1], s:01ff3000, e:03000000
(XEN) DEBUG bootmem_region_list[2], s:03600000, e:1c000000
(XEN) DEBUG bootmem_region_list[3], s:1e000000, e:1fc00000
(XEN) DEBUG bootmem_region_list[4], s:1fd03000, e:1ffd0000
== Start to call init_heap_pages() from first region ==
(XEN) --- s:00000001, e:00001ff2, 0, 0
(XEN) DEBUG :  s:00001000, e:01ff2000, total_avail_pages:0  
(XEN) DEBUG total_avail_pages after 1st region 8177
== End of 1st region, 8k pages reclaimed ==
== But it seems the reset of boot_mem_region is corrupt?? ==
(XEN) --- s:ffffffffffffffff, e:ffffffffffffffff, 0, 0
(XEN) --- s:ffffffffffffffff, e:ffffffffffffffff, 0, 0
(XEN) --- s:ffffffffffffffff, e:ffffffffffffffff, 0, 0
(XEN) --- s:ffffffffffffffff, e:ffffffffffffffff, 0, 0
(XEN) --- s:ffffffffffffffff, e:ffffffffffffffff, 0, 0
(XEN) DEBUG : total_avail_pages:8178
(XEN) Domain heap initialised

Patch :
 void __init end_boot_allocator(void)
 {
     unsigned int i;

+       for ( i = 0; i < nr_bootmem_regions; i++ )
+               printk("DEBUG bootmem_region_list[%d], s:%08lx, e:%08lx\n",
+                       i, pfn_to_paddr(bootmem_region_list[i].s), 
pfn_to_paddr(bootmem_region_list[i].e));
+
     /* Pages that are free now go to the domain sub-allocator. */
     for ( i = 0; i < nr_bootmem_regions; i++ )
     {
         struct bootmem_region *r = &bootmem_region_list[i];
+
+       printk("--- s:%08lx, e:%08lx, %d, %d\n", r->s, r->e, 
phys_to_nid(pfn_to_paddr(r->s)), cpu_to_node(0));
         if ( (r->s < r->e) &&
              (phys_to_nid(pfn_to_paddr(r->s)) == cpu_to_node(0)) )
         {
+           printk("DEBUG :  s:%08lx, e:%08lx, total_avail_pages:%lu\n",
+                       pfn_to_paddr(bootmem_region_list[i].s), 
pfn_to_paddr(bootmem_region_list[i].e), total_avail_pages);
             init_heap_pages(mfn_to_page(r->s), r->e - r->s);
             r->e = r->s;
             break;
         }
     }
+
+    printk("DEBUG total_avail_pages after 1st region %lu\n", 
total_avail_pages);
+
     for ( i = nr_bootmem_regions; i-- > 0; )
     {
         struct bootmem_region *r = &bootmem_region_list[i];
-        if ( r->s < r->e )
+       printk("--- s:%08lx, e:%08lx, %d, %d\n", r->s, r->e, 
phys_to_nid(pfn_to_paddr(r->s)), cpu_to_node(0));
+        if ( r->s < r->e ) {
+           printk("DEBUG :  s:%08lx, e:%08lx, total_avail_pages:%lu\n",
+                       pfn_to_paddr(bootmem_region_list[i].s), 
pfn_to_paddr(bootmem_region_list[i].e), total_avail_pages);
             init_heap_pages(mfn_to_page(r->s), r->e - r->s);
+       }
     }



















 



















































_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users