RE: [Xen-devel] RE:2Gb Memory lost in 2.6.32.36 - 50G memory, maxmem=50G, 2G used for pagetable, clamping (max:5G) is not helping.

[MaoXiaoyun <tinnycloud@xxxxxxxxxxx>]

Hi Konrad;
 
      I realized that in 32.36 kernel dom0 can balloon up to use whole memory address, in my case it is 48G. But it consumes 845M memory
for sparse memory to manage them, Well, my dom0 doesn't need so big flexable,  and I learnt that use "mem=" option in grub can limit
dom0 memory, I think it works for me now. 
 
Any comments?
 
title Red Hat Enterprise Linux Server (2.6.32.36xen)
        root (hd0,0)
        kernel /xen-4.0.1.gz  iommu=off x2apic=off console=com1,vga com1=115200,8n1 noreboot dom0_mem=6144M dom0_max_vcpus=4 dom0_vcpus_pin cpuidle=0 cpufreq=none 
        module /vmlinuz-2.6.32.36xen ro root=LABEL=/ hda=noprobe console=hvc0 mem=8192M
        module /initrd-2.6.32.36xen.img
 
After dom0 start,
root@xmao# free -m
             total       used       free     shared    buffers     cached
Mem:          4135        508       3626          0         21        166
-/+ buffers/cache:        320       3814
Swap:         1027          0       1027
 
=======================
xen_release_chunk: looking at area pfn bf7e0-bf7ec: 12 pages freed
xen_release_chunk: looking at area pfn c0000-e0000: 131072 pages freed
xen_release_chunk: looking at area pfn f0000-fec00: 60416 pages freed
xen_release_chunk: looking at area pfn fec01-fec8a: 137 pages freed
xen_release_chunk: looking at area pfn fec8b-fee00: 373 pages freed
xen_release_chunk: looking at area pfn fee01-fff00: 4351 pages freed
released 196361 pages of unused memory
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 000000000009a000 (usable)
 Xen: 000000000009a400 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 00000000bf790000 (usable)
 Xen: 00000000bf790000 - 00000000bf79e000 (ACPI data)
 Xen: 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS)
 Xen: 00000000bf7d0000 - 00000000bf7e0000 (reserved)
 Xen: 00000000bf7ec000 - 00000000c0000000 (reserved)
 Xen: 00000000e0000000 - 00000000f0000000 (reserved)
 Xen: 00000000fec00000 - 00000000fec01000 (reserved)
 Xen: 00000000fec8a000 - 00000000fec8b000 (reserved)
 Xen: 00000000fee00000 - 00000000fee01000 (reserved)
 Xen: 00000000fff00000 - 0000000100000000 (reserved)
 Xen: 0000000100000000 - 0000000c6ff09000 (usable)
user-defined physical RAM map:
 user: 0000000000000000 - 000000000009a000 (usable)
 user: 000000000009a400 - 0000000000100000 (reserved)
 user: 0000000000100000 - 00000000bf790000 (usable)
 user: 00000000bf790000 - 00000000bf79e000 (ACPI data)
 user: 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS)
 user: 00000000bf7d0000 - 00000000bf7e0000 (reserved)
 user: 00000000bf7ec000 - 00000000c0000000 (reserved)
 user: 00000000e0000000 - 00000000f0000000 (reserved)
 user: 00000000fec00000 - 00000000fec01000 (reserved)
 user: 00000000fec8a000 - 00000000fec8b000 (reserved)
 user: 00000000fee00000 - 00000000fee01000 (reserved)
 user: 00000000fff00000 - 0000000100000000 (reserved)
 user: 0000000100000000 - 0000000200000000 (usable)

 

---

From: tinnycloud@xxxxxxxxxxx
To: konrad.wilk@xxxxxxxxxx
CC: xen-devel@xxxxxxxxxxxxxxxxxxx; jeremy@xxxxxxxx
Subject: RE: [Xen-devel] RE:2Gb Memory lost in 2.6.32.36 - 50G memory, maxmem=50G, 2G used for pagetable, clamping (max:5G) is not helping.
Date: Wed, 25 May 2011 17:08:49 +0800

Hi Konrad:
 
     I think I've located where the lost memory going. It is in bootmem when called sparse_init();
 
    I enabled bootmem_debug=1 in kernel start args, and log every __reserve and __free memory with 'count' taged.
 
    Like ___free & __reserve code(end of mail) :
 
    Attached is the output log.
   
1) We give dom0 5630M, that is 5630 * 1024 /4 = 1441280 pages
 
2) Line 204-237 is the stage of early reservations. 
Add up memory in *early reservations* except adderss beyond 15ffe0(line 228) :
     6329(line 225) + 23 + 1508 + 23470 = 31330 pages ( 123M)
 
3) I will caculated the mem reserved after line 237.
 
In line 237 , tot_res_pages is 11631467 and line 1162, its values is  11881636, so 11881636 - 11631467 = 250169 pages are reserved.
In line 203, tot_free_pages is 12776995 and line 1158, its values is 12810787, so 12810787 - 12776995  = 33792 pages are freed
So 250169  - 33792 = 216377 pages (216377 *4 / 1024 =  845M) are still in bootmem, I've been located it is in sparse_init.
 
4) Line 162, 196361 ( 768M )  pages of unused memory are been relesed to Xen.
 
5) Line 1169 ,  Memory is 3717504k =  3630M ,  3630 + 768 + 845 + 123 = 5366.        5630 - 5366 = 264 M (Did I miscaulated? )
  
Questions
 
1) Normally, sparse_init will only reserver about 173M in previous kernel, and in .36 is it meant to reserve so much?
2) where is 264M memory I last caculated, did I make mistake?
 
Many thanks.
 
 
 
 
 
 
 
 
============================code to log========mm/bootmem.c====
 
253 static void __init __free(bootmem_data_t *bdata,
254 <+++<+++<+++unsigned long sidx, unsigned long eidx)
255 {
256 <+++unsigned long idx;
257     static int count;
258     static int tot_free_pages;
259
260     tot_free_pages += (eidx - sidx);
261
262 <+++bdebug("count %d nid=%td start=%lx end=%lx pages %d tot_free_pages %d\n", ++count, bdata - bootmem_node_data,
263 <+++<+++sidx + bdata->node_min_pfn,
264 <+++<+++eidx + bdata->node_min_pfn,
265         eidx - sidx,
266         tot_free_pages
267         );
268
 
277 static int __init __reserve(bootmem_data_t *bdata, unsigned long sidx,
278  unsigned long eidx, int flags)
279 {
280     unsigned long idx;
281     int exclusive = flags & BOOTMEM_EXCLUSIVE;
282     static int count;
283     static int tot_res_pages;
284
285     tot_res_pages += (eidx - sidx);
286 <+++bdebug("count %d nid=%td start=%lx end=%lx flags=%x pages=%d, tot_res_pages %d\n",                                                                  
287         ++count,
288 <+++<+++bdata - bootmem_node_data,
289 <+++<+++sidx + bdata->node_min_pfn,
290 <+++<+++eidx + bdata->node_min_pfn,
291 <+++<+++flags,
292         eidx - sidx,
293         tot_res_pages
294         );
    
 

 
> Date: Thu, 19 May 2011 12:16:56 -0400
> From: konrad.wilk@xxxxxxxxxx
> To: tinnycloud@xxxxxxxxxxx
> CC: xen-devel@xxxxxxxxxxxxxxxxxxx; jeremy@xxxxxxxx
> Subject: Re: [Xen-devel] RE:2Gb Memory lost in 2.6.32.36 - 50G memory, maxmem=50G, 2G used for pagetable, clamping (max:5G) is not helping.
>
> > > What happens if you provide dom0_mem=max:5G to clamp on the size?
> > >
> >
> > Seems nothing different(what is clamp means? ^_^ )
>
> http://en.wikipedia.org/wiki/Clamp_(tool)
>
> .. snip..
> > Xen: 0000000100000000 - 0000000c6ff09000 (usable)
> > DMI present.
> > AMI BIOS detected: BIOS may corrupt low RAM, working around it.
> > last_pfn = 0xc6ff09 max_arch_pfn = 0x400000000
> .. snip..
> > xen_commandline : msi=1 iommu=off x2apic=off console=com1,vga com1=115200,8n1 noreboot dom0_mem=max:5G dom0_max_vcpus=4 dom0_vcpus_pin cpuidle=0 cpufreq=none xsave=0
>
> OK, I would have expected your last entry in the E820 to be truncated
> to 15fe00000, but it is not with the max:5G parameter. Looks like
> you found a bug - but I am not sure if it is the Xen hypervisor or
> the E820 parsing code in Linux. Fortunatly for you, I just got my hands on 32G
> machine so I can work on this. Can you remind me in a week if I you do
> not see any patches from me?

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel