[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] Running out of Xen heap space with large memory

  • To: Xen-devel@xxxxxxxxxxxxxxxxxxx
  • From: beth kon <eak@xxxxxxxxxx>
  • Date: Tue, 20 Nov 2007 14:00:57 -0500
  • Delivery-date: Tue, 20 Nov 2007 11:02:22 -0800
  • List-id: Xen developer discussion <xen-devel.lists.xensource.com>

Hi. I have been debugging a hang when booting Xen with 512G memory. I am testing on an x3950, 8 nodes, 128 way, with xen 3.0.3. In order to get past the 166G limit, IETH added the no-pv-compat flag to the xen boot line.

The boots have been hanging when memory is greater than about 432G. I found that, as memory size increases, the size of the xen heap decreases, because additional memory is being used for the memory housekeeping (seen in init_boot_allocator). Eventually (around 432G) the xen heap is reported as 0MB:

(XEN) Command line: /xen.gz-2.6.18-53.el5 numa=on dom0_mem=512m com2=19200,8n1
console=com2 no-pv-compat
(XEN)  0000000000000000 - 0000000000098000 (usable)
(XEN)  0000000000098c00 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000007fea6000 (usable)
(XEN)  000000007fea64c0 - 000000007fef6380 (ACPI data)
(XEN)  000000007fef6380 - 0000000090000000 (reserved)
(XEN)  00000000fec00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000007e00000000 (usable)
(XEN) System RAM: 514046MB (526383352kB)
(XEN) ACPI: [SRAT:0x00] ignored 32 entries of 64 found
(XEN) BETH xenheap_phys_start = 18677760, xenheap_phys_end=16777216
(XEN) BETH s = 4294967296, e=16777216
(XEN) Xen heap: 0MB (0kB)
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Unknown interrupt

The boot hangs at this point (no surprise). Note that the "BETH" debug statements show xenheap_phys_start>xenheap_phys_end.

I realize that interest in this may be low since the no-pv-compat flag has been removed in xen-unstable. The resulting 166G limit was discussed here:

If pages being passed among domains is the issue, wouldn't the no-pv-compat flag address that by not allowing any 32 bit guests on the machine? I assume page stealing is restricted to domains running on the same hypervisor, right? Why was the no-pv-compat flag removed?

And as Raj discussed:
Even if Raj provides these changes, I believe the above issue with running out of heap space would still exist. I see that init_boot_allocator is unchanged between 3.0.3 and unstable. Any suggestions on how this issue could be corrected?

I will continue to look at this code, but any suggestions from people who are more familiar with it would be greatly appreciated.

Elizabeth Kon (Beth)
IBM Linux Technology Center
Open Hypervisor Team
email: eak@xxxxxxxxxx

Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.