RE: [Xen-devel] bring up Hypervisor on large (512GB) memory

["Tian, Kevin" <kevin.tian@xxxxxxxxx>]

>From: Mukesh Rathor
>Sent: Tuesday, February 10, 2009 11:09 AM
>
>Hi all,
>
>Trying to bring up the hypervisor on 512GB :
>
>1. Started with xen 3.1.4 (last oracle release), and 512GB, the system
>panic'd right away:
>    (XEN) Early fatal page fault at e008:ffff828c801ce0ad
>    (cr2=ffff8300cfc00000, ec=0002)
>
>I noticed an anomaly here in the RAM map:
>(XEN) Xen-e820 RAM map:
>(XEN)  0000000000000000 - 000000000009f400 (usable)
>(XEN)  000000000009f400 - 00000000000a0000 (reserved)
>(XEN)  00000000000f0000 - 0000000000100000 (reserved)
>(XEN)  0000000000100000 - 00000000cfd4c000 (usable)
>(XEN)  00000000cfd4c000 - 00000000cfd56000 (ACPI data)
>(XEN)  00000000cfd56000 - 00000000cfd57000 (usable)
>(XEN)  00000000cfd57000 - 00000000e0000000 (reserved)
>(XEN)  00000000fec00000 - 00000000fed00000 (reserved)
>(XEN)  00000000fee00000 - 00000000fee10000 (reserved)
>(XEN)  00000000ffc00000 - 0000000100000000 (reserved)
>(XEN)  0000000100000000 - 000000802ffff000 (usable)  <-------
>
>Seems that should be 0000008000000000 ???

802ffff000 seems more reasonable since there's a big hole reserved 
under 4G and thus max page in address space is shifted up to 
exceed 512GB size

>
>
>2. Reduce some RAM, and booting with 440GB, map makes more sense:
>
>(XEN) Xen-e820 RAM map:
>(XEN)  0000000000000000 - 000000000009f400 (usable)
>(XEN)  000000000009f400 - 00000000000a0000 (reserved)
>(XEN)  00000000000f0000 - 0000000000100000 (reserved)
>(XEN)  0000000000100000 - 00000000cfd4c000 (usable)
>(XEN)  00000000cfd4c000 - 00000000cfd56000 (ACPI data)
>(XEN)  00000000cfd56000 - 00000000cfd57000 (usable)
>(XEN)  00000000cfd57000 - 00000000e0000000 (reserved)
>(XEN)  00000000fec00000 - 00000000fed00000 (reserved)
>(XEN)  00000000fee00000 - 00000000fee10000 (reserved)
>(XEN)  00000000ffc00000 - 0000000100000000 (reserved)
>(XEN)  0000000100000000 - 0000006e00000000 (usable)
>

When you say "reducing some RAM", did you plug off DIMMs 
physically or just use "mem=" option to have Xen ignore pages
beyond that size? If the latter, then it makes sense to observe
6e00000000.

>Panic again:
>(XEN) Early fatal page fault at e008:ffff828c80210460
>(cr2=ffff8300cfc00000, ec=0002)
>
>
>The panic is trying to allocate bitmap in 
>init_boot_allocator(). The bitmap
>starts at cfac0000 and given the size dc1000, won't fit.
>
>
>3. Moving to unstable 19164, looks like things are even more 
>tighter!! I
>    couldn't even boot with 64GB. bitmap_start:cfac0000, 
>bitmap_size:201000,
>    alloc_bitmap:ffff8300cfac0000
>
>
>The only solution I can think of is moving the bitmap 
>elsewhere, above 4GB in
>this case:
>    figure the size of bitmap, DIRECT map space, allocate the map,
>    mark it reserved in the RAM map, and should work!
>
>    I'd have add a loop around init_boot_allocator() in __start_xen()
>    iterating thru the RAM map again, and finding space above 16M.
>
>
>Am I on the right track?
>

that bitmap follows xen image immediately, and there's a limitation
to only choose e820 entry (< BOOTSTRAP_DIRECTMAP_END) for
Xen relocation. I don't know the tricks behind that limitation. But if
you want to move bitmap higher, is it more generic to allow Xen image
itself relocated to >4G trunk which also solves your case here? Of
course I may lose some background here...

Or alternative is to have reloc_size covering bitmap size with smaller
change.

Thanks,
Kevin
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel