RE: [Xen-devel] Uses of &frame_table[xfn]

["Tian, Kevin" <kevin.tian@xxxxxxxxx>]

>From: Magenheimer, Dan
>(HP Labs Fort Collins)
>Sent: 2005年12月24日 4:34
>>
>> Very early on in boot we increase max_pfn beyond nr_pages and
>> allocate
>> a suitably large mem_map so that there are struct page's covering the
>> whole space.
>
>OK.  That is done on Xen/ia64 also but it is done in Xen
>rather than by changing Xenlinux.  Where is the code in
>Xenlinux/x86 that does this?  (It may be educational...)

linux-2.6-xen-sparse/arch/xen/i386/kernel/setup.c for find_max_pfn, which 
increases max_pfn to the point larger than machine frames allocated to domain 
and thus reserve an area to balance requests from backend driver.

>>
>> Yeah, they have struct page, but there is no memory mapped at that
>> kernel virtual address (the phys2mach entry points nowhere).
>
>Just to make sure I understand:  After domain0 starts and before
>any other domains start, the balloon list contains the dom0 kernel
>virtual address of all pages that _could_ be backed by RAM
>in the future, but aren't yet?  Then is the balloon list a
>dom0-land representation of _all_ the pages in Xen's domheap
>or just a subset?  (I'm guessing the latter... Xen uses the
>non-dom0 pages from the domheap to create new domains, but
>if the domheap eventually runs dry, Xen sucks pages away from
>dom0, correct?)

Not exactly. In the start, the balloon list only contains the page structs of 
frames between xen_start_info->nr_pages and max_pfn since they are nomap in 
phys2mach table in the early init phase. Later balloon list will expand/shrink 
based upon control panel request (if not enough free memory) and backend 
drivers (free some driver pages), etc. 

Further, it's not Xen to suck pages directly from dom0, instead it's customer 
needing more machine frames to check and do the request.

>>
>> Alloc'ing N pages would not guarantee to get exactly the
>> pages we want
>> (i.e., the ones with no RAM). The mem initialization as is does not
>> look particularly x86 specific. Is the check for PageReserved a
>> problem? If so, we can probably work around that -- it's only used to
>> skip over highmem pages in non-highmem i386 kernels.
>
>No, I think PageReserved is also used in Linux/ia64 to mark pages
>that are holes in the memmap, so that check is OK.  The problem
>is that on ia64 it is NOT the case that all dom0 pages below
>start_info->nr_pages "have RAM", e.g. some (perhaps most) of the
>pages below nr_pages are PageReserved.  But some of them
>may be PageReserved because they do not "have RAM" (meaning on
>ia64 that there is a hole there) and some may be PageReserved
>for other reasons (e.g. that's where kernel text lives) so I
>don't think any kind of linear-page-walk algorithm
>will work.

So one intuitive way is to borrow same phys2mach concept into ia64 xenlinux. 
Though more times required in the start than simply workaround this balloon 
specific problem, it can save much time to do more ugly hacks later when we 
want share more code among architectures.

Actually I can't come up easy way to do balloon for ia64 dom0, if physical 
still EQUALS to machine by current model.

You may allocate N free pages from dom0's free pool and append them to balloon 
list. OK, now dom0 won't use them anymore since they're allocated already. Next 
step immediately is to FREE them into domain heap so that Xen can allocate them 
for new domain creation. 

Then later all those N machine frames are allocated to new domain1, and thus 
their ownership changed. Now problem comes out. When dom0 is under pressure for 
memory, how can dom0 retrieves new machine frames back?

For current ia64 dom0, since it's physical address is equal (thus bound) to 
machine address, there's no way for ia64 dom0 to ask those ballooned frames 
back since they are still in use by domain1. Also it's difficult for ia64 dom0 
to get new frames since new machine frames means new physical frame which 
further means memory hotplug.

So we finally reach such a point, once balloon driver is on work upon current 
model, ia64 dom0 can only see its free pages fewer and fewer, but no way (or 
difficult) to get them back. Even when domain1 occupying those N machine frames 
is destroyed later, we still need to change memory hypercall interface since 
dom0 only wants exactly those N machine frames back instead of any other free 
frames.

All such troubles come from no phys2mach mapping table within ia64 dom0! So 
Dan, maybe it's time to revise current ia64 xenlinux design to make life 
easier. ;-)

>
>I expect that in the future you may not want to have this
>linear-page-walk code in a driver anyway as it may become
>subject to lots of CONFIG_WEIRD_MEMORY options on different
>arch's (maybe even x86-64?).
>
>Thanks,
>Dan
>

X86-64 should be mostly same as x86 since they all have contiguous/linear 
physical frames in xen world. Not sure whether larger memory will do any 
difference there.

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel