Re: [Xen-devel] Converting heap page_infos to contiguous virtual

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 14/07/16 11:25, George Dunlap wrote:
> On 13/07/16 21:57, Boris Ostrovsky wrote:
>> On 07/13/2016 04:34 PM, Andrew Cooper wrote:
>>> On 13/07/2016 21:17, Boris Ostrovsky wrote:
>>>> On 07/13/2016 04:02 PM, Andrew Cooper wrote:
>>>>> On 13/07/16 20:44, Boris Ostrovsky wrote:
>>>>>> I would like to clear a bunch of Xen heap pages at once (i.e. not
>>>>>> page-by-page).
>>>>>>
>>>>>> Greatly simplifying things, let's say I grab (in common/page_alloc.c)
>>>>>>     pg = page_list_remove_head(&heap(node, zone, order)
>>>>>>
>>>>>> and then
>>>>>>
>>>>>>     mfn_t mfn =
>>>>>> _mfn(page_to_mfn(pg));                                        
>>>>>>     char *va = mfn_to_virt(mfn_x(mfn));
>>>>>>     memset(va, 0, 4096 * (1 << order));
>>>>>>
>>>>>>
>>>>>> Would it be valid to this?
>>>>> In principle, yes.  The frame_table is in order.
>>>>>
>>>>> However, mfn_to_virt() will blow up for RAM above the 5TB boundary.  You
>>>>> need to map_domain_page() to get a mapping.
>>>> Right, but that would mean going page-by-page, which I want to avoid.
>>>>
>>>> Now, DIRECTMAP_SIZE is ~128TB (if my math is correct) --- doesn't it
>>>> imply that it maps this big a range contiguously (modulo PDX hole)?
>>> Your maths is correct, and yet you will end up with problems if you
>>> trust it.
>>>
>>> That is the magic mode for the idle and monitor pagetables.  In the
>>> context of a 64bit PV guest, the cutoff is at 5TB, at which point you
>>> venture into the virtual address space reserved for guest kernel use. 
>>> (It is rather depressing that the 64bit PV guest ABI is the factor
>>> limiting Xen's maximum RAM usage.)
>> I don't know whether it would make any difference but the pages that I am
>> talking about are not in use by any guest, they are free. (This question
>> is for scrubbing rewrite that I am working on. Which apparently you
>> figured out judged by what you are saying below)
> Is this start-of-day scrubbing (when there are no guests), or scrubbing
> on guest destruction?
>
> If the former, it seems like it might not be too difficult to arrange
> that we're in a context that has all the RAM mapped.

This will be runtime scrubbing of pages.  This topic has come up at
several hackathons.

Currently, domain destroy on a 1TB VM takes ~10 mins of synchronously
scrubbing RAM in continuations of the domain_kill() hypercall (and those
databases VMs really like their RAM).

ISTR the plan was to have a page_info "dirty" flag and a dirty page list
which is scrubbed while idle (or per-node, more likely). 
alloc_{dom/xen}_heap_pages() can pull off the dirty or free list, doing
a small synchronous scrub if it was dirty and needs to be clean. 
domain_kill() can just do a pagelist_splice() to move all memory onto
the dirty list, and save 10 minutes per TB.  The boot time memory scrub
can then be implemented in terms of setting the dirty flag by default,
rather than being an explicit step.

(Although I really shouldn't be second-guessing what Boris is planning
to implement ;p)

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel