Re: [Xen-devel] [PATCHv11 3/9] kexec: add infrastructure for handling kexec images

[Daniel Kiper <daniel.kiper@xxxxxxxxxx>]

On Tue, Nov 19, 2013 at 12:40:08PM +0000, Jan Beulich wrote:
> >>> On 18.11.13 at 22:50, Daniel Kiper <daniel.kiper@xxxxxxxxxx> wrote:
> > On Mon, Nov 18, 2013 at 03:24:03PM +0000, Jan Beulich wrote:
> >> >>> On 18.11.13 at 15:23, Daniel Kiper <daniel.kiper@xxxxxxxxxx> wrote:
> >> > On Mon, Nov 18, 2013 at 01:43:35PM +0000, Jan Beulich wrote:
> >> >> >>> On 18.11.13 at 14:24, Daniel Kiper <daniel.kiper@xxxxxxxxxx> wrote:
> >> >> > On Mon, Nov 18, 2013 at 12:53:39PM +0000, Jan Beulich wrote:
> >> >> >
> >> >> > [...]
> >> >> >
> >> >> >> The issue is not a limit on mappable pages, but the fact that there's
> >> >> >> potentially no struct page_info for some or all of the crash area.
> >> >> >
> >> >> > OK, are they allocated at boot time for whole system memory or just
> >> >> > for pages owned by Xen hypervsior? What about pages owned by domains?
> >> >>
> >> >> Sorry, I don't understand the question.
> >> >
> >> > Is struct page_info created for every page of system/machine memory?
> >> > Are they created at Xen boot time? Is it possible to create them later
> >> > when Xen is runnig?
> >>
> >> No - if they aren't created, then generally because there's no
> >> virtual address space to cover struct page_info itself or the 1:1
> >> mapping that would also be needed for a "normal" page.
> >
> > AIUI, frame_table is used to store struct page_info. It starts at
> > 0xffff82e000000000
> > on x86_64 and its size is 128 GiB. sizeof(struct page_info) == 32 on x86_64.
> > Hence, frame_table can store struct page_info for 16 TiB of RAM. However,
> > on the other hand 1:1 mapping has 5 TiB size + continuation of 1:1 mapping
> > 119.5 TiB == 124.5 TiB. So main ceiling here is frame_table. Could we
> > increase its size? Do you have machine with more than 16 TiB of RAM?
> > Probably yes.
> >
> > I think this issue is not kexec specific. Probably it hurts Xen in general
> > because,
> > AIUI, pages are not accessible if they do not have relevant struct page_info
> > in frame_table (or 1:1 mapping in page table).
>
> It would certainly have helped if you looked at the relevant
> changes to that code. We can't simply go beyond 16Tb, as that
> means crossing the 44-bit boundary (turning into the 32-bit
> boundary for MFNs).

I could not find any real explanation/comment/doc why 44-bit boundary.
Could you give me a hint? I have a feeling that MFN bits 32-39 were used
as flags somewhere? However, I could not find relevant code right now.

> And yes, the problem _is_ kexec specific - memory not usable
> for "normal" purposes gets ignored.

What do you mean by "normal" purposes? Xen heap, domain heap, etc.
If yes then, AIUI, it means that in general Xen will work on machines
with so huge amount of RAM but all memory above 16 TiB will be not
available. I suppose that sooner or later we would like to make Xen
working with whole memory on such machines. So are there any plans
to fix this issue? Just curious...

> > Hmmm... For what 1:1 mapping is used if same page could be mapped by
> > map_domain_page()?
>
> This is purely simplification and a performance optimization:
> Obviously you don't want e.g. each caller of xmalloc() to a
> map/unmap operation.

Right. Does it mean that whole system memory has 1:1 mapping?
Or are there some regions deliberately omitted?

> >> >> > As I can see we access crash region as it was owned by domain. AIUI,
> >> >> > it was done in that way because we wanted to be in line with normal
> >> >> > kexec case. However, I am not sure right know that this is good idea.
> >> >> > Maybe we should do something else in crash dump case?
> >> >> >
> >> >> > We could establish an limit (4 GiB?) for crash region as a workaround.
> >> >>
> >> >> Are you taking of a size limit or an address one?
> >> >
> >> > Size.
> >>
> >> So how would limiting the size help?
> >
> > I thought that struct page_info for every page is created in another way.
> > Currently we could require that crash dump region should end below or
> > at 16 TiB or increase frame_table size if it is possible.
>
> The former is exactly what I was asking to be done.

Probably I will prepare relevant patches next week.

Do you have machine with so huge amount of RAM to do some tests?

Daniel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel