Re: [Xen-devel] a ton of kernel issues

[Ian Campbell <Ian.Campbell@xxxxxxxxxx>]

On Wed, 2011-12-14 at 13:11 +0000, Jan Beulich wrote:
> >>> On 14.12.11 at 13:16, Ian Campbell <Ian.Campbell@xxxxxxxxxx> wrote:
> > It works with xm/xend because they set the maximum reservation for
> > guests to static-max on boot. xl (and, I think, xapi) instead set the
> > maximum reservation to the current balloon target and change it
> 
> Hmm, is that even compatible with the old (non-pvops) kernels? I
> always assumed that the maximum reservation was a true maximum,
> not something that can change at any time during the life of a VM.
> What's the purpose of this being adjustable and there also being a
> current target?

The target is a target which we ask the guest to aim for, the maximum is
a hard maximum which Xen will actually enforce.

Toolstack can set them the same if they want to make sure that the guest
does not try and exceed its current target. Setting the maximum doesn't
reduce the memory a guest is currently using but it does ensure that
progress towards the target never regresses.

The xapi toolstack has been doing this for several years and we aren't
aware of any incompatibilities, there were some bugs in various kernels
(they would give up ballooning altogether if they hit the hard limit)
but they were all fixed quite some time ago IIRC.

> Is the old tool stack also setting up a suitable E820 table?

I believe so, this behaviour long predates xl. The difference with xend
is that it doesn't set the maximum to the target but rather to the
static-max, this is a toolstack policy decision.

> Assuming
> so, is it at all reasonable to have the outer world establish a static
> upper bound on the VM if such an immutable upper bound doesn't
> really exist

Yes, this is what many host administrators want, they want to control
this stuff via the host configuration APIs rather than by digging into
each guest.

>  (rather than having the VM determine for itself - via
> "mem=" - how far it wants to be able to go)?

mem= takes precedence over the e820 table, so we effectively have the
best of both worlds.

Ian.

> 
> Jan
> 
> > dynamically as the target is changed (as a method of enforcing the
> > targets). However the pvops kernel incorrectly uses the maximum
> > reservation at boot to size the physical address space for guests.
> > 
> > The patch below fixes this.
> > 
> > Ian.
> > 
> > 8<-------------------------------------------------------------
> > 
> > From 649ca3b7ddca1cdda85c27e34f806f30484172ec Mon Sep 17 00:00:00 2001
> > From: Ian Campbell <ian.campbell@xxxxxxxxxx>
> > Date: Wed, 14 Dec 2011 12:00:38 +0000
> > Subject: [PATCH] xen: only limit memory map to maximum reservation for 
> > domain 0.
> > 
> > d312ae878b6a "xen: use maximum reservation to limit amount of usable RAM"
> > clamped the total amount of RAM to the current maximum reservation. This is
> > correct for dom0 but is not correct for guest domains. In order to boot a 
> > guest
> > "pre-ballooned" (e.g. with memory=1G but maxmem=2G) in order to allow for
> > future memory expansion the guest must derive max_pfn from the e820 
> > provided 
> > by
> > the toolstack and not the current maximum reservation (which can reflect 
> > only
> > the current maximum, not the guest lifetime max). The existing algorithm
> > already behaves this correctly if we do not artificially limit the maximum
> > number of pages for the guest case.
> > 
> > For a guest booted with maxmem=512, memory=128 this results in:
> >  [    0.000000] BIOS-provided physical RAM map:
> >  [    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
> >  [    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
> > -[    0.000000]  Xen: 0000000000100000 - 0000000008100000 (usable)
> > -[    0.000000]  Xen: 0000000008100000 - 0000000020800000 (unusable)
> > +[    0.000000]  Xen: 0000000000100000 - 0000000020800000 (usable)
> > ...
> >  [    0.000000] NX (Execute Disable) protection: active
> >  [    0.000000] DMI not present or invalid.
> >  [    0.000000] e820 update range: 0000000000000000 - 0000000000010000 
> > (usable) ==> (reserved)
> >  [    0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 
> > (usable)
> > -[    0.000000] last_pfn = 0x8100 max_arch_pfn = 0x1000000
> > +[    0.000000] last_pfn = 0x20800 max_arch_pfn = 0x1000000
> >  [    0.000000] initial memory mapped : 0 - 027ff000
> >  [    0.000000] Base memory trampoline at [c009f000] 9f000 size 4096
> > -[    0.000000] init_memory_mapping: 0000000000000000-0000000008100000
> > -[    0.000000]  0000000000 - 0008100000 page 4k
> > -[    0.000000] kernel direct mapping tables up to 8100000 @ 27bb000-27ff000
> > +[    0.000000] init_memory_mapping: 0000000000000000-0000000020800000
> > +[    0.000000]  0000000000 - 0020800000 page 4k
> > +[    0.000000] kernel direct mapping tables up to 20800000 @ 
> > 26f8000-27ff000
> >  [    0.000000] xen: setting RW the range 27e8000 - 27ff000
> >  [    0.000000] 0MB HIGHMEM available.
> > -[    0.000000] 129MB LOWMEM available.
> > -[    0.000000]   mapped low ram: 0 - 08100000
> > -[    0.000000]   low ram: 0 - 08100000
> > +[    0.000000] 520MB LOWMEM available.
> > +[    0.000000]   mapped low ram: 0 - 20800000
> > +[    0.000000]   low ram: 0 - 20800000
> > 
> > With this change "xl mem-set <domain> 512M" will successfully increase the
> > guest RAM (by reducing the balloon).
> > 
> > There is no change for dom0.
> > 
> > Signed-off-by: Ian Campbell <ian.campbell@xxxxxxxxxx>
> > Cc: stable@xxxxxxxxxx 
> > Cc: David Vrabel <david.vrabel@xxxxxxxxxx>
> > Cc: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
> > ---
> >  arch/x86/xen/setup.c |   18 +++++++++++++++---
> >  1 files changed, 15 insertions(+), 3 deletions(-)
> > 
> > diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
> > index 38d0af4..a54ff1a 100644
> > --- a/arch/x86/xen/setup.c
> > +++ b/arch/x86/xen/setup.c
> > @@ -173,9 +173,21 @@ static unsigned long __init xen_get_max_pages(void)
> >     domid_t domid = DOMID_SELF;
> >     int ret;
> >  
> > -   ret = HYPERVISOR_memory_op(XENMEM_maximum_reservation, &domid);
> > -   if (ret > 0)
> > -           max_pages = ret;
> > +   /*
> > +    * For the initial domain we use the maximum reservation as
> > +    * the maximum page.
> > +    *
> > +    * For guest domains the current maximum reservation reflects
> > +    * the current maximum rather than the static maximum. In this
> > +    * case the e820 map provided to us will cover the static
> > +    * maximum region.
> > +    */
> > +   if (xen_initial_domain()) {
> > +           ret = HYPERVISOR_memory_op(XENMEM_maximum_reservation, &domid);
> > +           if (ret > 0)
> > +                   max_pages = ret;
> > +   }
> > +
> >     return min(max_pages, MAX_DOMAIN_PAGES);
> >  }
> >  
> > -- 
> > 1.7.2.5
> > 
> > 
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxxxxxxxx 
> > http://lists.xensource.com/xen-devel 
> 
> 



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel