Re: [Xen-devel] Can't always start 32 bit domains after 64 bit domains

["Jan Beulich" <JBeulich@xxxxxxxx>]

>>> On 19.11.16 at 22:22, <srn@xxxxxxxxx> wrote:
> Last night on a 288GiB server with less than 64GiB of 32 bit
> domUs, we used the standard xendomains script which starts VMs
> in alphabetical order.
> 
> Some 32 bit domUs at the very end were unable to start. The
> error message we received is the following:
> 
> xc: error: panic: xc_dom_x86.c:944: arch_setup_meminit: failed to allocate 
> 0x80000 pages: Internal error
> xc: error: panic: xc_dom_boot.c:154: xc_dom_boot_mem_init: can't allocate 
> low memory for domain: Out of memory
> libxl: error: libxl_dom.c:719:libxl__build_pv: xc_dom_boot_mem_init failed: 
> Device or resource busy
> libxl: error: libxl_create.c:1144:domcreate_rebuild_done: cannot (re-)build 
> domain: -3
> 
> After shutting down some 64 bit domUs, we could start the remainder
> of the 32 bit domUs. Finally we started the rest of the 64 bit
> domUs such that everything was booted.
> 
> My current understanding is that on a server with more than 168GiB
> of memory, I should still be able to around 128GiB of 32-bit PV
> domUs, regardless of what order the domUs are started in.

You don't clarify what you base this understanding of yours on; I don't
think this is the case. What exact memory will be allocated for 64-bit
guests isn't very predictable. In particular, the preference of allocating
1Gb pages as long as available may result in allocations eating into the
low 128Gb of memory earlier than you expect. You may want to analyze
system state by looking at debug key output at relevant points in time.

Back in the xend days someone here had invented a (crude) mechanism
to set aside memory for 32-bit PV domains, but I don't think dealing with
this situation in xl has ever seen any interest.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel