Re: [Xen-devel] Can't always start 32 bit domains after 64 bit domains

[Sarah Newman <srn@xxxxxxxxx>]

On 11/21/2016 11:37 AM, Sarah Newman wrote:
> On 11/21/2016 05:21 AM, Andrew Cooper wrote:
>> On 21/11/16 10:05, Jan Beulich wrote:
> 
>>>>> Back in the xend days someone here had invented a (crude) mechanism
>>>>> to set aside memory for 32-bit PV domains, but I don't think dealing with
>>>>> this situation in xl has ever seen any interest.
>>>> If I wanted to add that, where would it go?
>>> I don't know, that's a question for xl folks I guess.
>>
>> IIRC, given no other constraints, the Xen heap allocation allocates from
>> the top down to help this exact case.
> 
> Yes, that's what I thought it did too, though I can't find my source 
> information for that.
> 
>>
>> I suspect that libxl's preference towards NUMA allocation of domains
>> interferes with this, by adding a NUMA constraints to memory allocations
>> for 64bit PV guests.
> 
> I ran xl info -n (which I didn't know about before) and that shows the 
> problem much more clearly.
> 
> If that's the reason not all the higher memory is being used first: is a 
> potential workaround to pin 64 bit domains to the second physical core on
> boot, and 32 bit domains to the first physical core on boot, and then change 
> the allowed cores with 'xl vcpu-pin' after the domain is loaded?

Free memory on a test server with no domains:
node:    memsize    memfree    distances
   0:    148480     142983      10,21
   1:    147456     144645      21,10

Free memory booting 116 256M 64-bit domains, limited to cpus='all,^0-1' on boot:
node:    memsize    memfree    distances
   0:    148480     128416      10,21
   1:    147456     129669      21,10

Free memory booting 116 256M 64-bit domains, limited to cpus='12-23' on boot:
node:    memsize    memfree    distances
   0:    148480     143397      10,21
   1:    147456     114693      21,10

This looks like a viable workaround. Where should I document it?

--Sarah

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel