[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [Memory Accounting] was: Re: PVH dom0 creation fails - the system freezes

  • To: Roger Pau Monné <roger.pau@xxxxxxxxxx>
  • From: Juergen Gross <jgross@xxxxxxxx>
  • Date: Thu, 26 Jul 2018 15:58:43 +0200
  • Autocrypt: addr=jgross@xxxxxxxx; prefer-encrypt=mutual; keydata= xsBNBFOMcBYBCACgGjqjoGvbEouQZw/ToiBg9W98AlM2QHV+iNHsEs7kxWhKMjrioyspZKOB ycWxw3ie3j9uvg9EOB3aN4xiTv4qbnGiTr3oJhkB1gsb6ToJQZ8uxGq2kaV2KL9650I1SJve dYm8Of8Zd621lSmoKOwlNClALZNew72NjJLEzTalU1OdT7/i1TXkH09XSSI8mEQ/ouNcMvIJ NwQpd369y9bfIhWUiVXEK7MlRgUG6MvIj6Y3Am/BBLUVbDa4+gmzDC9ezlZkTZG2t14zWPvx XP3FAp2pkW0xqG7/377qptDmrk42GlSKN4z76ELnLxussxc7I2hx18NUcbP8+uty4bMxABEB AAHNHkp1ZXJnZW4gR3Jvc3MgPGpncm9zc0BzdXNlLmRlPsLAeQQTAQIAIwUCU4xw6wIbAwcL CQgHAwIBBhUIAgkKCwQWAgMBAh4BAheAAAoJELDendYovxMvi4UH/Ri+OXlObzqMANruTd4N zmVBAZgx1VW6jLc8JZjQuJPSsd/a+bNr3BZeLV6lu4Pf1Yl2Log129EX1KWYiFFvPbIiq5M5 kOXTO8Eas4CaScCvAZ9jCMQCgK3pFqYgirwTgfwnPtxFxO/F3ZcS8jovza5khkSKL9JGq8Nk czDTruQ/oy0WUHdUr9uwEfiD9yPFOGqp4S6cISuzBMvaAiC5YGdUGXuPZKXLpnGSjkZswUzY d9BVSitRL5ldsQCg6GhDoEAeIhUC4SQnT9SOWkoDOSFRXZ+7+WIBGLiWMd+yKDdRG5RyP/8f 3tgGiB6cyuYfPDRGsELGjUaTUq3H2xZgIPfOwE0EU4xwFgEIAMsx+gDjgzAY4H1hPVXgoLK8 B93sTQFN9oC6tsb46VpxyLPfJ3T1A6Z6MVkLoCejKTJ3K9MUsBZhxIJ0hIyvzwI6aYJsnOew cCiCN7FeKJ/oA1RSUemPGUcIJwQuZlTOiY0OcQ5PFkV5YxMUX1F/aTYXROXgTmSaw0aC1Jpo w7Ss1mg4SIP/tR88/d1+HwkJDVW1RSxC1PWzGizwRv8eauImGdpNnseneO2BNWRXTJumAWDD pYxpGSsGHXuZXTPZqOOZpsHtInFyi5KRHSFyk2Xigzvh3b9WqhbgHHHE4PUVw0I5sIQt8hJq 5nH5dPqz4ITtCL9zjiJsExHuHKN3NZsAEQEAAcLAXwQYAQIACQUCU4xwFgIbDAAKCRCw3p3W KL8TL0P4B/9YWver5uD/y/m0KScK2f3Z3mXJhME23vGBbMNlfwbr+meDMrJZ950CuWWnQ+d+ Ahe0w1X7e3wuLVODzjcReQ/v7b4JD3wwHxe+88tgB9byc0NXzlPJWBaWV01yB2/uefVKryAf AHYEd0gCRhx7eESgNBe3+YqWAQawunMlycsqKa09dBDL1PFRosF708ic9346GLHRc6Vj5SRA UTHnQqLetIOXZm3a2eQ1gpQK9MmruO86Vo93p39bS1mqnLLspVrL4rhoyhsOyh0Hd28QCzpJ wKeHTd0MAWAirmewHXWPco8p1Wg+V+5xfZzuQY0f4tQxvOpXpt4gQ1817GQ5/Ed/wsDtBBgB CAAgFiEEhRJncuj2BJSl0Jf3sN6d1ii/Ey8FAlrd8NACGwIAgQkQsN6d1ii/Ey92IAQZFggA HRYhBFMtsHpB9jjzHji4HoBcYbtP2GO+BQJa3fDQAAoJEIBcYbtP2GO+TYsA/30H/0V6cr/W V+J/FCayg6uNtm3MJLo4rE+o4sdpjjsGAQCooqffpgA+luTT13YZNV62hAnCLKXH9n3+ZAgJ RtAyDWk1B/0SMDVs1wxufMkKC3Q/1D3BYIvBlrTVKdBYXPxngcRoqV2J77lscEvkLNUGsu/z W2pf7+P3mWWlrPMJdlbax00vevyBeqtqNKjHstHatgMZ2W0CFC4hJ3YEetuRBURYPiGzuJXU pAd7a7BdsqWC4o+GTm5tnGrCyD+4gfDSpkOT53S/GNO07YkPkm/8J4OBoFfgSaCnQ1izwgJQ jIpcG2fPCI2/hxf2oqXPYbKr1v4Z1wthmoyUgGN0LPTIm+B5vdY82wI5qe9uN6UOGyTH2B3p hRQUWqCwu2sqkI3LLbTdrnyDZaixT2T0f4tyF5Lfs+Ha8xVMhIyzNb1byDI5FKCb
  • Cc: Wei Liu <wei.liu2@xxxxxxxxxx>, Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, George Dunlap <dunlapg@xxxxxxxxx>, Jan Beulich <JBeulich@xxxxxxxx>, abelgun@xxxxxxxxxx, xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>, Boris Ostrovsky <boris.ostrovsky@xxxxxxxxxx>, David Woodhouse <dwmw2@xxxxxxxxxxxxx>, bercarug@xxxxxxxxxx
  • Delivery-date: Thu, 26 Jul 2018 13:59:02 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Openpgp: preference=signencrypt

On 26/07/18 15:50, Roger Pau Monné wrote:
> On Thu, Jul 26, 2018 at 01:22:33PM +0200, Juergen Gross wrote:
>> On 26/07/18 13:11, Roger Pau Monné wrote:
>>> On Thu, Jul 26, 2018 at 10:45:08AM +0100, George Dunlap wrote:
>>>> On Thu, Jul 26, 2018 at 12:07 AM, Boris Ostrovsky
>>>> <boris.ostrovsky@xxxxxxxxxx> wrote:
>>>>> On 07/25/2018 02:56 PM, Andrew Cooper wrote:
>>>>>> On 25/07/18 17:29, Juergen Gross wrote:
>>>>>>> On 25/07/18 18:12, Roger Pau Monné wrote:
>>>>>>>> On Wed, Jul 25, 2018 at 05:05:35PM +0300, bercarug@xxxxxxxxxx wrote:
>>>>>>>>> On 07/25/2018 05:02 PM, Wei Liu wrote:
>>>>>>>>>> On Wed, Jul 25, 2018 at 03:41:11PM +0200, Juergen Gross wrote:
>>>>>>>>>>> On 25/07/18 15:35, Roger Pau Monné wrote:
>>>>>>>>>>>>> What could be causing the available memory loss problem?
>>>>>>>>>>>> That seems to be Linux aggressively ballooning out memory, you go 
>>>>>>>>>>>> from
>>>>>>>>>>>> 7129M total memory to 246M. Are you creating a lot of domains?
>>>>>>>>>>> This might be related to the tools thinking dom0 is a PV domain.
>>>>>>>>>> Good point.
>>>>>>>>>> In that case, xenstore-ls -fp would also be useful. The output should
>>>>>>>>>> show the balloon target for Dom0.
>>>>>>>>>> You can also try to set the autoballoon to off in /etc/xen/xl.cfg to 
>>>>>>>>>> see
>>>>>>>>>> if it makes any difference.
>>>>>>>>>> Wei.
>>>>>>>>> Also tried setting autoballooning off, but it had no effect.
>>>>>>>> This is a Linux/libxl issue that I'm not sure what's the best way to
>>>>>>>> solve. Linux has the following 'workaround' in the balloon driver:
>>>>>>>> err = xenbus_scanf(XBT_NIL, "memory", "static-max", "%llu",
>>>>>>>>                &static_max);
>>>>>>>> if (err != 1)
>>>>>>>>     static_max = new_target;
>>>>>>>> else
>>>>>>>>     static_max >>= PAGE_SHIFT - 10;
>>>>>>>> target_diff = xen_pv_domain() ? 0
>>>>>>>>             : static_max - balloon_stats.target_pages;
>>>>>>> Hmm, shouldn't PVH behave the same way as PV here? I don't think
>>>>>>> there is memory missing for PVH, opposed to HVM's firmware memory.
>>>>>>> Adding Boris for a second opinion.
>>>>> (Notwithstanding Andrews' rant below ;-))
>>>>> I am trying to remember --- what memory were we trying not to online for
>>>>> HVM here?
>>>> My general memory of the situation is this:
>>>> * Balloon drivers are told to reach a "target" value for max_pages.
>>>> * max_pages includes all memory assigned to the guest, including video
>>>> ram, "special" pages, ipxe ROMs, bios ROMs from passed-through
>>>> devices, and so on.
>>>> * Unfortunately, the balloon driver doesn't know what their max_pages
>>>> value is and can't read it.
>>>> * So what the balloon drivers do at the moment (as I understand it) is
>>>> look at the memory *reported as RAM*, and do a calculation:
>>>>   visible_ram - target_max_pages = pages_in_balloon
>>>> You can probably see why this won't work -- the result is that the
>>>> guest balloons down to (target_max_pages + non_ram_pages).  This is
>>>> kind of messy for normal guests, but when you have a
>>>> populate-on-demand guest, that leaves non_ram_pages amount of PoD ram
>>>> in the guest.  The hypervisor then spends a huge amount of work
>>>> swapping the PoD pages around under the guest's feet, until it can't
>>>> find any more zeroed guest pages to use, and it crashes the guest.
>>>> The kludge we have right now is to make up a number for HVM guests
>>>> which is slightly larger than non_ram_pages, and tell the guest to aim
>>>> for *that* instead.
>>>> I think what we need is for the *toolstack* to calculate the size of
>>>> the balloon rather than the guest, and tell the balloon driver how big
>>>> to make its balloon, rather than the balloon driver trying to figure
>>>> that out on its own.
>>> Maybe the best option would be for the toolstack to fetch the e820
>>> memory map and set the target based on the size of the RAM regions in
>>> there for PVH Dom0? That would certainly match the expectations of the
>>> guest.
>>> Note that for DomUs if hvmloader (or any other component) inside of
>>> the guest changes the memory map it would also have to adjust the
>>> value in the xenstore 'target' node.
>> How would it do that later when the guest is already running?
> hvmloader should modify the 'target' xenstore node if it changes the
> memory map.
> So the value provided by the toolstack would match the amount of RAM
> in the memory map up to the point where the guest is started, from
> there on anything inside the guest changing the memory map should also
> change the xenstore value.

So what should libxl write into target when the user specifies a new
value via "xl mem-set" then? It doesn't know whether the guest is still
trying to reach the old target, so it can't trust the current memory
allocated and target value in Xenstore to match.


Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.