[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH v1] domctl: hold domctl lock while domain is destroyed


  • To: Dmitry Isaikin <isaikin-dmitry@xxxxxxxxx>
  • From: Roger Pau Monné <roger.pau@xxxxxxxxxx>
  • Date: Thu, 16 Sep 2021 15:08:03 +0200
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=zKZBz67gWrmfGh5fqXsQOaiN41yOf90IY5B3x2Hnsos=; b=LBkRAmcSeOFphr324EuqoDhPlMN4ReZAGm4nu8PW5Etpm0PPITDyAKzKQo0y9SqBK/WpFFdn3BEQ4zlzwuW7itwwzWBKZmtVbP27r7OhaEeHDzviWlPPpGjNGNbFgE4r9y83cxL72O6S0mxRuyJxQb6NRVmWnvjtl0JcGLfnjbqTU4D8a2I5USS0XdXiiHYOKZG1I/mgNS9fb6spjImF4j4oei3Yv5Rx7JVxrBTaAMlPcJlXgGXIjV5bIbVbnWJQOEEIoCdPjV8ZX9xI7TmLfaiLyEcXGBk2J4mNSIo6KgQDuhlrZEMO1egcalqA3OMuKRRoM31RdlSHIQAuxu9dOw==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=moSY1Dr4eGwyAHCgh6vfaRsX/x0mBD5S2K4HD4U0esPZotWhNopalpb1IKGMx09HSqClDBoZGVyNBU/U5FX16tpEQlhBDHDn7iJLbEt7xhX78HpaH7ajnrFMzvkwYCybLoc72HjoJ1bBT9Yp7kDawThvvKlVg8vpwXD+KVAuIrtHPCOdu+detXtU8Zuk0h2A7inZKgE6aJBjrHbFhwNnkgRmaEKnx3vmY3TyTdqKx7kzLtfnyebQgtSX53N8232hhb4yHOs1ZYPk+RuKEZz80ih6u4mdLKXyLici5kYtiQp4uNUZA8Xec9aUKk4o4+GbbJKD9fsGXbvAlYMZFBiNsw==
  • Authentication-results: esa6.hc3370-68.iphmx.com; dkim=pass (signature verified) header.i=@citrix.onmicrosoft.com
  • Cc: Jan Beulich <jbeulich@xxxxxxxx>, Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, George Dunlap <george.dunlap@xxxxxxxxxx>, Ian Jackson <iwj@xxxxxxxxxxxxxx>, Julien Grall <julien@xxxxxxx>, Stefano Stabellini <sstabellini@xxxxxxxxxx>, Wei Liu <wl@xxxxxxx>, <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • Delivery-date: Thu, 16 Sep 2021 13:14:21 +0000
  • Ironport-data: A9a23:wlqu0aC7Ce/7BhVW/7/kw5YqxClBgxIJ4kV8jS/XYbTApD53gmFWm GcWXD3TOf2PZmL2fN0kYYm3oU1X7MSGz4RmQQY4rX1jcSlH+JHPbTi7wuYcHM8wwunrFh8PA xA2M4GYRCwMo/u1Si6FatANl1ElvU2zbue6WLOs1hxZH1c+EX9w0Ew7wYbVv6Yz6TSHK1LV0 T/Ni5W31G+Ng1aY5UpNtspvADs21BjDkGtwUm4WPJinj3eH/5UhN7oNJLnZEpfNatI88thW5 Qr05OrREmvxp3/BAz4++1rxWhVirrX6ZWBihpfKMkQLb9crSiEai84G2PQghUh/2naVoMt0j +d0tt+gFQIMZIzgtPQhTEwNe81+FfUuFL7vJHG+tYqYzlHccmuqyPJrZK00FdRGoKAtWzgIr KFGbmBWBvyAr7veLLaTUO5ji95lNMD2FIgepmth3XfSCvNOrZXrHv+auoEGg2lYasZmLMjee c9ITwRTKyvLPU0UP0YeMo40tbL97pX4W2IB8w/EzUYt2EDI1xB42rXpNNvTe/SJSN9Tk0Leo XjJl0zmBjkKOdrZziCKmlq8i+mKkS7lVYY6ELyj6uUskFCV3nYUChAdSR28u/bRokyjX9NSL WQE9yxoqrI9nGSqVdD8XBy+5XKBvwIGSvJBGus25RHL0Le8ywSTC3UATzVBQMc7r8JwTjsvv neIgNHBFTFpqKeSS3+W6vGTtzzaBMQOBTZcP2leF1JDuoS95tFo5v7Scjp9OP/2kuLuHHbC+ jPJvHQ7ou8yoeIb67ruqDgrnAmQSoj1oh8dv1uMBz/0sVwnPeZJdKTztgOKtq8owJKxCwDb5 SlawZD2APUmUMnV/BFhVtnhC11ACxytCzTanVcnNJ0o7T3FF5WLLN0IvW0WyKuENK85ld7Vj K374lg5CHx7ZiLCgUpLj2SZUJxCIU/IT4+NaxwsRoASCqWdjSfelM2UWaJ144wLuBN0+U3YE czAGftA8F5AUfg3pNZIb75FjNfHORzSNUuMHMumnnxLIJK1ZWKPSKdtDbd9Rrlit8u5TPHu2 48HbaOikkwHOMWnO3W/2dNDfDgicClgbbir+pM/SwJ2Clc/cI3XI6SKmu1Jlk0Mt/k9q9okC VnmBhYDmQag3CaeQehIA1g6AI7SsV9EhStTFQQnPEqy2mhlZoCq7awFcIAwc6Vh/+tmpcOYh dFbIJvo7i1nRmuV9jIDQ4P6qYA+Jh2niRjXZ3iuYSQlfo4mTAvMo4e2cgzq/SgILyy2qcph/ OHwilKFGcIOF1Z4EcLbSPOz1Fft73ITr/1/AhnTKd5JdUSyrIUzc37tjuU6Kt0nIAnYwmfIz B6fBBoV/LGfo4I8/NTTq7qDqoOlT7l3EkZARjGJ5reqLyjKuGGkxNYYAuqPeDncUkLy+bmjO roJn62tbqVfkQ8T4YRmErttwaYv3PfVpudXnlZ+AXHGT1W3Ebc8cHOI6tZC6/9WzbhDtAroB k/WootGOa+EMd/OGUIKIFZ3dfyK0PwZl2WA7fkxJ0mmtiZ78KDeDBdXNhiIzidcMKF0IMUux uJ44Jwa7Am2ixwLNNeaj38LqzTQfyJYC6h35IsHBILLixYwzgAQaJPRPSb6/ZWTZogeKUItO DKV2PLPirk0KpAuqJbv+awhBdZguKk=
  • Ironport-hdrordr: A9a23:diBGiKD8UebtnerlHeg3sceALOsnbusQ8zAXPh9KJiC9I/b1qy nxppkmPH/P6Qr4WBkb6Le90Y27MAnhHP9OkPIs1NKZMjUO11HYTr2KgbGSpgEIXheOi9K1tp 0QDZSWaueAdGSS5PySiGLTc6dCsai6GeKT9J/jJh9WPH5XgspbnmFE42igYylLrF4sP+tEKH PQ3LsMmxOQPVAsKuirDHgMWObO4/XNiZLdeBYDQzoq8hOHgz+E4KPzV0Hw5GZTbxp/hZMZtU TVmQ3w4auu99m91x/nzmfWq7BbgsHoxNdvDNGFzuIVNjLvoAC1Y5kJYczOgBkF5MWUrHo6mt jFpBkte+x19nPqZ2mw5SDg3gHxuQxeokPK+Bu9uz/OsMb5TDU1B45qnoRCaCbU7EImoZVVzL 9L93jxjesXMTrw2ADGo/TYXRBjkUS55VA4l/QIsnBZWYwCLJdMsI0k+l9PGptoJlO11GkeKp gsMCjg3ocVTbvDBEqp+1WHgebcEUjbJy32AnTr4aeuonxrdHMQ9Tpv+CVQpAZGyHqRI6M0kt gsCZ4Y542mePVmGp6VNN1xMfdfNVa9My4kEFjiaWgPR5t3cE4klfbMkcEIDaeRCdo18Kc=
  • Ironport-sdr: oW5rQi6Uoj8yvAyG/Z05o7vFxqbxhBDcciWdfMmU5vVLKy7CUM5To9io+pGAoCFHY7BDnF6LWS 8jnjpZrDbXMrwvcgqu1OaBLsRbncO8aYwyjF61xwoqMHG9+9koKGIx1Ni/Cu3Ccy+7LyZtmIlK AJped7FAIK2zEuHnbmxiZrzdrMvD+n2oYhygZWY1UNzwxyBeNb8QTLjRJi0ldOcAdqutIErWgK K/vS0awPdwdLo1YrChnuoi3MDW2KtCd46JyPdROvQ4NutU2Da9CKtFJsiJQDJQjTc5RdTw5V1Q M6/h/NmaUjsa3i8zmkaPvvpN
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On Thu, Sep 16, 2021 at 02:30:39PM +0200, Jan Beulich wrote:
> On 16.09.2021 13:10, Dmitry Isaikin wrote:
> > From: Dmitry Isaykin <isaikin-dmitry@xxxxxxxxx>
> > 
> > This significantly speeds up concurrent destruction of multiple domains on 
> > x86.
> 
> This effectively is a simplistic revert of 228ab9992ffb ("domctl:
> improve locking during domain destruction"). There it was found to
> actually improve things; now you're claiming the opposite. It'll
> take more justification, clearly identifying that you actually
> revert an earlier change, and an explanation why then you don't
> revert that change altogether. You will want to specifically also
> consider the cleaning up of huge VMs, where use of the (global)
> domctl lock may hamper progress of other (parallel) operations on
> the system.
> 
> > I identify the place taking the most time:
> > 
> >     do_domctl(case XEN_DOMCTL_destroydomain)
> >       -> domain_kill()
> >            -> domain_relinquish_resources()
> >                 -> relinquish_memory(d, &d->page_list, PGT_l4_page_table)
> > 
> > My reference setup: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, Xen 4.14.
> > 
> > I use this command for test:
> > 
> >     for i in $(seq 1 5) ; do xl destroy test-vm-${i} & done
> > 
> > Without holding the lock all calls of `relinquish_memory(d, &d->page_list, 
> > PGT_l4_page_table)`
> > took on my setup (for HVM with 2GB of memory) about 3 seconds for each 
> > destroying domain.
> > 
> > With holding the lock it took only 100 ms.
> 
> I'm further afraid I can't make the connection. Do you have an
> explanation for why there would be such a massive difference?
> What would prevent progress of relinquish_memory() with the
> domctl lock not held?

I would recommend to Dmitry to use lock profiling with and without
this change applied and try to spot which lock is causing the
contention as a starting point. That should be fairly easy and could
share some light.

Regards, Roger.



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.