[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] How to deal with hypercalls returning -EFAULT


  • To: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • From: Juergen Gross <jgross@xxxxxxxx>
  • Date: Thu, 14 Jun 2018 08:38:02 +0200
  • Autocrypt: addr=jgross@xxxxxxxx; prefer-encrypt=mutual; keydata= xsBNBFOMcBYBCACgGjqjoGvbEouQZw/ToiBg9W98AlM2QHV+iNHsEs7kxWhKMjrioyspZKOB ycWxw3ie3j9uvg9EOB3aN4xiTv4qbnGiTr3oJhkB1gsb6ToJQZ8uxGq2kaV2KL9650I1SJve dYm8Of8Zd621lSmoKOwlNClALZNew72NjJLEzTalU1OdT7/i1TXkH09XSSI8mEQ/ouNcMvIJ NwQpd369y9bfIhWUiVXEK7MlRgUG6MvIj6Y3Am/BBLUVbDa4+gmzDC9ezlZkTZG2t14zWPvx XP3FAp2pkW0xqG7/377qptDmrk42GlSKN4z76ELnLxussxc7I2hx18NUcbP8+uty4bMxABEB AAHNHkp1ZXJnZW4gR3Jvc3MgPGpncm9zc0BzdXNlLmRlPsLAeQQTAQIAIwUCU4xw6wIbAwcL CQgHAwIBBhUIAgkKCwQWAgMBAh4BAheAAAoJELDendYovxMvi4UH/Ri+OXlObzqMANruTd4N zmVBAZgx1VW6jLc8JZjQuJPSsd/a+bNr3BZeLV6lu4Pf1Yl2Log129EX1KWYiFFvPbIiq5M5 kOXTO8Eas4CaScCvAZ9jCMQCgK3pFqYgirwTgfwnPtxFxO/F3ZcS8jovza5khkSKL9JGq8Nk czDTruQ/oy0WUHdUr9uwEfiD9yPFOGqp4S6cISuzBMvaAiC5YGdUGXuPZKXLpnGSjkZswUzY d9BVSitRL5ldsQCg6GhDoEAeIhUC4SQnT9SOWkoDOSFRXZ+7+WIBGLiWMd+yKDdRG5RyP/8f 3tgGiB6cyuYfPDRGsELGjUaTUq3H2xZgIPfOwE0EU4xwFgEIAMsx+gDjgzAY4H1hPVXgoLK8 B93sTQFN9oC6tsb46VpxyLPfJ3T1A6Z6MVkLoCejKTJ3K9MUsBZhxIJ0hIyvzwI6aYJsnOew cCiCN7FeKJ/oA1RSUemPGUcIJwQuZlTOiY0OcQ5PFkV5YxMUX1F/aTYXROXgTmSaw0aC1Jpo w7Ss1mg4SIP/tR88/d1+HwkJDVW1RSxC1PWzGizwRv8eauImGdpNnseneO2BNWRXTJumAWDD pYxpGSsGHXuZXTPZqOOZpsHtInFyi5KRHSFyk2Xigzvh3b9WqhbgHHHE4PUVw0I5sIQt8hJq 5nH5dPqz4ITtCL9zjiJsExHuHKN3NZsAEQEAAcLAXwQYAQIACQUCU4xwFgIbDAAKCRCw3p3W KL8TL0P4B/9YWver5uD/y/m0KScK2f3Z3mXJhME23vGBbMNlfwbr+meDMrJZ950CuWWnQ+d+ Ahe0w1X7e3wuLVODzjcReQ/v7b4JD3wwHxe+88tgB9byc0NXzlPJWBaWV01yB2/uefVKryAf AHYEd0gCRhx7eESgNBe3+YqWAQawunMlycsqKa09dBDL1PFRosF708ic9346GLHRc6Vj5SRA UTHnQqLetIOXZm3a2eQ1gpQK9MmruO86Vo93p39bS1mqnLLspVrL4rhoyhsOyh0Hd28QCzpJ wKeHTd0MAWAirmewHXWPco8p1Wg+V+5xfZzuQY0f4tQxvOpXpt4gQ1817GQ5/Ed/wsDtBBgB CAAgFiEEhRJncuj2BJSl0Jf3sN6d1ii/Ey8FAlrd8NACGwIAgQkQsN6d1ii/Ey92IAQZFggA HRYhBFMtsHpB9jjzHji4HoBcYbtP2GO+BQJa3fDQAAoJEIBcYbtP2GO+TYsA/30H/0V6cr/W V+J/FCayg6uNtm3MJLo4rE+o4sdpjjsGAQCooqffpgA+luTT13YZNV62hAnCLKXH9n3+ZAgJ RtAyDWk1B/0SMDVs1wxufMkKC3Q/1D3BYIvBlrTVKdBYXPxngcRoqV2J77lscEvkLNUGsu/z W2pf7+P3mWWlrPMJdlbax00vevyBeqtqNKjHstHatgMZ2W0CFC4hJ3YEetuRBURYPiGzuJXU pAd7a7BdsqWC4o+GTm5tnGrCyD+4gfDSpkOT53S/GNO07YkPkm/8J4OBoFfgSaCnQ1izwgJQ jIpcG2fPCI2/hxf2oqXPYbKr1v4Z1wthmoyUgGN0LPTIm+B5vdY82wI5qe9uN6UOGyTH2B3p hRQUWqCwu2sqkI3LLbTdrnyDZaixT2T0f4tyF5Lfs+Ha8xVMhIyzNb1byDI5FKCb
  • Cc: Lars Kurth <lars.kurth@xxxxxxxxxx>, Boris Ostrovsky <boris.ostrovsky@xxxxxxxxxx>, "committers@xxxxxxxxxxxxxx" <committers@xxxxxxxxxxxxxx>
  • Delivery-date: Thu, 14 Jun 2018 06:38:28 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Openpgp: preference=signencrypt

On 13/06/18 18:25, Andrew Cooper wrote:
> On 13/06/18 16:27, Juergen Gross wrote:
>> Currently the release of Xen 4.11 is blocked due to a sporadic failure
>> of the OSSTEST guest-saverestore[.2]. During that test a hypercall
>> issued by libxc via the Linux privcmd driver returns -EFAULT in spite
>> of all hypercall buffers locked in memory via mlock() (or similar flags
>> specified in a mmap() call).
>>
>> My analysis has revealed that modern Linux kernels might make such
>> locked user pages unaccessible for very short periods of time. This can
>> happen e.g. when pages are subject to compaction or migration.
>>
>> There are multiple ways to mitigate this problem:
>>
>> 1. Trying to switch page migration or compaction off in dom0.
>>    Pros: - no change in Xen necessary
> 
> Pro: can likely retrofitted to existing environments without further
> code changes.
> 
> (Not that I disagree with your Con's in this case)
> 
>>    Cons: - new cases might come up in the future
>>          - easy to miss, failures are really very sporadic and might
>>            happen only after updating the kernel
>>
>> 2. Add a bandaid to Xen tools by retrying hypercalls which have failed
>>    with -EFAULT (either for all or only for some hypercalls)
>>    Pros: - no interface change necessary
>>    Cons: - not all hypercalls might be just repeatable
>>          - problem isn't solved but just worked around
> 
> We'd have to whitelist hypercalls which are safe to repeat like this. 

Right.

> Most wont be.  Any mutable operation which -EFAULTs can't safely be
> restarted, because we can't distinguish an early fault (Xen reading the
> parameters) from a late fault (Xen trying to update a userspace pointer
> with the result).

Even some of the late fault cases are repeatable.

> 
>>
>> 3. Modify the interface to the privcmd driver to pass information about
>>    used buffers to the kernel in order to lock them there. Either add a
>>    new interface for hypercall buffer management or add the list of
>>    buffers to the privcmd ioctl data structure.
>>    Pros: - problem is really solved
>>    Cons: - split solution between kernel and Xen, both must be changed
> 
> To be clear, you mean suggesting changing libxc here, rather than the
> hypervisor?

Yes.

> Getting this problem fixed properly would be a distinct improvement over
> the whack-a-mole which has been played in the past.

Looking more into the privcmd driver I'm rather sure now I can solve
this via mmap() plus an ioctl(). So the kernel would allocate the
bounce buffers and map them into user space. This will require changes
only in libxencall without having to modify its external interface.

> 
>>
>> 4. Modify the interface between hypervisor and kernel: instead of just
>>    returning -EFAULT let the hypervisor behave more like copy_to_user by
>>    raising a page fault which can then be fixed up in the kernel. This
>>    change must be activated by the kernel, of course.
>>    Pros: - rather simple change in the kernel "doing the right thing"
>>          - hypercall bounce buffer handling in libxc/libxencall can be
>>            switched off for a kernel supporting this chnage
>>    Cons: - split solution between kernel and Xen, both must be changed
>>          - not sure how complex the required hypervisor change will be
> 
> Sadly, as I've just realised...
> 
> Con: Cannot be used to replace all -EFAULTs.
> 
> Faults when copying data in can be resolved by passing #PF to the
> kernel, but faults when trying to update guest state (continuation, or
> completion information) cannot be safely resumed at a later point.

Hmm, seems you are right. Sigh.

> 
>>
>> It should be noted that we can either select only one of above solutions
>> or one of 3/4 and additionally one of 1/2 as a fallback for old kernels.
>>
>> How to proceed?
> 
> Much as I hate to say it (as I do like this idea), I don't idea 4 is a
> viable alternative to 3.

Yeah, but idea 3 via mmap() isn't too bad in the end...


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.