Re: [Embedded-pv-devel] [Xen-devel] Deadlock in /proc/xen/xenbus watch+read on 3.17+ (maybe earlier)

[Iurii Konovalenko <iurii.konovalenko@xxxxxxxxxxxxxxx>]

Hi, guys!

When I read, that I am not alone and that issue depends on kernel
version, I decided to continue investigation.
And I found why our threads locks on read/write operations.
On Linux kernel 3.14+ syscalls of file read and write changed a bit:
fdget() function was replaced by fdget_pos() - it is fdget() function
plus additional position mutex lock for files with FMODE_ATOMIC_POS
(files for inodes with S_IFREG flag set - regular nodes). As I thought
our xen files are not regular and nonseekable, I hoped this flag is
not set. But it is set. It is because our file system is created by
function simple_fill_super(), and inside it this flag is hardly set:
inode->i_mode = S_IFREG | files->mode;
So, as a fast hack I made a patch: just made copy of this function for
xen, which does not set this flag. It works for me. Could you please
check if it works for you.
Best regards.

Iurii Konovalenko | Senior Software Engineer
GlobalLogic
P +3.8044.492.9695 M +38.099.932.2909
S yufuntik
www.globallogic.com
http://www.globallogic.com/email_disclaimer.txt


On Thu, Mar 19, 2015 at 12:38 PM, Ian Campbell <ian.campbell@xxxxxxxxxx> wrote:
> On Thu, 2015-03-19 at 02:19 +0100, Marek Marczykowski-GÃrecki wrote:
>> Hi,
>>
>> I've hit some deadlock in kernel xenstore client exposed via
>> /proc/xen/xenbus.
>
> Sounds similar to what Iurii also reported last night in "Userspace PV
> backend hangs".
>
> Iurri's case was all 3.14 kernels, which is in your range too.
>
>>  Steps to reproduce are simple:
>> int main() {
>>       struct xs_handle *xs;
>>       xs = xs_open(0);
>>       xs_watch(xs, "domid", "token");
>>       xs_read(xs, 0, "name", NULL);
>>       return 0;
>> }
>>
>> xs_watch internally creates new thread, which uses read to wait for the
>> watch. And in the same time, the program tries to read some value,
>> but actually it hangs at sending the command (before even sending a path to 
>> be
>> read). Strace gives this (simplified for readability):
>> [pid  2494] write(3, "\4\0\0\0\0\0\0\0\0\0\0\0\f\0\0\0", 160 = 16
>> [pid  2494] write(3, "domid\0", 6)      = 6
>> [pid  2494] write(3, "token\0", 6)      = 6
>> [pid  2495] read(3,  <unfinished ...>
>> [pid  2494] futex(0x71c0d4, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
>> [pid  2495] <... read resumed>
>> "\17\0\0\0\377\377\377\377\220~\255\27\f\0\0\0", 16) = 16
>> [pid  2495] read(3, "domid\0token\0", 12) = 12
>> [pid  2495] read(3, "\4\0\0\0\0\0\0\0\0\0\0\0\3\0\0\0", 16) = 16
>> [pid  2495] read(3, "OK\0", 3)          = 3
>> [pid  2495] futex(0x71c0d4, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x71c0d0,
>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1} <unfinished ...>
>> [pid  2494] <... futex resumed> )       = 0
>> [pid  2495] <... futex resumed> )       = 1
>> [pid  2494] futex(0x71c0a8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
>> [pid  2495] futex(0x71c0a8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
>> [pid  2494] <... futex resumed> )       = -1 EAGAIN (Resource
>> temporarily unavailable)
>> [pid  2495] <... futex resumed> )       = 0
>> [pid  2494] futex(0x71c0a8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
>> [pid  2495] read(3,  <unfinished ...>
>> [pid  2494] <... futex resumed> )       = 0
>> [pid  2494] rt_sigaction(SIGPIPE, {SIG_DFL, [], SA_RESTORER,
>> 0x7fc78c1488f0}, NULL, 8) = 0
>> [pid  2494] rt_sigaction(SIGPIPE, {SIG_IGN, [], SA_RESTORER,
>> 0x7fc78c1488f0}, {SIG_DFL, [], SA_RESTORER, 0x7fc78c1488f0}, 8) = 0
>> [pid  2494] write(3, "\2\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0", 16
>>
>> And thats all - 2494 is waiting on write, 2495 is waiting on read.
>>
>> On 3.12.x it is working. On 3.17.0 and 3.18.7 it is broken. I haven't
>> checked versions in the middle.
>>
>> Any ideas?
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@xxxxxxxxxxxxx
>> http://lists.xen.org/xen-devel
>
>

Attachment: 0001-arm-xen-hack-to-make-xen-files-unregular.patch
Description: Text Data

_______________________________________________
Embedded-pv-devel mailing list
Embedded-pv-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/embedded-pv-devel