Re: [Embedded-pv-devel] [Xen-devel] Deadlock in /proc/xen/xenbus watch+read on 3.17+ (maybe earlier)

[Vitaly Chernooky <vitalii.chernookyi@xxxxxxxxxxxxxxx>]

Hi All!

This issue is discussed in details here:Âhttps://lkml.org/lkml/2014/2/17/324

With best regards,

On Thu, Mar 19, 2015 at 2:10 PM, Iurii Konovalenko <iurii.konovalenko@xxxxxxxxxxxxxxx> wrote:

Hi, guys!

When I read, that I am not alone and that issue depends on kernel
version, I decided to continue investigation.
And I found why our threads locks on read/write operations.
On Linux kernel 3.14+ syscalls of file read and write changed a bit:
fdget() function was replaced by fdget_pos() - it is fdget() function
plus additional position mutex lock for files with FMODE_ATOMIC_POS
(files for inodes with S_IFREG flag set - regular nodes). As I thought
our xen files are not regular and nonseekable, I hoped this flag is
not set. But it is set. It is because our file system is created by
function simple_fill_super(), and inside it this flag is hardly set:
inode->i_mode = S_IFREG | files->mode;
So, as a fast hack I made a patch: just made copy of this function for
xen, which does not set this flag. It works for me. Could you please
check if it works for you.
Best regards.

Iurii Konovalenko | Senior Software Engineer
GlobalLogic
P +3.8044.492.9695 M +38.099.932.2909
S yufuntik
www.globallogic.com
http://www.globallogic.com/email_disclaimer.txt

On Thu, Mar 19, 2015 at 12:38 PM, Ian Campbell <ian.campbell@xxxxxxxxxx> wrote:
> On Thu, 2015-03-19 at 02:19 +0100, Marek Marczykowski-GÃrecki wrote:
>> Hi,
>>
>> I've hit some deadlock in kernel xenstore client exposed via
>> /proc/xen/xenbus.
>
> Sounds similar to what Iurii also reported last night in "Userspace PV
> backend hangs".
>
> Iurri's case was all 3.14 kernels, which is in your range too.
>
>>Â Steps to reproduce are simple:
>> int main() {
>>Â Â Â Âstruct xs_handle *xs;
>>Â Â Â Âxs = xs_open(0);
>>Â Â Â Âxs_watch(xs, "domid", "token");
>>Â Â Â Âxs_read(xs, 0, "name", NULL);
>>Â Â Â Âreturn 0;
>> }
>>
>> xs_watch internally creates new thread, which uses read to wait for the
>> watch. And in the same time, the program tries to read some value,
>> but actually it hangs at sending the command (before even sending a path to be
>> read). Strace gives this (simplified for readability):
>> [pid 2494] write(3, "\4\0\0\0\0\0\0\0\0\0\0\0\f\0\0\0", 160 = 16
>> [pid 2494] write(3, "domid\0", 6)   = 6
>> [pid 2494] write(3, "token\0", 6)   = 6
>> [pid 2495] read(3, <unfinished ...>
>> [pid 2494] futex(0x71c0d4, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
>> [pid 2495] <... read resumed>
>> "\17\0\0\0\377\377\377\377\220~\255\27\f\0\0\0", 16) = 16
>> [pid 2495] read(3, "domid\0token\0", 12) = 12
>> [pid 2495] read(3, "\4\0\0\0\0\0\0\0\0\0\0\0\3\0\0\0", 16) = 16
>> [pid 2495] read(3, "OK\0", 3)     = 3
>> [pid 2495] futex(0x71c0d4, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x71c0d0,
>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1} <unfinished ...>
>> [pid 2494] <... futex resumed> )   Â= 0
>> [pid 2495] <... futex resumed> )   Â= 1
>> [pid 2494] futex(0x71c0a8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
>> [pid 2495] futex(0x71c0a8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
>> [pid 2494] <... futex resumed> )   Â= -1 EAGAIN (Resource
>> temporarily unavailable)
>> [pid 2495] <... futex resumed> )   Â= 0
>> [pid 2494] futex(0x71c0a8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
>> [pid 2495] read(3, <unfinished ...>
>> [pid 2494] <... futex resumed> )   Â= 0
>> [pid 2494] rt_sigaction(SIGPIPE, {SIG_DFL, [], SA_RESTORER,
>> 0x7fc78c1488f0}, NULL, 8) = 0
>> [pid 2494] rt_sigaction(SIGPIPE, {SIG_IGN, [], SA_RESTORER,
>> 0x7fc78c1488f0}, {SIG_DFL, [], SA_RESTORER, 0x7fc78c1488f0}, 8) = 0
>> [pid 2494] write(3, "\2\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0", 16
>>
>> And thats all - 2494 is waiting on write, 2495 is waiting on read.
>>
>> On 3.12.x it is working. On 3.17.0 and 3.18.7 it is broken. I haven't
>> checked versions in the middle.
>>
>> Any ideas?
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@xxxxxxxxxxxxx
>> http://lists.xen.org/xen-devel
>
>

_______________________________________________
Embedded-pv-devel mailing list
Embedded-pv-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/embedded-pv-devel

--

Vitaly Chernooky |ÂSenior Developer - Product Engineering and Development

GlobalLogic

P +380.44.4929695 ext.1136 M +380.63.6011802ÂS cvv_2k

www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

_______________________________________________
Embedded-pv-devel mailing list
Embedded-pv-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/embedded-pv-devel