Re: [Xen-devel] Problem Reading from XenStore in DomU

[Dagaen Golomb <dgolomb@xxxxxxxxxxxxxx>]

>>> I'm having an interesting issue. I am working on a project that
>>> requires me to share memory between dom0 and domUs. I have this
>>> successfully working using the grant table and the XenStore to
>>> communicate grefs.
>>>
>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>> My work also requires some kernel modifications, and we have made
>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>> of Xen 4.7 rc1.
>>>
>>> The guest I mentioned before with the default kernel can still read
>>> and write the XenStore just fine when on Xen 4.7 rc1 and with dom0
>>> running our kernel.
>>>
>>> The issue I'm having is with another newly created guest (i.e., it is
>>> not a copy of the working one, this is because I needed more space and
>>> couldn't expand the original disk image). It is also running Ubuntu
>>> 12.04 but has been upgraded to our modified kernel. On this guest I
>>> can write to the XenStore, and see that the writes were indeed
>>> successful using xenstore-ls in dom0. However, when this same guest
>>> attempts to read from the XenStore, it doesn't return an error code
>>> but instead just blocks indefinitely. I've waiting many minutes to
>>> make sure its not just blocking for a while, it appears like it will
>>> block forever. The block is happening when I start the transaction.
>>> I've also tried not using a transaction, in which case it blocks on
>>> the read itself.
>>>
>>> I have an inkling this may be something as simple as a configuration
>>> issue, but I can't seem to find anything. Also, the fact that writes
>>> work fine but reads do not is perplexing me.
>>>
>>> Any help would be appreciated!
>>
>> Nothing should block like this.  Without seeing your patch, I can't
>> comment as to whether you have accidentally broken things.
>
> I don't see any way the patch could be causing this. It simply adds
> another function and case clause to an already-existing hypercall, and
> when you call the hypercall with that option it returns the current
> budget of a passed-in vcpu. It doesn't even come close to touching
> grant mechanics, and doesn't modify any state - it simply returns a
> value that previously was hidden in the kernel.
>
>> Other avenues of investigation are to look at what the xenstored process
>> is doing in dom0 (is it idle? or is it spinning?), and to look in the
>> xenstored log file to see if anything suspicious occurs.
>
> I tried booting into older, stock kernels. They all work with the
> read. However, I do not see why the kernel modification would be the
> issue as described above. I also have the dom0 running this kernel and
> it reads and writes the XenStore just dandy. Are there any kernel
> config issues that could do this? This is very confusing. I have one
> instance of it working with reads so it can be inherent to the kernel
> build itself...
>
> Xenstore (or any grep-ed "xen" process for that matter) are acting
> fine. No changes in memory or CPU when blocking vs before blocking.
> xenstored-access.log is this:
>
> [20160515T19:45:42.207Z]  A992         newconn
> [20160515T19:45:42.210Z]  A992         endconn
> [20160515T19:45:42.223Z]  A993         newconn
> [20160515T19:45:42.223Z]  A993         write     /local/domain/15/mbroe 
> \017\001
> [20160515T19:45:42.223Z]  A993         endconn
> [20160515T19:45:42.232Z]  A994         newconn
> [20160515T19:45:42.232Z]  A994.1       setperms  /local/domain/15/mbroe n0 b15
> [20160515T19:45:42.233Z]  A994.1       setperms
> /local/domain/15/mbroe/gref n0 b15
> [20160515T19:45:42.233Z]  A994.1       setperms
> /local/domain/15/mbroe/min_budget n0 b15
> [20160515T19:45:42.234Z]  A994.1       commit
> [20160515T19:45:42.234Z]  A994         endconn
> [20160515T19:46:25.215Z]  A995         newconn
> [20160515T19:46:25.215Z]  A995         endconn
> [20160515T19:46:29.540Z]  A996         newconn
> [20160515T19:46:29.541Z]  A996         watch
> /local/domain/15/mbroe/max_block max_block
> [20160515T19:46:29.541Z]  A996         w event
> /local/domain/15/mbroe/max_block max_block
> [20160515T19:46:29.542Z]  A996.1       commit
> [20160515T19:46:29.543Z]  A996         watch
> /local/domain/15/mbroe/num_blocks num_blocks
> [20160515T19:46:29.543Z]  A996         w event
> /local/domain/15/mbroe/num_blocks num_blocks
> [20160515T19:46:29.544Z]  A996.2       commit
> [20160515T19:46:29.545Z]  A996.3       write     /local/domain/15/mbroe/gref 8
> [20160515T19:46:29.546Z]  A996.3       commit
> [20160515T19:46:29.546Z]  A996.4       write
> /local/domain/15/mbroe/min_budget 10
> [20160515T19:46:29.547Z]  A996.4       commit
> [20160515T19:46:29.548Z]  A996         endconn
>
> ** Above is dom0, running our kernel, doing both read and write just
> fine, and setting permission for dom15. **
>
> [20160515T19:47:44.562Z]  D15          watch
> /local/domain/15/mbroe/gref FFFF88007B2F3390
> [20160515T19:47:44.562Z]  D15          w event
> /local/domain/15/mbroe/gref FFFF88007B2F3390
>
> ** Above is shown while dom15 is trying to read, but blocking **
>
> [20160515T19:48:39.392Z]  D15          unwatch
> /local/domain/15/mbroe/gref FFFF88007B2F3390
>
> ** This line is after terminating dom15 program while its still waiting.
>
> The only issue I could see is the FFFF88007B2F3390. This is supposed
> to be the key name a presume (such as gref). Maybe its an issue with
> the compiled binary, and it ends up watching on a key that doesn't
> exist (nor ever will). I will look into this as it looks promising!
> Thanks!

Update: I checked further up the logs and the other working kernels
produce values like this as well. Also, I am using the same exact
binary between kernel versions, not recompiling.

>
> I'm don't see any other logs for xenstore, if there are more I please
> point them to me. xenstored.log in same directory is recognized as
> binary fine, and when I open anyways all I see is "Xen Storage Daemon,
> version 1.0" repeatedly.

Regards,
Dagaen Golomb
Ph.D. Student, University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel