Re: Mirage Xenstore Stub Domain

[Anil Madhavapeddy <anil@xxxxxxxxxx>]

The patch makes sense; it would normally point to a suspicious deadlock since the domain should eventually stop spinning and unblock, but in this case the console ring isn't being drained (whereas it normally would be).

I'll apply this for now, thanks James!  The dreadful "always activate all devices" hack is still in the module for now, which I'll remove just as soon as we have a test case for 8 or so devices spinning and waking up.  There's a small race somewhere in the wakeup that's causing an occasional hang, but it should be easy to track down with a testcase.

-anil

On 16 Jul 2013, at 17:29, James Bielman <jamesjb@xxxxxxxxx> wrote:

Yeah, I think we get stuck because "Activations.run" traverses the callback sequence left-to-right, then when the thread is awakened, it re-adds itself on the right, which means the iteration never terminates when the blocked thread can't make progress.

This patch fixes the issue for me by adding waiters on the front of the callback list, but I'm not sure it's the best solution.

Thanks,

James

On Tue, Jul 16, 2013 at 9:04 AM, James Bielman <jamesjb@xxxxxxxxx> wrote:

From what I can tell, it's getting stuck in "Activations.run" once the ring fills up.  Is it possible that it could spin in "Lwt_sequence.iter_node_l", repeatedly sleeping and waking itself up, and never finishing?

James

On Tue, Jul 16, 2013 at 12:04 AM, Dave Scott <Dave.Scott@xxxxxxxxxxxxx> wrote:

Hi,

Last time I looked at xen-unstable there was a proposal to make the xenstore domain builder (is that init-xenstore-domain?) itself daemonise and drain the console ring, since xenconsoled can't work in the normal way. I've not followed the discussion closely since, maybe they came up with another solution? It's worth a quick mailing list archive search.

So I agree an emergency console ring is useful (actually I was going to write logs to it for now) and it should be possible...

Btw why is write_all causing us to spin on the CPU? Surely we would just block if the console ring wasn't being drained? Suspicious...

--
Dave Scott

On Jul 15, 2013, at 11:40 PM, "Anil Madhavapeddy" <anil@xxxxxxxxxx> wrote:

> It would certainly be nicer to arrange for the Xenstore stub domain to have access to an emergency console ring in this case.  Is there a Xenconsoled stub domain too, Dave, or would this have to be the hypervisor console ring?
>
> -anil
>
> On 15 Jul 2013, at 22:59, James Bielman <jamesjb@xxxxxxxxx> wrote:
>
>> I've hit another snag with the Xenstore stub domain.  I've built "ocaml-xenstore-xen" with Mirari, and I'm launching it early in the dom0 boot process with "init-xenstore-domain".  The xenstore kernel seems to hang with 100% CPU usage trying to write to the console (which makes some sense to me, since nothing is reading from the ring buffer yet).  My guess is that the loop in "OS.Console.write_all" is the culprit---if I stub this function out to return zero, it starts up and works just fine.
>>
>> I've been trying to figure out if this is a bootstrapping issue that's somewhat unavoidable or a bug, but I don't yet have enough of a handle on how LWT works to be sure.
>>
>> James
>>
>>
>>
>> On Mon, Jul 8, 2013 at 4:25 PM, James Bielman <jamesjb@xxxxxxxxx> wrote:
>> Thanks! It builds for me now after following your process and hacking up a quick and dirty mirari config file.
>>
>> Cheers,
>> James
>>
>>
>> On Mon, Jul 8, 2013 at 3:10 PM, David Scott <scott.dj@xxxxxxxxx> wrote:
>> Hi James,
>>
>> I think you're right -- some key interfaces have changed. Since some of the changes are fairly fresh in my memory, I did a first pass at fixing the code.
>>
>> The blog post is a bit out of date now. The best way to start is to:
>>
>> opam init
>> opam switch 4.00.1
>> opam remote add xapi-project git://github.com/xapi-project/opam-repo-dev
>> opam remote add mirage git://github.com/mirage/opam-repo-dev
>> opam install mirage-xen xenstore
>>
>> and then clone the repo and build in the 'xen' directory. For me the files compile but fail to link. The next thing to do is to investigate using the 'mirari' tool to link a xen kernel, like the other examples in the mirage/mirage-skeleton repo. This is fallout from the recent (very good IMHO) change to avoid using a special 'opam compiler switch' to install mirage kernels. You can now use a regular (unpatched) compiler and link in the 'mirage-xen' package.
>>
>> Sorry for the build breakage!
>>
>> Cheers
>> Dave
>>
>>
>> On Mon, Jul 8, 2013 at 10:10 PM, James Bielman <jamesjb@xxxxxxxxx> wrote:
>> Hi all,
>>
>> I am attempting to build the Mirage Xenstore stub domain by following these instructions on the Mirage blog:
>>
>>  http://www.openmirage.org/blog/xenstore-stub-domain
>>
>> However, I'm running into some trouble---it looks like the interfaces for shared memory rings and grant tables have changed since the "xs_transport_domain" module was written.  I'm using Mirage as installed from "opam" and the "ocaml-xenstore-xen" module from the Git  repository at:
>>
>>   git://github.com/djs55/ocaml-xenstore-xen
>>
>> Is there is an updated repository elsewhere that works with the latest Mirage?  If not, I may take a stab at bringing it up to date, and would appreciate any insight available on gotchas I might run into.
>>
>> Thanks,
>> James
>>
>>
>>
>>
>> --
>> Dave Scott
>
>

<0001-Add-activation-event-callbacks-in-reverse-order.patch>