Re: [MirageOS-devel] Problem with Lwt and TCP/IP

[Magnus Skjegstad <magnus@xxxxx>]

On Mon, 23 Mar 2015, at 16:24, Dave Scott wrote:
> 
> > On 23 Mar 2015, at 15:55, Magnus Skjegstad <magnus@xxxxx> wrote:
> > The workaround seems to run fine under Xen and the iperf-like throughput
> > between two threads with [4] is about 400 000 KBit/s in Xen/Virtualbox.
> > In Unix mode the throughput is about 1 200 000 KBit/s. Unfortunately,
> > the test also sometimes deadlocks in Unix mode.
> 
> Iâd love to know where itâs deadlocking. Perhaps try reproducing it with
> tracing enabled?
> 
> http://openmirage.org/wiki/profiling
> 
> [ I hope it reproduces at a lower throughput! ]

The deadlock seems to occur with tracing enabled as well. I'll try to
figure out where it is.

> > I thought the deadlock could be caused by the faster write threads
> > starving the listeners, so I added an Lwt.pause in iperf_self.ml after
> > each call to write [5]. The test then runs fine under Unix, with a
> > reduced throughput of about 500 000 KBit/s. Unfortunately, the call to
> > Lwt.pause is much slower under Xen and the throughput is around 46
> > Kbit/s (!). 
> > 
> > I am not sure what would be the best way to fix this. If there is a
> > race, I guess it is likely that it is in either mirage-tcpip or in my
> > code. I have not been able to find one in the vnetif code, but maybe
> > there is something I have overlooked that is triggered under high load.
> > The code works fine when I run iperf without TCP/IP and with async
> > disabled [6] (tested with throughput around 17 000 000 KBit/s).
> > 
> > A solution could also be to try to speed up Lwt.pause in Xen, so that we
> > at least could run the same iperf test on both platforms. In the run
> > loop in [7] the domain is blocked when there are no new events on the
> > event channel, even when there are paused threads. By adding a check for
> > Lwt.paused_count () = 0 before blocking the domain (code here [8]), the
> > Xen w/Lwt.pause iperf throughput is 60 000 KBit/s (still slow, but
> > useable). 
> 
> The Lwt.pause thing sounds like itâs probably masking a bug elsewhere
> that we ought to track down.
> 
> I donât really understand how pause is supposed to be used. In fact Iâd
> never noticed it existed until you pointed it out! The docs just say
>
> - pause () is a sleeping thread which is wake up on the next call to
> Lwt.âwakeup_paused.
> - wakeup_paused () wakes up all threads which suspended themselves with
> Lwt.âpause. This function is called by the scheduler, before entering the
> main loop

It is not clear to me either. I hoped I could use pause as an
alternative to yield in Mirage, but that may be the wrong way to use it.
From the code in [1] and [2] it could look like the main difference
between yield and pause is that paused threads are resumed before and
after the main loop, while yielded threads are only resumed after.

> 
> It sounds like thereâs no guarantee that the scheduler is going to be run
> any time soon. With your proposed modification, how similar would a
> paused thread be to a thread sleeping for 0 seconds?
> 

I wrote a test unikernel to compare the performance of pause and sleep
here [3]. The results are from a cubieboard. In the test Lwt.pause is
faster than OS.Time.sleep 0.0 on Unix (211k calls per second vs 71k),
but not in Xen without the patch (3 calls per second vs 110k). With the
patch it is around 190k vs 129k in Xen.

The iperf throughput with Lwt.pause+patch seems to be about the same as
the throughput with OS.Time.sleep though. On my Cubieboard in Xen it
varies around 6-8000 kbit/s, compared to around 26000 kbit/s without
pause/sleep.

1. https://github.com/ocsigen/lwt/blob/master/src/unix/lwt_main.ml
2. https://github.com/ocsigen/lwt/blob/master/src/core/lwt.ml#L1190
3. https://gist.github.com/MagnusS/089c06913594ebdf21b1#file-results-md

Magnus

_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel