Re: [MirageOS-devel] Logging

[David Sheets <sheets@xxxxxxxxxxxx>]

On Wed, May 13, 2015 at 12:44 AM, Anil Madhavapeddy <anil@xxxxxxxxxx> wrote:
> On 12 May 2015, at 17:58, Rudi Grinberg <rudi.grinberg@xxxxxxxxx> wrote:
>
>
>
>> I like your "only one function" idea.
>> The alternative of passing around the unique function is to have a global
>> reference containing the dispatcher function. It's the solution used by
>> Lwt_log. It's mostly equivalent (and it works ok in practice) but I think
>> it's indeed better to pass the function explicitly.
>
> It should be possible to stash the logging function somewhere more
> convenient than every single function, but this depends on the library. In
> Cohttp and Conduit, the `ctx` context is a good place. For DNS, it's
> probably enough to pass it in to the library functions since they are fairly
> stateless.
>
> Iâm curious about this comment about cohttp. There was actually a request on
> the bug tracker asking for better logging in cohttp but I mostly turned it
> down because it seems like any kind of logging you might want cohttp to do
> can be easily accomplished outside of cohttp. For example, if youâre using a
> cohttp as an http client, you only need to wrap your call function to print
> the request before itâs sent and the response just after it arrives. Adding
> this kind of logging to a cohttp server is similar.
>
> In fact, for the client use case we could really make it trivial for a user
> to add his own logging if we exposed a function that would construct all the
> little helpers such as get/post/put/delete/etc. from a basic log aware call
> function.
>
> Of course, this approach is inadequate if you want to log something that
> pertains cohttpâs internals. Which is exactly why we have the COHTTP_DEBUG
> env var today. I was never a huge fan of his approach either and have
> thought about improving it by introducing a functor (if you can call that an
> improvement) IO -> IO that would take a normal IO backend and return a
> logging one (or at least a COHTTP_DEBUG accepting one). But Iâm not yet sure
> if this an improvement.
>
> Anyhow, this is getting off topic. I just mean to bring some for less
> intrusive approaches for logging.
>
> It's definitely on topic -- the ad-hoc use of COHTTP_DEBUG and CONDUIT_DEBUG
> is what I'm hoping to get rid of in our libraries.  It's also essential to
> have access to similar functionality when compiled to a Xen unikernel as
> well as a Unix binary, and this is just a matter of establishing the right
> conventions.
>
> For Cohttp, I like the idea of initialising a given `ctx` with a log level,
> and the library uses that to decide whether to dump its internals or not
> (similarly to `curl -vvv`).  By default, it would just log the URLs
> themselves.  A functor transformer is also possible, but it seems likely
> that we'll want to be able to turn the detailed debugging on dynamically
> rather than statically.
>
> There's also the sequencing of log messages.  Although logging a single call
> is easy, Cohttp does also support pipelining, and so showing a sequence such
> as:
>   - connected to 1.2.3.4:80
>     - sending /foo
>     - sending /bar
>     - received response for /foo: 404
>     - sending /bar2
>     - received chunked response for /bar
>     - sending /bar3
>   - connected to 4.3.2.1:80
>      ...
>
> would be useful.  This also naturally composes with other libraries such as
> DNS gaining access to the same log stream.

It seems to me that, on the backend, you want a stream of timestamped
events hopefully with a fixed serialization but defaulting to a string
in an envelope. Each event can also, optionally, carry a set of tags.

Later, when consuming the logs for analysis, processing, debugging,
etc, individual streams can be filtered, tags can be treated as
hierarchical containers or resource trackers, distributed streams can
be synchronized on message send/receive events, clocks can be
compared, etc.

Because this post-processing will, in most cases, be necessary, we
should plan for it while keeping the logging user's burden as low as
possible.

Based on these ideas, I propose we create an interface which
explicitly exposes event sinks which encapsulate tag sets. After brief
component- (or resource- or stage-)wide configuration of the tag set,
a logging user simply needs to refer to the appropriate sink or a
function into which it has been closed. Historical reconstructors can
attribute meaning to tags like "cohttp/ip1/1.2.3.4:80". Tags can be
made private if desired through cunning use of tag prefixes. The root
sink can be provided as a parameter and could include "early" filters
which cause certain derivative sinks to become entirely disabled.

Tag set containment provides one type of hierarchical relation but in
many cases there are cross-cutting hierarchies or flows which should
be easily isolable.

This design would require some complexity in the log-reading system
and could require coordination of conventions by logging users.
However, I believe it is relatively simple and flexible for the user
but sufficiently general to grow with the needs of the community.

I don't have a great idea about the structured log events except maybe
to use S-expressions and provide a just-log-a-string-dammit function
as well as a log-this-sexp value function. Implicits would be nice
here.

David

> -anil
>
> _______________________________________________
> MirageOS-devel mailing list
> MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
> http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel
>

_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel