Re: A KernelShark plugin for Xen traces analysis

[Dario Faggioli <dfaggioli@xxxxxxxx>]

On Tue, 2021-04-13 at 11:46 -0400, Steven Rostedt wrote:
> On Tue, 13 Apr 2021 16:28:36 +0200
> Giuseppe Eletto <giuseppe.eletto@xxxxxxxxxxxx> wrote:
> > 
> > In fact, KernelShark is a well known tool for graphical
> > visualization
> > Linux kernel traces, obtained via "ftrace" and "trace-cmd". Anyway
> > thanks
> > to its modular architecture, it is now possible to implement
> > plugins which
> > open and display traces with arbitrary format, for example, as in
> > in
> > this case, traces of the Xen hypervisor.
> 
> I'm guessing you have trace events coming from Xen itself?
> 
Yes, basically, we can say that a Xen system has "its own trace-cmd".
It's called `xentrace`, you run it from Dom0 and you get a (binary)
file which contains a bunch of events.

Not that differently from a trace-cmd's "trace.dat" file, but the
events in there comes from tracepoints within the hypervisor (which, of
course, use a different tracing mechanism than ftrace).

> 
> > A screenshot of the plugin in action is available here:
> > https://github.com/giuseppe998e/kernelshark-xentrace-plugin/raw/master/.github/img/ks-xentrace.png
> > 
> > I'm happy to receive whatever feedback you may have about it,
> > and to answer any question.
> > 
> 
> Thanks for doing this. What would be nice is to have the xen traces
> along
> side the linux tracing. 
>
Indeed! :-P

> Perhaps we can update trace-cmd agent to work with
> Xen as well. Does xen implement vsock or some other way to
> communicate
> between the guests and the Dom0 kernel? 
>
Not vsock, AFAIK. But we probably can use something else/come up with
something new.

>  1. On each guest, run as root: trace-cmd agent --xen
>  2. On Dom0 run: trace-cmd record -e (events on Dom0) \
>      --xen (commands to do tracing in Xen HV) \
>      -A <guest-name1> -e (events on guest)
> 
> And then you would get a trace.dat file for Dom0 and the guest, and
> also
> have a trace file for Xen (however that is done). 
>
Yep, and the implementation of the `--xen (commands to do tracing in
Xen HV)` part, can just "call-in" to xentrace (somehow), and we'll get
the trace.xen file that then can be interpreted with this plugin.

> And then on KernelShark,
> we have a KVM plugin in development that does this. But you can do
> the same
> with Xen.
> 
I think that one of the trickiest aspects would be synchronizing the
timestamps in the 3 traces.

*I guess* that the dom0 trace and the guest traces could at least use
the PTP algorithm that is currently implemented in the trace-cmd
patches (but not KVM specific one). For synch'ing the Xen trace with
them, well, I don't really know... We'd have to think about it. :-P

> Perhaps we can do something like that with Xen as well. 
>
Would be awesome, IMO. :-)

Thanks and Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

Attachment: signature.asc
Description: This is a digitally signed message part