Re: A KernelShark plugin for Xen traces analysis

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 13/04/2021 16:46, Steven Rostedt wrote:
> Hi Giuseppe,
>
> On Tue, 13 Apr 2021 16:28:36 +0200
> Giuseppe Eletto <giuseppe.eletto@xxxxxxxxxxxx> wrote:
>
>> Hello,
>> I want to share with you a new plugin developed by me, under the
>> supervision of Dario Faggioli, which allows the new version of KernelShark
>> (the v2-beta) to open and view the Xen traces created using the "xentrace" 
>> tool.
>>
>> In fact, KernelShark is a well known tool for graphical visualization
>> Linux kernel traces, obtained via "ftrace" and "trace-cmd". Anyway thanks
>> to its modular architecture, it is now possible to implement plugins which
>> open and display traces with arbitrary format, for example, as in in
>> this case, traces of the Xen hypervisor.
> I'm guessing you have trace events coming from Xen itself?
>
>
>> For more information on how to build the plugin and/or
>> to view the source code I leave the repository below:
>> https://github.com/giuseppe998e/kernelshark-xentrace-plugin
>>
>>
>> In short:
>>
>> $ sudo apt install git build-essential libjson-c-dev
>> $ git clone --recurse-submodules
>> https://github.com/giuseppe998e/kernelshark-xentrace-plugin.git
>> $ cd kernelshark-xentrace-plugin/
>> $ make
>>
>> $ export XEN_CPUHZ=3G # Sets the CPU frequency ((G)hz/(M)hz/(K)hz/hz)
>> $ kernelshark -p out/ks-xentrace.so trace.xen
>>
>>
>> You will need the development version of KernelShark, available here:
>> https://git.kernel.org/pub/scm/utils/trace-cmd/kernel-shark.git
> This will soon be the main repository, as we are going to deprecate the
> version in the trace-cmd.git repo soon. And because libtracecmd 1.0 has
> already been released.
>
>
>> A screenshot of the plugin in action is available here:
>> https://github.com/giuseppe998e/kernelshark-xentrace-plugin/raw/master/.github/img/ks-xentrace.png
>>
>> I'm happy to receive whatever feedback you may have about it,
>> and to answer any question.
>>
> Thanks for doing this. What would be nice is to have the xen traces along
> side the linux tracing. Perhaps we can update trace-cmd agent to work with
> Xen as well. Does xen implement vsock or some other way to communicate
> between the guests and the Dom0 kernel? If not, we should add one. The you
> could do the following:
>
>  1. On each guest, run as root: trace-cmd agent --xen
>  2. On Dom0 run: trace-cmd record -e (events on Dom0) \
>      --xen (commands to do tracing in Xen HV) \
>      -A <guest-name1> -e (events on guest)
>
> And then you would get a trace.dat file for Dom0 and the guest, and also
> have a trace file for Xen (however that is done). And then on KernelShark,
> we have a KVM plugin in development that does this. But you can do the same
> with Xen.
>
> For KVM, we have:
>
>  1. On each guest: trace-cmd agent
>  2. On the host: trace-cmd record -e kvm -e sched -e irq \
>       -A guest-name -e all
>     The above produces trace.dat for the host trace, and 
>      trace-<guest-name>.dat for the guest.
>  3. kernelshark trace.dat -a trace-Fedora21.dat
>
> (I have a guest called Fedora21).
>
>   http://rostedt.org/private/kernelshark-kvm.png
>
> Where you can see the kvm hypervisor task KVM-2356 is the host task running
> the guest VCPU 0, and you see the switch between the two.
>
> Perhaps we can do something like that with Xen as well. The plugin is still
> in the works, but should be published soon. And when it is, you could use
> that as a template for Xen.

A possibly tangential question.  Where does KernelShark's idea of CPUs
(i.e. real logical threads) come from?

In a Xen system, dom0 is just a VM, and particularly on larger servers,
may not be as many vcpus as the system has logical threads.

This causes major problems for `perf` support under Xen, which assumes
that the kernel's idea of CPUs matches that of the system.

When rendering a trace including Xen data, Xen can provide the real
system CPUs, and dom0 wants to be rendered as a VM under Xen, similar to
trace-Fedora21 in your screenshot above.  (Obviously, if you're doing
nested virt, things need to start nesting.)

~Andrew