Re: A KernelShark plugin for Xen traces analysis

[Dario Faggioli <dfaggioli@xxxxxxxx>]

On Wed, 2021-04-14 at 21:05 +0100, Andrew Cooper wrote:
> On 14/04/2021 14:43, Steven Rostedt wrote:
> > 
> > What I would envision how this would work, is that you would
> > produce a
> > set of tracing files. One for each guest (including Dom0), and one
> > for the
> > Xen hypervisor itself. The trick is to have a way to synchronize
> > the time
> > stamps. What we just did with KVM is to have all the tracing record
> > the
> > CPUs TSC, including the shift and multiplier that the CPU might
> > change for
> > the guests. Then we have a way to convert the TSC to nanoseconds.
> > This way
> > all tracing data has the same clock. It's somewhat complicated to
> > get
> > right, and requires access to how the guests clocks are modified by
> > the CPU.
> 
> Xen certainly has enough information to describe what TSC rate/epoch
> each guest is seeing, but I doubt any of this is coherently exposed
> at
> the moment.
> 
Yeah. In KVM its in, e.g.:

/sys/kernel/debug/kvm/85276-13/vcpu0/tsc-offset

Where 85276-13 is a combination of the PID of the main QEMU process and
one of the KVM file descriptors.

> > The Xen plugin would then need to read the how the threads in
> > xen.dat map
> > to the virtual CPUs of each of the guest files. Which would give
> > you the
> > layering.
> 
> Looks good.  I suspect we might need to do a little work on Xen's
> trace
> data to make this mesh together nicely.  In particular, Xen doesn't
> have
> a terribly good scheme on unique IDs for "a VM".
> 
True. However, Linux recycles PIDs too, doesn't it?

I mean, if PID 1234 is running when I start tracing and it dies and,
after a while, another task with PID 1234 is started KernelShark will
probably treat and display them as if they were the same task? Or
wouldn't it?

Currently, the Xen plugin combines the domain ID and the vCPU ID to
come up with a PID, that then will be used for assigning a color to
each vCPU.

This is pretty much arbitrary and can be changed, but it of course has
to be something that we can extract from from the trace event records
or, at least, from the trace file (e.g., as Andrew said, we do have
per-VM UUIDs, but they're currently not there).

Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

Attachment: signature.asc
Description: This is a digitally signed message part