Re: [Xen-devel] LTTng Xen port : finally in a repository near you

[Mathieu Desnoyers <compudj@xxxxxxxxxxxxxxxxxx>]

* INAKOSHI Hiroya (inakoshi.hiroya@xxxxxxxxxxxxxx) wrote:
> Mathieu Desnoyers wrote:
> > * INAKOSHI Hiroya (inakoshi.hiroya@xxxxxxxxxxxxxx) wrote:
> >> Hi Mathieu,
> >>
> >> thanks for your reply.  I can understand your opinion very well but a
> >> concern is that cpu ids on a guest OS are different from those on Xen
> >> because they are virtualized.  The number of vcpus in a guest OS is also
> >> different from that of pcpus as you mentioned.  I wondered if the two
> >> traces could be merged directly.  If you translate vcpu ids to pcpu ids
> >> in writing records in the trace buffer in Xen, this concern is solved in
> >> a natural way.
> >>
> > 
> > When you are executing code in dom0 or domUs, how do you plan to get the
> > physical CPU number on which the tracing is done ?
> 
> I am considering the way where dom0 or domUs call hypercalls to write
> records in the Xen's trace buffer.  In this setting, the vcpu info is
> located in the xen kernel stack and the pcpu is the one performing the
> hypercall.  So, I can resolve the mapping between vcpu id and pcpu id.
> 

The performance hit that goes with going through an hypercall for each
traced event would be too high. Typically, changing ring level involves
executing an interrupt routine, which takes a few thousands nanoseconds.
My tracing probes runs within the traced ring in about 270ns (as tested
on a Pentium 4 3GHz).

Mathieu

> Regards,
> Hiroya
> 
> > 
> >> Mathieu Desnoyers wrote:
> >>> * INAKOSHI Hiroya (inakoshi.hiroya@xxxxxxxxxxxxxx) wrote:
> >>>> Hi Mathieu,
> >>>>
> >>>> I am interested in LTTng-xen because I thought that it would be nice if
> >>>> I can get traces on both xen and guest linux at the same time.  I
> >>>> reviewed LTTng-xen and found that
> >>>>
> >>>> * LTTng and LTTng-xen have a quite similar structure,
> >>>> * a trace buffer resides in a hypervisor for LTTng-xen,
> >>>> * it is currently impossible to get traces from guest linux because
> >>>> there is no LTTng for 2.6.18-xen kernel, as you mentioned.
> >>>>
> >>>> I had coarsely ported LTTng to 2.6.18-xen, though it is only for
> >>>> i386.  Now I can get traces on xen and guest linux simultaneously, even
> >>>> though they put records in different trace buffers.
> >>> Hi Ikanoski,
> >>>
> >>> We did the same kind of coarse 2.6.18 port at our lab internally to get
> >>> traces from both Linux and Xen. The fact that the traces are recorded in
> >>> different buffers does not change anything to the fact that those trace
> >>> files can be copied in the same trace directory so they can be parsed
> >>> together by LTTV (traces coming from dom0, domUs and hypervisor). They
> >>> are synchronized by using the TSCs (hopefully, you will configure your
> >>> system to get a reliable TSC on AMD and older intels, see the
> >>> ltt-test-tsc kernel module in recent LTTng versions and ltt.polymtl.ca
> >>> website for info on that matter).
> >>>
> >>>
> >>>> Then I thought that
> >>>> it would be more useful if they put records in xen's trace buffer and I
> >>>> can analyze events
> >>> LTTV merges the information from all the valid trace files that appears
> >>> within the trace directory, so the analysis can be done on data coming
> >>> from userspace, kernels and hypervisor.
> >>>
> >>>> from xen and linux guests with a single lttd and
> >>>> lttctl running on Domain-0.  Do you have an opinion about that?
> >>>>
> >>> lttctl-xen and lttd-xen, although being quite similar to lttd and
> >>> lttctl, use hypercalls to get the data. The standard lttctl/lttd uses
> >>> debugfs files as a hook to the trace buffers.
> >>>
> >>> As a distribution matter, I prefer to leave both separate for now,
> >>> because lttctl-xen and lttd-xen is highly tied to the Xen tree.
> >>>
> >>> Also, merging the information within the buffers between Xen and Dom0 is
> >>> not such a great idea: The Hypervisor and dom0 can have a different
> >>> number of CPUs (Xen : real CPUs, dom0: vcpus). Since I use per-cpu
> >>> buffers, it does not fit.
> >>>
> >>> Also, I don't want dom0 to overwrite data from the Xen buffers easily:
> >>> it's better if we keep some protection between dom0 and the Hypervisor.
> >>>
> >>> Thanks for looking into this, don't hesitate to ask further questions,
> >>>
> >>> Mathieu
> >>>
> >>>> Regards,
> >>>> Hiroya
> >>>>
> >>>>
> >>>> Mathieu Desnoyers wrote:
> >>>>> Hello,
> >>>>>
> >>>>> I made a working version of the LTTng tracer for xen-unstable for x86.
> >>>>> Here is the pointer to my repository so you can try it out :
> >>>>>
> >>>>> hg clone http://ltt.polymtl.ca/cgi-bin/hgweb.cgi xen-unstable-lttng.hg
> >>>>>
> >>>>> Basic usage :
> >>>>>
> >>>>> (see lttctl-xen -h)
> >>>>>
> >>>>> lttctl-xen -c
> >>>>>
> >>>>> (in a different console)
> >>>>> lttd-xen -t /tmp/xentrace1
> >>>>>
> >>>>> (in the 1st console)
> >>>>> lttctl-xen -s
> >>>>>
> >>>>> (tracing is active)
> >>>>>
> >>>>> lttctl-xen -q
> >>>>> lttctl-xen -r
> >>>>>
> >>>>> lttd-xen should automatically quit after writing the last buffers as
> >>>>> soon as lttctl-xen -r is issued.
> >>>>>
> >>>>> Then, you must copy the XML facilities :
> >>>>>
> >>>>> (see the http://ltt.polymtl.ca > QUICKSTART to see how to install the
> >>>>> ltt-control package which contains the XML facilities in your system)
> >>>>>
> >>>>> lttctl-xen -e -t /tmp/xentrace1
> >>>>>
> >>>>> View in the visualiser : (see the QUICKSTART to see how to install it)
> >>>>>
> >>>>> lttv -m textDump -t /tmp/xentrace1
> >>>>>
> >>>>> (not tested yet) : to visualize a dom0 trace with the xen hypervisor
> >>>>> information, one would have to collect the dom0 kernel trace and the Xen
> >>>>> trace and open them together with :
> >>>>> lttv -m textDump -t /tmp/xentrace1 -t /tmp/dom0trace
> >>>>>
> >>>>> The current Linux kernel instrumentation is for 2.6.20. A backport might
> >>>>> be needed to 2.6.18 if there is no proper Xen support in 2.6.20 (I have
> >>>>> not followed the recent developments).
> >>>>>
> >>>>>
> >>>>> Currently broken/missing :
> >>>>>
> >>>>> - Ressources are not freed when the trace channels are destroyed. So you
> >>>>>   basically have to reboot between taking different traces.
> >>>>> - My code in the hypervisor complains to the console that subbuffers
> >>>>>   have not been fully read when the trace channels are destroyed. The
> >>>>>   error printing is just done too fast : lttd-xen is still there and
> >>>>>   reading the buffers at that point. It will get fixed with proper
> >>>>>   ressource usage tracking of both Xen and lttd-xen (same as the first
> >>>>>   point above).
> >>>>> - x86_64 not tested, powerpc local.h and ltt.h missing (should be ripped
> >>>>>   from my Linux kernel LTTng).
> >>>>>
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> Mathieu
> >>>>>
> >>>>>
> >>>>>
> >>>>> * Mathieu Desnoyers (compudj@xxxxxxxxxxxxxxxxxx) wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> My name is Mathieu Desnoyers, I am the current maintainer of the Linux 
> >>>>>> Trace
> >>>>>> Toolkit project, known as LTTng. This is a tracer for the 2.6 Linux 
> >>>>>> kernels
> >>>>>> oriented towards high performance and real-time applications.
> >>>>>>
> >>>>>> I have read your tracing thread and I am surprised to see how much 
> >>>>>> things
> >>>>>> you would like in a tracer are already implemented and tested in 
> >>>>>> LTTng. I am
> >>>>>> currently porting my tracer to Xen, so I think it might be useful for 
> >>>>>> you to
> >>>>>> know what it provides. My goal is to do not duplicate the effort and 
> >>>>>> save
> >>>>>> everyone some time.
> >>>>>>
> >>>>>> Here follows some key features of LTTng :
> >>>>>>
> >>>>>> Architecture independant data types
> >>>>>> Extensible event records
> >>>>>> Self-describing traces
> >>>>>> Variable size records
> >>>>>> Fast (200 ns per event record)
> >>>>>> Highly reentrant
> >>>>>> Does not disable interrupts
> >>>>>> Does not take lock on the critical path
> >>>>>> Supports NMI tracing
> >>>>>> Analysis/visualization tool (LTTV)
> >>>>>>
> >>>>>> Looking at the integration of the existing LTTng implementation into 
> >>>>>> Xen, I
> >>>>>> came up with those two points for my Christmas whichlist :
> >>>>>>
> >>>>>> Additionnal functionnalities that would be nice to have in Xen :
> >>>>>>
> >>>>>> - RCU-style updates : would allow freeing the buffers without impact 
> >>>>>> on tracing.
> >>>>>>     * I guess I could currently use :
> >>>>>>       for_each_domain( d )
> >>>>>>         for_each_vcpu( d, v )
> >>>>>>           vcpu_sleep_sync(v);
> >>>>>>       I think it will have a huge impact on the system, but it would 
> >>>>>> only be
> >>>>>>       performed before trace buffers free.
> >>>>>>
> >>>>>> - Polling for data in Xen from a dom0 process.
> >>>>>>   Xentrace currently polls the hypervisor each 100ms to see if there 
> >>>>>> is data
> >>>>>>   that needs to be consumed. Instead of an active polling, it would be 
> >>>>>> nice to
> >>>>>>   use the dom0 OS capability to put a process to sleep while waiting 
> >>>>>> for a
> >>>>>>   resource. It would imply creating a module, loaded in dom0, that 
> >>>>>> would wait
> >>>>>>   for a specific virq coming from the Hypervisor to wake up such 
> >>>>>> processes.
> >>>>>>   We could think of exporting a complete poll() interface through 
> >>>>>> sysfs or
> >>>>>>   procfs that would be a directory filled with the resources exported 
> >>>>>> from the
> >>>>>>   Hypervisor to dom0 (which could include wait for resource freed, 
> >>>>>> useful when
> >>>>>>   shutting down a domU instead of busy looping). It would help dom0 to 
> >>>>>> schedule
> >>>>>>   other processes while a process is waiting for the Hypervisor.
> >>>>>>
> >>>>>>
> >>>>>> You might also be interested in looking at :
> >>>>>> - the website (http://ltt.polymtl.ca)
> >>>>>> - LTTng Xen port design document (this one is different from the one 
> >>>>>> posted by
> >>>>>>   Jimi)
> >>>>>>   
> >>>>>> (http://ltt.polymtl.ca/svn/ltt/branches/poly/doc/developer/lttng-xen.txt)
> >>>>>> - OLS 2006 paper "The LTTng tracer : A Low Impact Performance and 
> >>>>>> Behavior
> >>>>>>   Monitor for GNU/Linux"
> >>>>>>   (http://ltt.polymtl.ca/papers/desnoyers-ols2006.pdf)
> >>>>>>
> >>>>>>
> >>>>>> Questions and constructive comments are welcome.
> >>>>>>
> >>>>>> Mathieu
> >>>>>>
> >>>>>>
> >>>>>> OpenPGP public key:              
> >>>>>> http://krystal.dyndns.org:8080/key/compudj.gpg
> >>>>>> Key fingerprint:     8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 
> >>>>>> 9A68 
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> Xen-devel mailing list
> >>>>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
> >>>>>> http://lists.xensource.com/xen-devel
> >>>>>>
> >>>>
> >>
> > 
> 
> 

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel