[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Hi,something about the xentrace tool



George Dunlap wrote:
You misunderstand me. :-)  I meant to write out (via DMA) several
pages at a time, straight from the HV trace buffers.  The default tbuf
size in xentrace is 20 pages, so if (as the plan is) xentrace would be
notified when it would be half full, we could easily write out 10
pages in one transaction.  The tbuf size could be increased if DMA
setup/teardown overhead were an issue on that scale.
...
Some of my recent traces have been on the order of 10 gigabytes.  I
haven't done much to modify xentrace, because I'm not worried about
the trace overhead at this point.  But I've had to pull some tricks to
get my analysis tools to run in anything like a reasonable amount of
time.

I am glad to discover that I misunderstood you. ;) But I am still having trouble understanding what the actual problem is, or even if one exists.

If you have a trace that is 10 gigabytes, that's several days (maybe weeks) worth of trace records, depending on the rate they're generated.

A memory to memory copy of 10 gigabytes will take mere seconds on any modern machine, and amortized over a few days, I don't see how it's worth any work to further reduce that or eliminate it. Is the system so cpu-bound that the loss of a few seconds over several days is that serious? Even compared to the disk I/O to write out 10 gb, which is probably several minutes, I don't see how the memory copies are a big deal. Perhaps kernel buffer cache effects are noticeable, but again at the data rate you're talking about, the cache will only get completely purged once every 5 or 10 hours.

If your analysis tools take a long time to run, I'd guess it's because of the size of the data, not because system resources are being hogged by xentrace; If you are generating that much data, maybe you consider methods to reduce it. Take a look the the trace code (xen/common/trace.c) and you'll see that there is a facility to mask out tracing of certain events, classes of events, and cpu's. You might use this to drastically reduce the number of trace records generated. For instance, if you are not interested in tracing I/O related events, you don't want to be storing TRC_MEM records, which account for a large percentage of the trace records generated on a busy system.


Rob


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.