RE: [Xen-devel] 10 million cycles disappearing

["Tian, Kevin" <kevin.tian@xxxxxxxxx>]

In your later test with memory allocation, is it still same case that
only some samples approach 10M, or the average close to 10M?
If average cost is high, then xenoprofile could be useful here to
show hotpot to you.

Thanks,
Kevin

>From: Dan Magenheimer [mailto:dan.magenheimer@xxxxxxxxxx] 
>Sent: 2009年4月9日 8:47
>
>The problem still occurs on latest tip (c/s 19515). :-(
>
>
>> -----Original Message-----
>> From: Tian, Kevin [mailto:kevin.tian@xxxxxxxxx]
>> Sent: Tuesday, April 07, 2009 6:33 PM
>> To: Dan Magenheimer; Xen-Devel (E-mail); jbeulich@xxxxxxxxxx
>> Subject: RE: [Xen-devel] 10 million cycles disappearing
>>
>>
>> I remember that some previous post (from Jan?) solved some heavy
>> operation incurred in some special action of writable page able. Not
>> recall the detail, but your description about subop with memory
>> allocation leads me to that part...
>>
>> Thanks,
>> Kevin
>>
>> >From: Dan Magenheimer
>> >Sent: 2009年4月8日 7:54
>> >
>> >I've been seeing a possible performance problem off and on
>> >and I've spent some time tracking it but haven't made much
>> >progress and have to give up for now, so I thought I'd at
>> >least document what I know and see if it sounds familiar
>> >to anyone.
>> >
>> >The problem: Something in Xen seems to periodically take about
>> >10M cycles.  I think it is an interrupt and I think it is
>> >taking a lock related to memory allocation and holding
>> >it for a LONG time (i.e. 10M cycles or close).
>> >
>> >I am measuring inside a hypercall using TSC, taking a TSC
>> >reading at entry to the hypercall code and at exit.  Xen
>> >is not pre-emptive, so it can't be switching context or
>> >something, right?  Nearly all of the readings are less
>> >than 100K cycles, but some samples are "huge" and
>> >usually at 9M-10M cycles.  Since I am recording the max
>> >difference between the TSCs, the max "huge" grows over
>> >a long period of time, but eventually converges close
>> >to 10M (and this is a 3Ghz processor).  I can see
>> >it grow using "watch".  And I've NEVER seen a reading
>> >over 10M.
>> >
>> >I am able to disable interrupts and still take
>> >measurements.  Roughly half of the measurements
>> >occur when doing a hypercall-subop that does no
>> >memory allocation and roughly half occur when doing
>> >a hypercall-subop that DOES do memory allocation.
>> >With interrupts disabled, the subop that DOES
>> >memory allocation still asymptotically approaches
>> >10M.  The one that does NOT do memory allocation,
>> >stays relatively small.
>> >
>> >I'm currently measuring on Xen 3.3.1 but I think I've
>> >seen similar results on xen-unstable.  A single 2-vcpu
>> >domain is running (in addition to domain0).
>> >
>> >Does any of that sound familiar?  Any smoking guns?
>> >
>> >_______________________________________________
>> >Xen-devel mailing list
>> >Xen-devel@xxxxxxxxxxxxxxxxxxx
>> >http://lists.xensource.com/xen-devel
>> >
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel