Xen project Mailing List

Re: [Xen-devel] [PATCH] Add a timer mode that disables pending missed ticks

Hi Keir, The latest change, c/s 16690, looks fine. I agree that the code in c/s 16690 is equivalent to the code I submitted. Also, your version is more concise. The error tests confirm the equivalence. With overnight cpu loads, the checked in version was accurate to +.048% for sles and +.038% for red hat. My version was +.046% and +.032% in a 2 hour test. I don't think the difference is significant. i/o loads produced errors of +.01%. Thanks for all your efforts on this issue. Regards, Dave Keir Fraser wrote:

Applied as c/s 16690, although the checked-in patch is smaller. I think the
only important fix is to pt_intr_post() and the only bit of the patch I
totally omitted was the change to pt_process_missed_ticks(). I don't think
that change can be important, but let's see what happens to the error
percentage...

-- Keir

On 4/1/08 23:24, "Dave Winchell" <dwinchell@xxxxxxxxxxxxxxx> wrote:

Hi Dan and Keir,

Attached is a patch that fixes some issues with the SYNC policy
(no_missed_ticks_pending).
I have not tried to make the change the minimal one, but, rather, just
ported into
the new code what I know to work well. The error for
no_missed_ticks_pending goes from
over 3% to .03% with this change according to my testing.

Regards,
Dave

Dan Magenheimer wrote:

Hi Dave --

Did you get your correction ported?  If so, it would be nice to see this get
into 3.1.3.

Note that I just did some very limited testing with timer_mode=2(=SYNC=no
missed ticks pending)
on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error I've
seen so far
is 0.012%.  But I haven't tried any exotic loads, just LTP.

Thanks,
Dan

-----Original Message-----
From: Dave Winchell [mailto:dwinchell@xxxxxxxxxxxxxxx]
Sent: Wednesday, December 19, 2007 12:33 PM
To: dan.magenheimer@xxxxxxxxxx
Cc: Keir Fraser; Shan, Haitao; xen-devel@xxxxxxxxxxxxxxxxxxx; Dong,
Eddie; Jiang, Yunhong; Dave Winchell
Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
disables pending
missed ticks

Dan,

I did some testing with the constant tsc offset SYNC method
(now called
no_missed_ticks_pending)
and found the error to be very high, much larger than 1 %, as
I recall.
I have not had a chance to submit a correction. I will try to
do it later
this week or the first week in January. My version of constant tsc
offset SYNC method
produces .02 % error, so I just need to port that into the
current code.

The error you got for both of those kernels is what I would expect
for the default mode, delay_for_missed_ticks.

I'll let Keir answer on how to set the time mode.

Regards,
Dave

Dan Magenheimer wrote:

Anyone make measurements on the final patch?

I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of

about 0.2% with no load.  This was xen-unstable tip today
with no options specified.  32-bit was about 0.01%.

I think I missed something... how do I run the various

accounting choices and which ones are known to be appropriate
for which kernels?

Thanks,
Dan

-----Original Message-----
From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
[mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx]On Behalf Of

Keir Fraser

Sent: Thursday, December 06, 2007 4:57 AM
To: Dave Winchell
Cc: Shan, Haitao; xen-devel@xxxxxxxxxxxxxxxxxxx; Dong, Eddie; Jiang,
Yunhong
Subject: Re: [Xen-devel] [PATCH] Add a timer mode that
disables pending
missed ticks


Please take a look at xen-unstable changeset 16545.

-- Keir

On 26/11/07 20:57, "Dave Winchell"

<dwinchell@xxxxxxxxxxxxxxx> wrote:

Keir,

The accuracy data I've collected for i/o loads for the
various time protocols follows. In addition, the data
for cpu loads is shown.

The loads labeled cpu and i/o-8 are on an 8 processor AMD box.
Two guests, red hat and sles 64 bit, 8 vcpu each.
The cpu load is usex -e36 on each guest.
(usex is available at http://people.redhat.com/anderson/usex.)
i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null.

The loads labeled i/o-32 are 32 instances of dd.
Also, these are run on 4 cpu AMD box.
In addition, there is an idle rh-32bit guest.
All three guests are 8vcpu.

The loads labeled i/o-4/32 are the same as i/o-32
except that the redhat-64 guest has 4 instances of dd.

Date Duration Protocol sles, rhat error load

11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu
11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu

11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu
11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu

11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu


11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8
11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8

11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8
11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8

11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8
11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8



11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32
11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32

11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32
11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32


Overhead measurements:

Progress in terms of number of passes through a fixed

system workload

on an 8 vcpu red hat with an 8 vcpu sles idle.
The workload was usex -b48.


ASYNC 167 min 145 passes .868 passes/min
SYNC 167 min 144 passes .862 passes/min
SYNC 1065 min 919 passes .863 passes/min
MIXED 221 min 196 passes .887 passes/min


Conclusions:

The only protocol which meets the .05% accuracy requirement for ntp
tracking under the loads
above is the SYNC protocol. The worst case accuracies for

SYNC, MIXED,

and ASYNC
are .022%, .12%, and .14%, respectively.

We could reduce the cost of the SYNC method by only

scheduling the extra

wakeups if a certain number
of ticks are missed.

Regards,
Dave

Keir Fraser wrote:

On 9/11/07 19:22, "Dave Winchell"

<dwinchell@xxxxxxxxxxxxxxx> wrote:

Since I had a high error (~.03%) for the ASYNC method a

couple of days ago,

I ran another ASYNC test. I think there may have been something
wrong with the code I used a couple of days ago for

ASYNC. It may have been

missing the immediate delivery of interrupt after context

switch in.

My results indicate that either SYNC or ASYNC give

acceptable accuracy,

each running consistently around or under .01%. MIXED has

a fairly high

error of
greater than .03%. Probably too close to .05% ntp

threshold for comfort.

I don't have an overnight run with SYNC. I plan to leave

SYNC running

over the weekend. If you'd rather I can leave MIXED

running instead.

It may be too early to pick the protocol and I can run

more overnight tests

next week.

I'm a bit worried about any unwanted side effects of the

SYNC+run_timer

approach -- e.g., whether timer wakeups will cause higher

system-wide CPU

contention. I find it easier to think through the

implications of ASYNC. I'm

surprised that MIXED loses time, and is less accurate than

ASYNC. Perhaps it

delivers more timer interrupts than the other approaches,

and each interrupt

event causes a small accumulated error?

Overall I would consider MIXED and ASYNC as favourites and

if the latter is

actually more accurate then I can simply revert the changeset that
implemented MIXED.

Perhaps rather than running more of the same workloads you

could try idle

VCPUs and I/O bound VCPUs (e.g., repeated large disc reads

to /dev/null)? We

don't have any data on workloads that aren't CPU bound, so

that's really an

obvious place to put any further effort imo.

-- Keir

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c
--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000
+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500
@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru

    missed_ticks = missed_ticks / (s_time_t) pt->period + 1;
    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) )
-        pt->do_not_freeze = !pt->pending_intr_nr;
+        pt->do_not_freeze = 1;
    else
        pt->pending_intr_nr += missed_ticks;
    pt->scheduled += missed_ticks * pt->period;
@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data)

    pt_lock(pt);

-    pt->pending_intr_nr++;
+    if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) {
+        pt->pending_intr_nr = 1;
+ pt->do_not_freeze = 0;
+    }
+    else
+ pt->pending_intr_nr++;

    if ( !pt->one_shot )
    {
@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct
        return;
    }

-    pt->do_not_freeze = 0;
-
    if ( pt->one_shot )
    {
        pt->enabled = 0;
@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct
            pt->last_plt_gtime = hvm_get_guest_time(v);
            pt->pending_intr_nr = 0; /* 'collapse' all missed ticks */
        }
+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
+     pt->pending_intr_nr--;
+     pt->last_plt_gtime = hvm_get_guest_time(v);
+ }
        else
        {
            pt->last_plt_gtime += pt->period_cycles;

_______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.