[Xen-devel] RE: A credit scheduler issue

["Kamble, Nitin A" <nitin.a.kamble@xxxxxxxxx>]

Keir, Emmanuel,
  Thanks for the detailed answers, and your views. I agree that my small
change should not affect correctness. I didn't see the migration often,
I saw the dom0 vcpus migrations happening 4, 5 times from boot to start
of xend. I think we should avoid these migrations; why waste the cache
hotness?
    How solid is the credit scheduler now for DomUs on a SMP box? On
32bit, PAE & 64bit? It would be a useful data point for me to debug the
HVM guest issues with the credit scheduler.

Thanks & Regards,
Nitin
------------------------------------------------------------------------
-----------
Open Source Technology Center, Intel Corp

>-----Original Message-----
>From: Emmanuel Ackaouy [mailto:ack@xxxxxxxxxxxxx]
>Sent: Friday, June 30, 2006 3:46 AM
>To: Kamble, Nitin A
>Cc: xen-devel@xxxxxxxxxxxxxxxxxxx; Keir Fraser; Ian Pratt
>Subject: Re: A credit scheduler issue
>
>Hi Nitin,
>
>On Thu, Jun 29, 2006 at 06:13:51PM -0700, Kamble, Nitin A wrote:
>>        I am trying to debug the credit scheduler to solve the many
HVM
>domain
>>    instability issues we have found with the credit scheduler.
>
>Great. As Keir pointed out though the problems you are seeing
>may not actually be in the credit scheduler itself.
>
>>        While debugging I notice an odd behavior; When running on a 2
CPU
>>    system, dom0 gets 2 vcpus by default. And even if there are no
other
>>    domains running in the system,  the dom0 vcpus are getting
migrated to
>>    different pcpus in the load balance. I think it is due to the
>preemption
>>    happening in the credit scheduler; and it is not necessary and is
>actually
>>    wasteful to move vcpus when no of vcpus in the system are equal to
no
>of
>>    pcpus.
>>
>>        I would like to know your thinking about this behavior. Is it
an
>>    intended in the design?
>
>This should be very rare. If a VCPU were woken up and put on
>the runq of an idle CPU, a peer physical CPU that is in the
>scheduler code at that exact time could potentially pick up
>the just woken up VCPU.
>
>We can do things to shorten this window, like not pick up a
>VCPU from a remote CPU that is currently idle and therefore
>probably racing with us to run said newly woken up VCPU on
>its runq. But I'm not sure this happens frequently enough to
>warrant the added complexity. On top of that, it seems to
>me this is more likely to happen to VCPUs that aren't doing
>very much work and therefore would not suffer a performance
>loss from migrating physical CPU on occasion.
>
>Are you seeing a lot of these migrations?
>
>>    I added this small fix to the scheduler to fix this behavior. And
with
>it
>>    I see the stability of Xen improved. Win2003 boot was crashing
with
>>    unhandled MMIO error on xen64 earlier with credit scheduler. I am
not
>>    seeing that crash with this small fix anymore. It is quiet
possible
>that
>>    there are more bugs I need to catch for HVM domains in the credit
>>    scheduler. And I would like to know your thoughts for this change.
>
>I don't agree with this change.
>
>When a VCPU is the only member of a CPU's runq, it's still
>waiting for a _running_ VCPU to yield or block. We should
>absolutely be picking up such a VCPU to run elsewhere on
>an idle CPU. Else, you'd end up with two VCPUs time-slicing
>on a processor while other processors in the system are idle.
>
>Your change effectively turns off migration on systems where
>the number of active VCPUs is less than 2 multiplied by the
>number of physical CPUs. I can see why that would hide any
>bugs in the context migrating paths, but that doesn't make
>it right. :-)
>
>>
>>    csched_runq_steal(struct csched_pcpu *spc, int cpu, int pri)
>>
>>    {
>>
>>        struct list_head *iter;
>>
>>        struct csched_vcpu *speer;
>>
>>        struct vcpu *vc;
>>
>>
>>
>>        /* If there are only 1 vcpu in the queue then stealing it from
the
>>    queue
>>
>>         * is not going not help in load balancing.
>>
>>         */
>>
>>        if (spc->runq.next->next == &spc->runq)
>>
>>                return NULL;
>>
>>
>>
>>    Thanks & Regards,
>>
>>    Nitin
>>
>>
----------------------------------------------------------------------
>-------------
>>
>>    Open Source Technology Center, Intel Corp
>>
>>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel