Re: [Xen-devel] Scheduler guts question

[Andrew Biggadike <biggadike@xxxxxxx>]

Magenheimer, Dan (HP Labs Fort Collins) <dan.magenheimer@xxxxxx> wrote:
> Looking at the code, I am missing some key connection
> between the interrupt routines and the scheduler routines.
> Perhaps there is some hidden call in a macro or in
> a function pointer indirection that I am missing.
>
> Could someone walk me through at a low level the sequence
> of Xen/x86 calls that occur when a timer interrupt results in
> some timer expiration that results in the currently running
> domain getting "paused" and another domain launched?

There are certainly better people to address this, since I just started
looking at this code last week, but here's what I've figured out so far.

start_of_day() calls scheduler_init(), in which an ac_timer named
s_timer is initialized for each CPU's schedule_data.  Each time one of
these goes off the s_timer_fn() function is invoked which raises the
SCHEDULER_SOFTIRQ.  Later, setup() calls schedulers_start(), which
invokes s_timer_fn() for the first time to raise the first scheduler
softirq.

__enter_scheduler(), which is attached to the scheduler softirq by the
call to open_softirq() in scheduler_init(), removes the ac_timer when
starting, then adds a new one based on the time slice provided by the
scheduler's do_schedule() routine.  See the "reprogram the timer"
comment in __enter_scheduler().  When that timer goes off the scheduler
softirq will be raised by s_timer_fn() again, causing
__enter_scheduler() to be invoked again.

The softirqs are handled in entry.S.  ret_from_intr checks whether there
are events pending and, if so, jumps to test_all_events.  (It appears
that several places jump to test_all_events and it naturally follows a
hypercall invocation.)  test_all_events checks for pending softirqs, and
jumps to process_softirqs to handle them.  This calls do_softirq() (in
softirq.c), which indirectly invokes __enter_scheduler() through the
softirq_handlers method table.

The ac_timer functions (s_timer_fn for the scheduler timer) are invoked
by ac_timer_softirq_action() which is invoked (also by do_softirq())
when AC_TIMER_SOFTIRQ is raised.  This ac_timer softirq is raised by
smp_apic_timer_interrupt() (and timer_interrupt() if there is no apic).
Note that smp_apic_timer_interrupt() is invoked by the
apic_timer_interrupt() function created by the
BUILD_SMP_TIMER_INTERRUPT() macro in irq.h (the use of the macro is in
i8259.c) and registered in the timer's interrupt gate in init_IRQ()
(which is also called by start_of_day()).


So, a timer interrupt gives:
o timer interrupt ->
   apic_timer_interrupt() ->
    smp_apic_timer_interrupt() -> raises AC_TIMER_SOFTIRQ

Then, next time leaving Xen (through entry.S):
o test_all_events: ->
   process_softirqs: ->
    do_softirq() ->
     softirq_handlers[AC_TIMER_SOFTIRQ] == ac_timer_softirq_action() ->
      s_timer_fn() -> raises SCHEDULER_SOFTIRQ 

Then, since AC_TIMER_SOFTIRQ == SCHEDULER_SOFTIRQ - 1, the next
iteration of do_softirq()'s loop invokes the scheduler:
o do_softirq() ->
   softirq_handlers[SCHEDULER_SOFTIRQ] == __enter_scheduler()


Note that I'm looking at 2.0.4.  Also, there's a good chance that some
of this is not quite right or complete, in which case I'd appreciate
being corrected by someone who has a better understanding of the code
base.

Andrew


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.sourceforge.net/lists/listinfo/xen-devel