Re: [Xen-devel] Hypervisor crash(!) on xl cpupool-numa-split

[Juergen Gross <juergen.gross@xxxxxxxxxxxxxx>]

On 02/16/11 14:54, George Dunlap wrote:
> Andre (and Juergen), can you try again with the attached patch?
>
> What the patch basically does is try to make "cpu_disable_scheduler()"
> do what it seems to say it does. :-)  Namely, the various
> scheduler-related interrutps (both per-cpu ticks and the master tick)
> is a part of the scheduler, so disable them before doing anything, and
> don't enable them until the cpu is really ready to go again.
>
> To be precise:
> * cpu_disable_scheduler() disables ticks
> * scheduler_cpu_switch() only enables ticks if adding a cpu to a pool,
> and does it after inserting the idle vcpu
> * Modify semantics, s.t., {alloc,free}_pdata() don't actually start or
> stop tickers
>   + Call tick_{resume,suspend} in cpu_{up,down}, respectively
> * Modify credit1's tick_{suspend,resume} to handle the master ticker as well.
>
> With this patch (if dom0 doesn't get wedged due to all 8 vcpus being
> on one pcpu), I can perform thousands of operations successfully.
>
> (NB this is not ready for application yet, I just wanted to check to
> see if it fixes Andre's problem)
After some thousand iterations the machine hang and after dumping Dom0
registers to console it continued running and crashed about a second later:

(XEN) cpupool_unassign_cpu(pool=0,cpu=9)
(XEN) cpupool_unassign_cpu(pool=0,cpu=9) ffff83083fff74c0
(XEN) cpupool_unassign_cpu ret=0
(XEN) cpupool_unassign_cpu(pool=0,cpu=4)
(XEN) cpupool_unassign_cpu(pool=0,cpu=4) ffff83083fff74c0
(XEN) cpupool_unassign_cpu ret=0
(XEN) cpupool_assign_cpu(pool=1,cpu=9)
(XEN) cpupool_assign_cpu(pool=1,cpu=9) ffff83083002de40
(XEN) Assertion 'timer->status >= TIMER_STATUS_inactive' failed at timer.c:279
(XEN) ----[ Xen-4.1.0-rc5-pre  x86_64  debug=y  Tainted:    C ]----
(XEN) CPU:    9
(XEN) RIP:    e008:[<ffff82c480126100>] active_timer+0xc/0x37
(XEN) RFLAGS: 0000000000010046   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) rdx: ffff830839d8ff18   rsi: 0000010dbb628a80   rdi: ffff83083ffbcf98
(XEN) rbp: ffff830839d8fd50   rsp: ffff830839d8fd50   r8:  ffff83083ffbcf90
(XEN) r9:  ffff82c480213680   r10: 00000000ffffffff   r11: 0000000000000010
(XEN) r12: ffff82c4802d3f80   r13: ffff82c4802d3f80   r14: ffff83083ffbcf98
(XEN) r15: ffff83083ffbcfc0   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 000000007809c000   cr2: 0000000000620048
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff830839d8fd50:
(XEN)    ffff830839d8fda0 ffff82c480126ef9 0000000000000000 0000010dbb628a80
(XEN)    0000000000000086 0000000000000009 ffff83083002de40 ffff83083002dd50
(XEN)    0000000000000009 0000000000000009 ffff830839d8fdc0 ffff82c480117906
(XEN)    ffff83083ffa3b40 ffff83083ffa5d70 ffff830839d8fe30 ffff82c4801214fa
(XEN)    ffff83083002dd00 0000000900000100 0000000000000286 ffff8300780da000
(XEN)    ffff83083ffbcf80 ffff83083ffbcf90 ffff82c480247e00 0000000000000009
(XEN)    00000000fffffff0 ffff83083002dd00 0000000000000000 ffff8300781cc198
(XEN)    ffff830839d8fe60 ffff82c4801019ff 0000000000000009 0000000000000009
(XEN)    ffff8300781cc198 ffff830839d990d0 ffff830839d8fe80 ffff82c480101bd9
(XEN)    ffff83107e80c5b0 ffff8300781cc000 ffff830839d8fea0 ffff82c480104f21
(XEN)    0000000000000009 ffff830839d990e0 ffff830839d8fee0 ffff82c480125b6c
(XEN)    ffff82c48024a020 ffff830839d8ff18 ffff82c48024a020 ffff830839d8ff18
(XEN)    ffff830839d99060 ffff830839d99040 ffff830839d8ff10 ffff82c48015645a
(XEN)    0000000000000000 ffff8300780da000 ffff8300780da000 ffffffffffffffff
(XEN)    ffff830839d8fe00 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 ffffffff8062bda0 ffff880fbb1e5fd8 0000000000000246
(XEN)    0000000000000000 000000010003347d 0000000000000000 0000000000000000
(XEN)    ffffffff800033aa 00000000deadbeef 00000000deadbeef 00000000deadbeef
(XEN)    0000010000000000 ffffffff800033aa 000000000000e033 0000000000000246
(XEN)    ffff880fbb1e5f08 000000000000e02b 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN)    [<ffff82c480126100>] active_timer+0xc/0x37
(XEN)    [<ffff82c480126ef9>] set_timer+0x102/0x218
(XEN)    [<ffff82c480117906>] csched_tick_resume+0x53/0x75
(XEN)    [<ffff82c4801214fa>] schedule_cpu_switch+0x1f1/0x25c
(XEN)    [<ffff82c4801019ff>] cpupool_assign_cpu_locked+0x61/0xd6
(XEN)    [<ffff82c480101bd9>] cpupool_assign_cpu_helper+0x9f/0xcd
(XEN)    [<ffff82c480104f21>] continue_hypercall_tasklet_handler+0x51/0xc3
(XEN)    [<ffff82c480125b6c>] do_tasklet+0xe1/0x155
(XEN)    [<ffff82c48015645a>] idle_loop+0x5f/0x67
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 9:
(XEN) Assertion 'timer->status >= TIMER_STATUS_inactive' failed at timer.c:279
(XEN) ****************************************


Juergen

--
Juergen Gross                 Principal Developer Operating Systems
TSP ES&S SWE OS6                       Telephone: +49 (0) 89 3222 2967
Fujitsu Technology Solutions              e-mail: juergen.gross@xxxxxxxxxxxxxx
Domagkstr. 28                           Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel