[Xen-devel] Re: Crash on blktap shutdown

[Jeremy Fitzhardinge <jeremy@xxxxxxxx>]

On 02/24/2010 03:20 PM, Daniel Stodden wrote:
> Jake, any immediate ideas?
>    
Just got another one on domain shutdown.  The crashing instruction is:
0xffffffff8104a3f2 <lock_timer_base+17>:    mov    0x28(%r12),%r14



r12 = 6b6b6b6b6b6b6c8b

0x6b is the use-after-free poison value.  So I think a use-after-free.

0xffffffff8104a3f2 is in lock_timer_base 
(/home/jeremy/git/linux/kernel/timer.c:620).
615             __acquires(timer->base->lock)
616     {
617             struct tvec_base *base;
618     
619             for (;;) {
620                     struct tvec_base *prelock_base = timer->base;
621                     base = tbase_get_base(prelock_base);
622                     if (likely(base != NULL)) {
623                             spin_lock_irqsave(&base->lock, *flags);
624                             if (likely(prelock_base == timer->base))


general protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC
last sysfs file: /sys/devices/virtual/blktap2/blktap0/remove
CPU 1
Modules linked in: e1000 evdev ahci dm_mod sd_mod mptspi mptscsih mptbase scsi_]
Pid: 6533, comm: xend Not tainted 2.6.32.9 #356 PowerEdge 1850
RIP: e030:[<ffffffff8104a3f2>]  [<ffffffff8104a3f2>] lock_timer_base+0x11/0x4d
RSP: e02b:ffff880021a73ce8  EFLAGS: 00010286
RAX: ffff88001d858f40 RBX: 6b6b6b6b6b6b6c8b RCX: 0000000000000000
RDX: ffffffff8104abda RSI: ffff880021a73d20 RDI: 6b6b6b6b6b6b6c8b
RBP: ffff880021a73d08 R08: 0000000000000000 R09: 0000000000000001
R10: ffffffff8104abda R11: ffff880003cd1810 R12: 6b6b6b6b6b6b6c8b
R13: ffff880021a73d20 R14: 000000000000011e R15: ffff880021a73e20
FS:  00007f164dffb910(0000) GS:ffff8800028fb000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000001d62140 CR3: 000000002eac8000 CR4: 0000000000000660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process xend (pid: 6533, threadinfo ffff880021a72000, task ffff88001d858f40)
Stack:
 6b6b6b6b6b6b6c8b 00000000ffffffff ffff88002f2802e8 000000000000011e
<0>  ffff880021a73d38 ffffffff8104a7b5 0000000000000001 ffffffff8104abda
<0>  6b6b6b6b6b6b6c8b 6b6b6b6b6b6b6cbb ffff880021a73d78 ffffffff8104ac68
Call Trace:
 [<ffffffff8104a7b5>] try_to_del_timer_sync+0x1b/0x81
 [<ffffffff8104abda>] ? del_timer_sync+0x0/0xa1
 [<ffffffff8104ac68>] del_timer_sync+0x8e/0xa1
 [<ffffffff8104abda>] ? del_timer_sync+0x0/0xa1
 [<ffffffff811e79b7>] ? kobject_release+0x0/0x66
 [<ffffffff811d842c>] blk_sync_queue+0x18/0x34
 [<ffffffff811d8457>] blk_cleanup_queue+0xf/0x4b
 [<ffffffff81254039>] blktap_device_destroy+0xad/0xd7
 [<ffffffff812512a5>] blktap_control_destroy_device+0x55/0x154
 [<ffffffff81390438>] ? mutex_lock_nested+0x2a5/0x2b4
 [<ffffffff81254de5>] blktap_sysfs_remove_device+0x39/0x49
 [<ffffffff81294170>] dev_attr_store+0x1b/0x1d
 [<ffffffff810fa9d4>] sysfs_write_file+0xf6/0x132
 [<ffffffff810b03f0>] vfs_write+0xad/0x14e
 [<ffffffff810b0c1b>] ? fget_light+0x52/0xeb
 [<ffffffff811eab92>] ? __up_read+0x1c/0xa2
 [<ffffffff810b054a>] sys_write+0x45/0x6c
 [<ffffffff81011c82>] system_call_fastpath+0x16/0x1b
Code: 55 31 d2 48 89 e5 31 f6 65 48 8b 3c 25 c0 cb 00 00 e8 95 77 00 00 c9 48 9
RIP  [<ffffffff8104a3f2>] lock_timer_base+0x11/0x4d
 RSP<ffff880021a73ce8>
---[ end trace 767ddf28dd1b4a3e ]---



> Daniel
>
> On Wed, 2010-02-24 at 17:55 -0500, Jeremy Fitzhardinge wrote:
>    
> > When rebooting the machine,  I got this crash from blktap.  The rip maps to 
> > line 262 in
> > 0xffffffff812548a1 is in blktap_request_pool_free 
> > (/home/jeremy/git/linux/drivers/xen/blktap/request.c:262).
> > 257             spin_lock_irqsave(&pool.lock, flags);
> > 258     
> > 259             pool.status = BLKTAP_POOL_CLOSING;
> > 260             while (atomic_read(&pool.reqs_in_use)) {
> > 261                     spin_unlock_irqrestore(&pool.lock, flags);
> > 262                     wait_event(pool.wait_queue, 
> > !atomic_read(&pool.reqs_in_use));
> > 263                     spin_lock_irqsave(&pool.lock, flags);
> > 264             }
> > 265     
> > 266             for (i = 0; i<   MAX_BUCKETS; i++) {
> >
> >
> > blktap_ring_vm_close: unmapping ring 6
> > blktap_ring_release: freeing device 6
> > general protection fault: 0000 [#2] SMP DEBUG_PAGEALLOC
> > last sysfs file: /sys/devices/virtual/net/eth0/address
> > CPU 1
> > Modules linked in: e1000 evdev ahci dm_mod sd_mod mptspi mptscsih mptbase scsi_]
> > Pid: 993, comm: tapdisk2 Tainted: G      D    2.6.32.8 #355 PowerEdge 1850
> > RIP: e030:[<ffffffff8125413b>]  [<ffffffff8125413b>] blktap_device_restart+0x7a8
> > RSP: e02b:ffff88002d767be8  EFLAGS: 00010092
> > RAX: ffff88002ea06b08 RBX: ffff88002f319090 RCX: 0000000000000000
> > RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6b6b
> > RBP: ffff88002d767bf8 R08: 0000000000000002 R09: 0000000000000001
> > R10: ffffffff8125412d R11: ffffffff811eaa4a R12: ffff88002f319330
> > R13: ffff88002f3191b8 R14: ffff8800242a3a50 R15: 0000000000000001
> > FS:  00007f7e3234a740(0000) GS:ffff8800028fb000(0000) knlGS:0000000000000000
> > CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> > CR2: 00000036a05a8d84 CR3: 000000002d364000 CR4: 0000000000000660
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Process tapdisk2 (pid: 993, threadinfo ffff88002d766000, task ffff8800242c3d00)
> > Stack:
> >    ffff88002f319090 ffff88002f319238 ffff88002d767c28 ffffffff81251b3b
> > <0>   ffff8800242a3a50 ffff88002f2c2870 ffff880002909820 ffff88002400ad60
> > <0>   ffff88002d767c48 ffffffff8109aead ffff8800242a3a50 ffff88002400ad00
> > Call Trace:
> >    [<ffffffff81251b3b>] blktap_ring_vm_close+0x39/0x12d
> >    [<ffffffff8109aead>] remove_vma+0x3b/0x71
> >    [<ffffffff8109b036>] exit_mmap+0x153/0x175
> >    [<ffffffff8103eef6>] mmput+0x3e/0xd9
> >    [<ffffffff81042b83>] exit_mm+0x100/0x10b
> >    [<ffffffff81044416>] do_exit+0x1b9/0x638
> >    [<ffffffff8104d797>] ? get_signal_to_deliver+0x2dd/0x36e
> >    [<ffffffff8100efef>] ? xen_restore_fl_direct_end+0x0/0x1
> >    [<ffffffff81044908>] do_group_exit+0x73/0x9c
> >    [<ffffffff8104d809>] get_signal_to_deliver+0x34f/0x36e
> >    [<ffffffff810111c4>] do_signal+0x6d/0x6b0
> >    [<ffffffff8104ef1f>] ? sys_getsid+0x88/0xaf
> >    [<ffffffff810bd680>] ? poll_select_copy_remaining+0xc9/0xee
> >    [<ffffffff8101182e>] do_notify_resume+0x27/0x47
> >    [<ffffffff81390f80>] ? trace_hardirqs_on_thunk+0x3a/0x3f
> >    [<ffffffff810549ef>] ? remove_wait_queue+0x12/0x45
> >    [<ffffffff81011f56>] int_signal+0x12/0x17
> > Code: a8 01 74 0a 48 89 df e8 24 e6 ff ff eb 46 4c 8d a3 a0 02 00 00 4c 89 e7 e
> > RIP  [<ffffffff8125413b>] blktap_device_restart+0x7a/0xa8
> >    RSP<ffff88002d767be8>
> > ---[ end trace 1b88501e9b8effb5 ]---
> >
> >         J
> >
> >      
>    

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel