[Xen-devel] pvops domU crashed under Xen 4 rc2

[Yasir Assam <mail@xxxxxxxxxxxxxxx>]

Xen 4 rc2, with linux pvops dom0 and a kernel.org domU (2.6.32.7), all 
64-bit.
I don't know if it's a bug in the domU kernel, dom0, Xen, or a hardware 
problem.
I did a full e2fsck on this partition a few days ago, and I ran a full 
Western Digital diagnostic on the HD (no problems), so the ext3 
corruption must have come in recently. I did another WD diag test after 
the crash, again no problems.
I've been getting random segfaults on this domU ever since upgrading to 
Xen 4 on a new motherboard (unfortunately I can't use the 2.6.18 
linux-xen dom0 because it doesn't have the drivers to boot on this 
motherboard).
Yesterday I stress tested the CPU by doing a full kernel rebuild on all 
4 cores (rebuilding in an infinite loop). I don't know if this is a good 
test, but there were no errors after 50 rebuilds. I've also done a full 
memtest86 at least 4 times, with no errors.
Console and e2fsck dump below.

Thanks,
Yasir

EXT3 Inode ffff88001f806048: orphan list check failed!
ffff88001f806048: f767acc8 00002086 81ed2bc0 00000b06
ffff88001f806058: adff8f36 000000ff 00752ddb 00752ddc
ffff88001f806068: 00752ddd 00752dde 00752ddf 00752de0
ffff88001f806078: 00752de1 00000000 00007ea0 00000000
ffff88001f806088: f767ac75 00002086 81ed2bc0 00000b06
ffff88001f806098: adff8f36 000000ff 00000000 00000000
ffff88001f8060a8: 00000000 c2c2c2c2 00000000 00000000
ffff88001f8060b8: 1f8060b8 ffff8800 0000671e 00000000
ffff88001f8060c8: f767ac75 00002086 81ed2bc0 00000b06
ffff88001f8060d8: adff8f36 000000ff c2c20000 c2c2c2c2
ffff88001f8060e8: 00000001 00000000 1f8060f0 ffff8800
ffff88001f8060f8: 1f8060f0 ffff8800 00000000 00000000
ffff88001f806108: 00b58375 00b58375 00000000 00000000
ffff88001f806118: 00000000 00000000 00100100 00000000
ffff88001f806128: 00200200 00000000 1f806130 ffff8800
ffff88001f806138: 1f806130 ffff8800 1f806140 ffff8800
ffff88001f806148: 1f806140 ffff8800 003047d0 00000000
ffff88001f806158: 00000000 00000001 000003e8 000003e8
ffff88001f806168: 00000000 00000000 00000001 00000000
ffff88001f806178: 0003b23b 00000000 4b67e2d9 00000000
ffff88001f806188: 00000000 00000000 387d252c 00000000
ffff88001f806198: 00000000 00000000 47fcbab7 00000000
ffff88001f8061a8: 00000000 00000000 000001e8 00000000
ffff88001f8061b8: 0000000c 816d0000 00000000 00000000
ffff88001f8061c8: 00000001 00000000 1f8061d0 ffff8800
ffff88001f8061d8: 1f8061d0 ffff8800 00000000 00000000
ffff88001f8061e8: 00000000 00000000 1f8061f0 ffff8800
ffff88001f8061f8: 1f8061f0 ffff8800 812b35d0 ffffffff
ffff88001f806208: 812b3500 ffffffff 366c4000 ffff8800
ffff88001f806218: 00000000 00000000 1f806228 ffff8800
ffff88001f806228: 1f806110 ffff8800 00000000 00000020
ffff88001f806238: 00000000 00000000 00000000 00000000
ffff88001f806248: 00000000 00000000 00010001 00000000
ffff88001f806258: 1f806258 ffff8800 1f806258 ffff8800
ffff88001f806268: 00000000 00000000 00000000 00000000
ffff88001f806278: 00000000 00000000 812b3890 ffffffff
ffff88001f806288: 000200da 00000000 365aa1f0 ffff8800
ffff88001f806298: 00000000 00000000 1f8062a0 ffff8800
ffff88001f8062a8: 1f8062a0 ffff8800 00000000 00000000
ffff88001f8062b8: 1f8062b8 ffff8800 1f8062b8 ffff8800
ffff88001f8062c8: 00000000 00000000 4b165187 00000000
ffff88001f8062d8: 00000000 00000000 1f8062e0 ffff8800
ffff88001f8062e8: 1f8062e0 ffff8800 00000001 00000000
ffff88001f8062f8: 1f8062f8 ffff8800 1f8062f8 ffff8800
ffff88001f806308: 00000000 00000000 00000040 00000000
ffff88001f806318: 00000000 00000000 00000000 00000000
ffff88001f806328: ffffffff ffffffff ffffffff ffffffff
ffff88001f806338: 00000000 00000000
Pid: 133, comm: kswapd0 Not tainted 2.6.32.7-xen-domU-64 #1
Call Trace:
 [<ffffffff810ea2f3>] ? ext3_destroy_inode+0x65/0x79
 [<ffffffff810a5fe2>] ? dispose_list+0xda/0x113
 [<ffffffff8100d115>] ? xen_force_evtchn_callback+0x9/0xa
 [<ffffffff8100d782>] ? check_events+0x12/0x20
 [<ffffffff810a622a>] ? shrink_icache_memory+0x20f/0x245
 [<ffffffff8129f197>] ? _spin_unlock_irqrestore+0x19/0x1d
 [<ffffffff81076884>] ? shrink_slab+0xe0/0x153
 [<ffffffff81076dac>] ? kswapd+0x4b5/0x65f
 [<ffffffff8107480b>] ? isolate_pages_global+0x0/0x216
 [<ffffffff810472f4>] ? autoremove_wake_function+0x0/0x2e
 [<ffffffff810768f7>] ? kswapd+0x0/0x65f
 [<ffffffff8104707d>] ? kthread+0x79/0x81
 [<ffffffff81010b8a>] ? child_rip+0xa/0x20
 [<ffffffff8100fd61>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8101051d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81010b80>] ? child_rip+0x0/0x20
BUG: unable to handle kernel paging request at 0000000000008052
IP: [<ffffffff8108f48e>] free_block+0x6a/0x115
PGD 241c9067 PUD 2f29d067 PMD 0
Oops: 0002 [#1] SMP
last sysfs file: /sys/devices/vbd-51713/block/xvda1/size
CPU 0
Pid: 133, comm: kswapd0 Not tainted 2.6.32.7-xen-domU-64 #1
RIP: e030:[<ffffffff8108f48e>]  [<ffffffff8108f48e>] free_block+0x6a/0x115
RSP: e02b:ffff8800372d9c30  EFLAGS: 00010082
RAX: 00002085989d0ae8 RBX: ffff88001f806c28 RCX: ffff880037214140
RDX: 000000000000804a RSI: ffff88001f806000 RDI: ffff88001f806c28
RBP: ffff880037207540 R08: 0000000000000008 R09: ffff8800372d8000
R10: 0000000000000002 R11: 000000008107aea3 R12: 0000000000000011
R13: ffff8800372232a0 R14: 000000000000000e R15: 000000000000001b
FS:  00007fa910cb36f0(0000) GS:ffff880001687000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000008052 CR3: 0000000024190000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kswapd0 (pid: 133, threadinfo ffff8800372d8000, task 
ffff8800372ce6f0)
Stack:
 ffff88001f806120 ffff880037223200 000000000000001b ffff880037207540
<0> ffff880037214140 ffff88001f8102e8 ffff88001f810c68 ffffffff8108f383
<0> ffff88001f810d30 ffff88001f810d30 ffff88001f810d40 0000000000000026
Call Trace:
 [<ffffffff8108f383>] ? kmem_cache_free+0x9b/0xe7
 [<ffffffff810a5fe2>] ? dispose_list+0xda/0x113
 [<ffffffff8100d115>] ? xen_force_evtchn_callback+0x9/0xa
 [<ffffffff8100d782>] ? check_events+0x12/0x20
 [<ffffffff810a622a>] ? shrink_icache_memory+0x20f/0x245
 [<ffffffff8129f197>] ? _spin_unlock_irqrestore+0x19/0x1d
 [<ffffffff81076884>] ? shrink_slab+0xe0/0x153
 [<ffffffff81076dac>] ? kswapd+0x4b5/0x65f
 [<ffffffff8107480b>] ? isolate_pages_global+0x0/0x216
 [<ffffffff810472f4>] ? autoremove_wake_function+0x0/0x2e
 [<ffffffff810768f7>] ? kswapd+0x0/0x65f
 [<ffffffff8104707d>] ? kthread+0x79/0x81
 [<ffffffff81010b8a>] ? child_rip+0xa/0x20
 [<ffffffff8100fd61>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8101051d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81010b80>] ? child_rip+0x0/0x20
Code: ff 48 6b c0 38 48 01 d0 66 83 38 00 79 04 48 8b 40 10 80 38 00 78 
04 0f 0b eb fe 48 8b 70 30 4a 8b 4c f5 08 48 8b 16 48 8b 46 08 <48> 89 
42 08 48 89 10 89 d8 48 c7 06 00 01 10 00 48 c7 46 08 00
RIP  [<ffffffff8108f48e>] free_block+0x6a/0x115
 RSP <ffff8800372d9c30>
CR2: 0000000000008052
---[ end trace a5afef6b2f15b3a2 ]---






# e2fsck -f urd-disk
e2fsck 1.41.9 (22-Aug-2009)
urd-disk: recovering journal
Clearing orphaned inode 2655619 (uid=0, gid=0, mode=0100644, size=0)
Clearing orphaned inode 2654223 (uid=101, gid=103, mode=0100600, size=0)
Clearing orphaned inode 2654218 (uid=101, gid=103, mode=0100600, size=0)
Clearing orphaned inode 2654215 (uid=101, gid=103, mode=0100600, size=0)
Clearing orphaned inode 2654213 (uid=101, gid=103, mode=0100600, size=0)
Clearing orphaned inode 2654211 (uid=101, gid=103, mode=0100600, size=0)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

urd-disk: ***** FILE SYSTEM WAS MODIFIED *****
urd-disk: 1049604/19660800 files (5.0% non-contiguous), 
26670547/39321600 blocks

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel