Re: [Xen-users] DomU causes Server to crash when using "disk = [ phy:LV, xvdb ....."

[Andrew Lyon <andrew.lyon@xxxxxxxxx>]

On Fri, Dec 18, 2009 at 5:44 PM,  <ist@xxxxxxxxxxxxxxxxxx> wrote:
>
> My shiny new server just dies on me whenever I use the phy:-Disk inside the
> DomU.
>
> The Dom0 has the following standard kernel from opensuse 11.2
> Linux Dom0 2.6.31.5-0.1-xen #1 SMP 2009-10-26 15:49:03 +0100 x86_64 x86_64
> x86_64 GNU/Linux
> (model name      : Intel(R) Core(TM) i7 CPU )
>
> One LVM is given to the DomU with:
> "disk=[ 'tap:aio:/data/xen/images/mail/xvda,xvda,w',
> 'phy:/dev/vg0/mail,xvdb,w' ]"
>
> It works in general, but after copying some data to it or doing a bonnie++
> test, the server crashes. (see logfiles below)
> I changed the filesystems, made some stress-tests on the LVM-volumes in the
> Dom0 itself and everything went well, no crash.
>
> But when accessing the LVM via xvdb out of one DomU the server dies
> repoducable after ca 2min of copying data for example.
>
> Can anyone give a hint? Can this be a memory/hardware issue?
>
> Thanks for any help
> Tom
>
>
> Logmessage 1 of 2:
>
>
> Message from syslogd@Dom0 at Dec 18 14:38:12 ...
>  kernel:[ 2566.695723] general protection fault: 0000 [#1] SMP
>
> Message from syslogd@Dom0 at Dec 18 14:38:12 ...
>  kernel:[ 2566.695734] last sysfs file: /sys/devices/xen-
> backend/vbd-2-51728/statistics/wr_sect
>
> Message from syslogd@Dom0 at Dec 18 14:38:12 ...
>  kernel:[ 2566.695874] Stack:
>
> Message from syslogd@Dom0 at Dec 18 14:38:12 ...
>  kernel:[ 2566.695892] Call Trace:
>
> Message from syslogd@Dom0 at Dec 18 14:38:12 ...
>  kernel:[ 2566.695932] Code: 8b 08 f7 c1 00 00 02 00 0f 85 c1 00 00 00 80 e5
> 01 0f 84 c4 00 00 00 48 8b 70 30 4a 8b 44 fb 08 48 8b 3e 48 8b 4e 08 48 89 4f
> 08 <48> 89 39 48 c7 06 00 01 10 00 48 c7 46 08 00 02 20 00 2b 56 18
> Dec 18 14:38:12 Dom0 kernel: [ 2566.695723] general protection fault: 0000
> [#1] SMP
> Dec 18 14:38:12 Dom0 kernel: [ 2566.695734] last sysfs file: /sys/devices/xen-
> backend/vbd-2-51728/statistics/wr_sect
> Dec 18 14:38:12 Dom0 kernel: [ 2566.695740] CPU 2
> Dec 18 14:38:12 Dom0 kernel: [ 2566.695744] Modules linked in: joydev st
> sr_mod ide_gd_mod ide_cd_mod xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4
> xt_state nf_conntrack xt_physdev iptable_filter ip_tables x_tables bridge stp
> llc netbk xenblk cdrom blkbk blkback_pagemap blktap xenbus_be edd ipv6
> reiserfs loop snd_hda_codec_realtek snd_hda_intel snd_hda_codec
> ide_pci_generic jmicron i2c_i801 ata_generic snd_hwdep iTCO_wdt 8250_pnp
> snd_pcm thermal processor ohci1394 snd_timer iTCO_vendor_support thermal_sys
> i2c_core pcspkr 8250 r8169 pata_jmicron sg wmi snd serial_core ieee1394
> soundcore hwmon button snd_page_alloc linear uhci_hcd ehci_hcd dm_snapshot
> dm_mod raid10 raid456 raid6_pq async_xor async_memcpy async_tx xor raid1 raid0
> atiixp sata_nv sata_sil sata_via via82cxxx ide_core 3w_9xxx 3w_xxxx
> Dec 18 14:38:12 Dom0 kernel: [ 2566.695871] Process events/2 (pid: 29,
> threadinfo ffff8801f09d4000, task ffff8801f09d2900)
> Dec 18 14:38:12 Dom0 kernel: [ 2566.695874] Stack:
> Dec 18 14:38:12 Dom0 kernel: [ 2566.695876]  ffff8800dbf1b450 ffff8800033e41c0
> 0000000000007d00 000000002e7c2b28
> Dec 18 14:38:12 Dom0 kernel: [ 2566.695880] <0> ffff8801f09d5d60
> ffff8801f0a75600 ffff8801f0a3f640 ffff8801f0a7ccc0
> Dec 18 14:38:12 Dom0 kernel: [ 2566.695886] <0> 000000000000000b
> ffff8801f0a3f680 ffff8801f09d5dd0 ffffffff8010fb87
> Dec 18 14:38:12 Dom0 kernel: [ 2566.695892] Call Trace:
> Dec 18 14:38:12 Dom0 kernel: [ 2566.695900]  [<ffffffff8010fb87>]
> drain_array+0xb7/0x180
> Dec 18 14:38:12 Dom0 kernel: [ 2566.695905]  [<ffffffff8010ff26>]
> cache_reap+0xa6/0x160
> Dec 18 14:38:12 Dom0 kernel: [ 2566.695911]  [<ffffffff80069463>]
> run_workqueue+0x83/0x230
> Dec 18 14:38:12 Dom0 kernel: [ 2566.695917]  [<ffffffff800696c4>]
> worker_thread+0xb4/0x140
> Dec 18 14:38:12 Dom0 kernel: [ 2566.695922]  [<ffffffff8006fb96>]
> kthread+0xb6/0xc0
> Dec 18 14:38:12 Dom0 kernel: [ 2566.695928]  [<ffffffff8000d38a>]
> child_rip+0xa/0x20
>
>
> Logmessage 2 of 2 (another crash)
>
> Dec 18 18:12:58 Dom0 kernel: [  881.314646] ------------[ cut here
> ]------------
> Dec 18 18:12:58 Dom0 kernel: [  881.314662] WARNING: at
> /usr/src/packages/BUILD/kernel-
> xen-2.6.31.5/linux-2.6.31/drivers/char/tty_buffer.c:92
> tty_buffer_free+0x8b/0xa0()
> Dec 18 18:12:58 Dom0 kernel: [  881.314672] Hardware name: MS-7522
> Dec 18 18:12:58 Dom0 kernel: [  881.314678] Modules linked in: xt_tcpudp
> nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_physdev
> iptable_filter ip_tables x_tables bridge stp llc netbk xenblk cdrom blkbk
> blkback_pagemap blktap xenbus_be ipv6 edd reiserfs loop ide_pci_generic
> jmicron ata_generic iTCO_wdt i2c_i801 snd_hda_codec_realtek thermal
> iTCO_vendor_support i2c_core pcspkr processor wmi snd_hda_intel pata_jmicron
> 8250_pnp thermal_sys ohci1394 r8169 snd_hda_codec 8250 ieee1394 sg button
> serial_core snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc hwmon
> linear uhci_hcd ehci_hcd dm_snapshot dm_mod raid10 raid456 raid6_pq async_xor
> async_memcpy async_tx xor raid1 raid0 atiixp sata_nv sata_sil sata_via
> via82cxxx ide_core 3w_9xxx 3w_xxxx
> Dec 18 18:12:58 Dom0 kernel: [  881.314742] Pid: 2209, comm: sshd Tainted: G
> D    2.6.31.5-0.1-xen #1
> Dec 18 18:12:58 Dom0 kernel: [  881.314746] Call Trace:
> Dec 18 18:12:58 Dom0 kernel: [  881.314755]  [<ffffffff800119b9>]
> try_stack_unwind+0x189/0x1b0
> Dec 18 18:12:58 Dom0 kernel: [  881.314762]  [<ffffffff8000f466>]
> dump_trace+0xa6/0x1e0
> Dec 18 18:12:58 Dom0 kernel: [  881.314766]  [<ffffffff800114c4>]
> show_trace_log_lvl+0x64/0x90
> Dec 18 18:12:58 Dom0 kernel: [  881.314771]  [<ffffffff80011513>]
> show_trace+0x23/0x40
> Dec 18 18:12:58 Dom0 kernel: [  881.314777]  [<ffffffff8046af06>]
> dump_stack+0x81/0x9e
> Dec 18 18:12:58 Dom0 kernel: [  881.314782]  [<ffffffff8004f4c0>]
> warn_slowpath_common+0x80/0xd0
> Dec 18 18:12:58 Dom0 kernel: [  881.314787]  [<ffffffff8004f532>]
> warn_slowpath_null+0x22/0x40
> Dec 18 18:12:58 Dom0 kernel: [  881.314791]  [<ffffffff802d598b>]
> tty_buffer_free+0x8b/0xa0
> Dec 18 18:12:58 Dom0 kernel: [  881.314796]  [<ffffffff802d5a38>]
> flush_to_ldisc+0x98/0x1d0
> Dec 18 18:12:58 Dom0 kernel: [  881.314801]  [<ffffffff802d5c33>]
> tty_flush_to_ldisc+0x23/0x40
> Dec 18 18:12:58 Dom0 kernel: [  881.314807]  [<ffffffff802d0561>]
> n_tty_poll+0x81/0x1c0
> Dec 18 18:12:58 Dom0 kernel: [  881.314812]  [<ffffffff802cb362>]
> tty_poll+0xa2/0xc0
> Dec 18 18:12:58 Dom0 kernel: [  881.314817]  [<ffffffff8012d76d>]
> do_select+0x39d/0x6f0
> Dec 18 18:12:58 Dom0 kernel: [  881.314822]  [<ffffffff8012de08>]
> core_sys_select+0x1b8/0x300
> Dec 18 18:12:58 Dom0 kernel: [  881.314826]  [<ffffffff8012e201>]
> sys_select+0x51/0x130
> Dec 18 18:12:58 Dom0 kernel: [  881.314831]  [<ffffffff8000c868>]
> system_call_fastpath+0x16/0x1b
> Dec 18 18:12:58 Dom0 kernel: [  881.314838]  [<00007f0331b776c3>]
> 0x7f0331b776c3
> Dec 18 18:12:58 Dom0 kernel: [  881.314841] ---[ end trace f0ce0fc5e34477b7
> ]---
> Dec 18 18:12:58 Dom0 kernel: [  881.314850] general protection fault: 0000
> [#2] SMP
> Dec 18 18:12:58 Dom0 kernel: [  881.314854] last sysfs file: /sys/devices/xen-
> backend/vbd-1-51728/statistics/wr_sect
> Dec 18 18:12:58 Dom0 kernel: [  881.314857] CPU 0
> Dec 18 18:12:58 Dom0 kernel: [  881.314859] Modules linked in: xt_tcpudp
> nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_physdev
> iptable_filter ip_tables x_tables bridge stp llc netbk xenblk cdrom blkbk
> blkback_pagemap blktap xenbus_be ipv6 edd reiserfs loop ide_pci_generic
> jmicron ata_generic iTCO_wdt i2c_i801 snd_hda_codec_realtek thermal
> iTCO_vendor_support i2c_core pcspkr processor wmi snd_hda_intel pata_jmicron
> 8250_pnp thermal_sys ohci1394 r8169 snd_hda_codec 8250 ieee1394 sg button
> serial_core snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc hwmon
> linear uhci_hcd ehci_hcd dm_snapshot dm_mod raid10 raid456 raid6_pq async_xor
> async_memcpy async_tx xor raid1 raid0 atiixp sata_nv sata_sil sata_via
> via82cxxx ide_core 3w_9xxx 3w_xxxx
> Dec 18 18:12:58 Dom0 kernel: [  881.314916] Pid: 2209, comm: sshd Tainted: G
> D W  2.6.31.5-0.1-xen #1 MS-7522
> Dec 18 18:12:58 Dom0 kernel: [  881.314920] RIP: e030:[<ffffffff80244fab>]
> [<ffffffff80244fab>] memcpy_c+0xb/0x20
> Dec 18 18:12:58 Dom0 kernel: [  881.314926] RSP: e02b:ffff88001ceb77a0
> EFLAGS: 00010006
> Dec 18 18:12:58 Dom0 kernel: [  881.314929] RAX: ffff88001e0e4f2a RBX:
> ffff88001ca3a800 RCX: 000000000000001a
> Dec 18 18:12:58 Dom0 kernel: [  881.314932] RDX: 0000000000000006 RSI:
> d6caf671b8b6e228 RDI: ffff88001e0e4f2a
> Dec 18 18:12:58 Dom0 kernel: [  881.314936] RBP: ffff88001ceb7888 R08:
> ffff88001ca3ad1c R09: 000000000000f724
> Dec 18 18:12:58 Dom0 kernel: [  881.314939] R10: 0000000000000005 R11:
> 0000000000000000 R12: 0000000000001000
> Dec 18 18:12:58 Dom0 kernel: [  881.314942] R13: 00000000000000d6 R14:
> d6caf671b8b6e228 R15: 0000000000000f29
> Dec 18 18:12:58 Dom0 kernel: [  881.314949] FS:  00007f033463b7b0(0000)
> GS:ffffc90000000000(0000) knlGS:0000000000000000
> Dec 18 18:12:58 Dom0 kernel: [  881.314953] CS:  e033 DS: 0000 ES: 0000 CR0:
> 000000008005003b
> Dec 18 18:12:58 Dom0 kernel: [  881.314956] CR2: 00007fd6341ad000 CR3:
> 000000001f1d4000 CR4: 0000000000002660
> Dec 18 18:12:58 Dom0 kernel: [  881.314960] DR0: 0000000000000000 DR1:
> 0000000000000000 DR2: 0000000000000000
> Dec 18 18:12:58 Dom0 kernel: [  881.314963] DR3: 0000000000000000 DR6:
> 00000000ffff0ff0 DR7: 0000000000000400
> Dec 18 18:12:58 Dom0 kernel: [  881.314967] Process sshd (pid: 2209,
> threadinfo ffff88001ceb6000, task ffff88001e772200)
> Dec 18 18:12:58 Dom0 kernel: [  881.314971] Stack:
> Dec 18 18:12:58 Dom0 kernel: [  881.314972]  ffffffff802d2ce6 ffffffff807395e6
> 00000000000000d6 ffff88001ca3ad1c
> Dec 18 18:12:58 Dom0 kernel: [  881.314977] <0> 0000000000000000
> ffffffff8080dc8e 000000000000f6ed ffff88001ceb77f8
> Dec 18 18:12:58 Dom0 kernel: [  881.314982] <0> 0000000054eca714
> 0000000000000000 0000000000000000 ffff88001ceb7838
> Dec 18 18:12:58 Dom0 kernel: [  881.314988] Call Trace:
> Dec 18 18:12:58 Dom0 kernel: [  881.314992]  [<ffffffff802d2ce6>]
> n_tty_receive_buf+0x306/0x490
> Dec 18 18:12:58 Dom0 kernel: [  881.314997]  [<ffffffff802d5b3b>]
> flush_to_ldisc+0x19b/0x1d0
> Dec 18 18:12:58 Dom0 kernel: [  881.315002]  [<ffffffff802d5c33>]
> tty_flush_to_ldisc+0x23/0x40
> Dec 18 18:12:58 Dom0 kernel: [  881.315007]  [<ffffffff802d0561>]
> n_tty_poll+0x81/0x1c0
> Dec 18 18:12:58 Dom0 kernel: [  881.315011]  [<ffffffff802cb362>]
> tty_poll+0xa2/0xc0
> Dec 18 18:12:58 Dom0 kernel: [  881.315015]  [<ffffffff8012d76d>]
> do_select+0x39d/0x6f0
> Dec 18 18:12:58 Dom0 kernel: [  881.315020]  [<ffffffff8012de08>]
> core_sys_select+0x1b8/0x300
> Dec 18 18:12:58 Dom0 kernel: [  881.315024]  [<ffffffff8012e201>]
> sys_select+0x51/0x130
> Dec 18 18:12:58 Dom0 kernel: [  881.315029]  [<ffffffff8000c868>]
> system_call_fastpath+0x16/0x1b
> Dec 18 18:12:58 Dom0 kernel: [  881.315035]  [<00007f0331b776c3>]
> 0x7f0331b776c3
> Dec 18 18:12:58 Dom0 kernel: [  881.315037] Code: 81 ea d8 1f 00 00 48 3b 42
> 20 73 07 48 8b 50 f9 31 c0 c3 31 d2 48 c7 c0 f2 ff ff ff c3 90 90 90 48 89 f8
> 89 d1 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 66 66 66 66 2e 0f 1f 84 00
> 00 00 00 00
> Dec 18 18:12:58 Dom0 kernel: [  881.315069] RIP  [<ffffffff80244fab>]
> memcpy_c+0xb/0x20
> Dec 18 18:12:58 Dom0 kernel: [  881.315072]  RSP <ffff88001ceb77a0>
> Dec 18 18:12:58 Dom0 kernel: [  881.315075] ---[ end trace f0ce0fc5e34477b8
> ]---
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users
>

There are many known issues with 11.2 2.6.31.5-0.1-xen which have
recently been fixed, try using a newer kernel from
ftp://ftp.suse.com/pub/projects/kernel/kotd/openSUSE-11.2/ and I think
you will find it works a lot better.

Andy

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users