|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-users] Kernel BUG on drbd disconnect
Hi all, [xen-users: Please CC me in replies, I am not subscribed to this list.] First of all, please excuse this cross-posting, but I am, so to say, stumped. My cluster of 2 Debian etch boxes (on Intel CPUs with vmx support) running drbd8 on Linux kernel packages linux-image-2.6.18-4-xen-686 (2.6.18.dfsg.1-12etch2) or linux-image-2.6.18-5-xen-686 (2.6.18.dfsg.1-13etch4), the latter sometimes exposing a Xen networking bug similar to http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=451297 on one of the systems (but I haven't started debugging this one). I have compiled my own drbd8 kernel modules, so that Debian package version is 8.0.7-1 for both drbd8-utils and drbd8-2.6.18-*-xen-686. Xen packages are from Debian etch, i.e. xen-utils-3.0.3-1, xen-hypervisor-3.0.3-1-i386-pae, etc. version 3.0.3-0-4. domU instances are also Debian etch using linux-image-2.6.18-4-xen-686 to start from dom0 and linux-modules-2.6.18-4-xen-686 installed in the domUs. Both boxes have a ca. 60GB LVM2 VG with >20 LVs of equal size and name on both. There is one drbd ressource for each LV (currently up to drbd28, but using only every second one for now), which are used as backend devices for the Xen domUs. At any time, only one node switches the respective drbd ressource to primary and then starts the domU (live migration is not yet used, so primary-primary does not happen right now). The drbd ressources all have a similar configuration: resource "xen-xxx" { protocol C; startup { wfc-timeout 120; ## 2 minutes. degr-wfc-timeout 120; ## 2 minutes. } disk { on-io-error detach; } net { max-buffers 128; allow-two-primaries; cram-hmac-alg "sha256"; shared-secret "xxxxxxxxxxxxxxxxxxxxxxxx"; after-sb-0pri disconnect; after-sb-1pri disconnect; after-sb-2pri disconnect; rr-conflict disconnect; } syncer { rate 50M; after "xen-yyy"; } on aaaa { device /dev/drbd26; disk /dev/mapper/xendomains-xxx; address 192.168.255.30:7814; meta-disk internal; } on bbbb { device /dev/drbd26; disk /dev/mapper/xendomains-xxx; address 192.168.255.29:7814; meta-disk internal; } } That is, syncer rate is moderate (1Gbps connection), max-buffers low so as not to run into out-of-memory issues (this used to be a problem), and the syncer is serialized. Sometimes, when rebooting one of the nodes, it will crash the respective other. This is reproducible, but not ever time. Just an hour ago, it happened again and we were able to capture the kernel output on the serial console after reboot was started on the other node (so to be clear: this is the one that wasn't touched by any manual intervention): drbd0: meta connection shut down by peer. drbd26: meta connection shut down by peer. drbd0: tl_clear() drbd26: tl_clear() drbd28: meta connection shut down by peer. drbd28: tl_clear() drbd24: PingAck did not arrive in time. drbd24: short read expecting header on sock: r=-512 drbd24: tl_clear() drbd20: PingAck did not arrive in time. drbd20: short read expecting header on sock: r=-512 drbd20: tl_clear() drbd14: PingAck did not arrive in time. drbd14: short read expecting header on sock: r=-512 drbd14: tl_clear() drbd4: PingAck did not arrive in time. drbd4: short read expecting header on sock: r=-512 drbd4: tl_clear() drbd10: PingAck did not arrive in time. drbd10: short read expecting header on sock: r=-512 drbd10: tl_clear() drbd6: PingAck did not arrive in time. drbd6: short read expecting header on sock: r=-512 drbd6: tl_clear() drbd22: PingAck did not arrive in time. drbd22: short read expecting header on sock: r=-512 drbd22: tl_clear() BUG: unable to handle kernel paging request at virtual address c081e000 printing eip: c01bb090 19529000 -> *pde = 00000001:19f48001 27948000 -> *pme = 00000000:07099067 01099000 -> *pte = 00000000:00000000 Oops: 0000 [#1] SMP Modules linked in: sha256 drbd cn xt_physdev button ac battery ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs bridge ip6t_LOG ipt_LOG xt_state ipt_REJECT xt_tcpudp ip6table_filter ip6_tables ipv6 iptable_mangle iptable_nat ip_nat ip_conntrack nfnetlink iptable_filter ip_tables x_tables xfs dm_crypt loop serio_raw serial_core psmouse rtc pcspkr shpchp pci_hotplug tsdev evdev ext3 jbd mbcache dm_mirror dm_snapshot dm_mod raid1 md_mod ide_generic ide_cd cdrom sd_mod generic usbhid tg3 skge ahci libata piix scsi_mod ide_core ehci_hcd uhci_hcd usbcore thermal processor fan CPU: 0 EIP: 0061:[<c01bb090>] Not tainted VLI EFLAGS: 00010202 (2.6.18-5-xen-686 #1) EIP is at csum_partial+0x88/0x120 eax: 00000000 ebx: c01bb0f0 ecx: 00000007 edx: 00000400 esi: c081e080 edi: 00000408 ebp: 00000064 esp: c031fd54 ds: 007b es: 007b ss: 0069 Process swapper (pid: 0, ti=c031e000 task=c02cf660 task.ti=c031e000) Stack: c081e000 00000064 c022d626 c081e000 00000400 00000000 00000010 e4f00d2c 00000000 00000050 00000464 c911cef4 c022e532 c911c800 00000400 e4f00d2c e58258ec e5825900 c031fe54 c0231f33 92a6a1ff c83c4a00 00000003 c031fe54 Call Trace: [<c022d626>] skb_checksum+0x112/0x27e [<c022e532>] pskb_expand_head+0xce/0x112 [<c0231f33>] skb_checksum_help+0x5d/0xac [<e93ef2ea>] ip_nat_fn+0x42/0x184 [iptable_nat] [<e93f8092>] ipt_local_hook+0x76/0xcc [iptable_mangle] [<e93ef61e>] ip_nat_local_fn+0x34/0xaa [iptable_nat] [<c024def0>] dst_output+0x0/0x7 [<c0246e28>] nf_iterate+0x30/0x61 [<c024def0>] dst_output+0x0/0x7 [<c0246f4e>] nf_hook_slow+0x3a/0x90 [<c024def0>] dst_output+0x0/0x7 [<c02500e8>] ip_queue_xmit+0x35f/0x3b3 [<c024def0>] dst_output+0x0/0x7 [<c0115f49>] rebalance_tick+0x116/0x2ae [<c025dab0>] tcp_transmit_skb+0x604/0x632 [<c025e80c>] tcp_retransmit_skb+0x4e2/0x5c7 [<c012e066>] hrtimer_run_queues+0x147/0x15f [<c0257960>] tcp_enter_loss+0x1a1/0x1fd [<c02608e3>] tcp_write_timer+0x0/0x5c9 [<c0260cdb>] tcp_write_timer+0x3f8/0x5c9 [<c0123440>] run_timer_softirq+0x101/0x15c [<c011f41e>] __do_softirq+0x5e/0xc3 [<c011f4bd>] do_softirq+0x3a/0x4a [<c0106131>] do_IRQ+0x48/0x53 [<c020c1cc>] evtchn_do_upcall+0x64/0x9b [<c0104a51>] hypervisor_callback+0x3d/0x48 [<c0107342>] raw_safe_halt+0x8c/0xaf [<c0102c5f>] xen_idle+0x22/0x2e [<c0102d7e>] cpu_idle+0x91/0xab [<c03236fc>] start_kernel+0x378/0x37f Code: 00 74 b6 83 e9 02 77 cd 74 16 83 c1 02 0f 84 9f 00 00 00 0f b6 1e 01 d8 83 d0 00 e9 92 00 00 00 66 03 06 83 d0 00 e9 87 00 00 00 <03> 46 80 13 46 84 13 46 88 13 46 8c 13 46 90 13 46 94 13 46 98 EIP: [<c01bb090>] csum_partial+0x88/0x120 SS:ESP 0069:c031fd54 <0>Kernel panic - not syncing: Fatal exception in interrupt BUG: warning at arch/i386/kernel/smp-xen.c:526/smp_call_function() [<c010cddd>] smp_call_function+0x53/0xf8 [<c011b4be>] printk+0x14/0x18 [<c010ce95>] smp_send_stop+0x13/0x1e [<c011aaeb>] panic+0x45/0xde [<c0105235>] die+0x242/0x276 [<c0111d05>] do_page_fault+0xa53/0xb76 [<c01bb0f0>] csum_partial+0xe8/0x120 [<c01112b2>] do_page_fault+0x0/0xb76 [<c0104a0f>] error_code+0x2b/0x30 [<c01bb0f0>] csum_partial+0xe8/0x120 [<c01bb090>] csum_partial+0x88/0x120 [<c022d626>] skb_checksum+0x112/0x27e [<c022e532>] pskb_expand_head+0xce/0x112 [<c0231f33>] skb_checksum_help+0x5d/0xac [<e93ef2ea>] ip_nat_fn+0x42/0x184 [iptable_nat] [<e93f8092>] ipt_local_hook+0x76/0xcc [iptable_mangle] [<e93ef61e>] ip_nat_local_fn+0x34/0xaa [iptable_nat] [<c024def0>] dst_output+0x0/0x7 [<c0246e28>] nf_iterate+0x30/0x61 [<c024def0>] dst_output+0x0/0x7 [<c0246f4e>] nf_hook_slow+0x3a/0x90 [<c024def0>] dst_output+0x0/0x7 [<c02500e8>] ip_queue_xmit+0x35f/0x3b3 [<c024def0>] dst_output+0x0/0x7 [<c0115f49>] rebalance_tick+0x116/0x2ae [<c025dab0>] tcp_transmit_skb+0x604/0x632 [<c025e80c>] tcp_retransmit_skb+0x4e2/0x5c7 [<c012e066>] hrtimer_run_queues+0x147/0x15f [<c0257960>] tcp_enter_loss+0x1a1/0x1fd [<c02608e3>] tcp_write_timer+0x0/0x5c9 [<c0260cdb>] tcp_write_timer+0x3f8/0x5c9 [<c0123440>] run_timer_softirq+0x101/0x15c [<c011f41e>] __do_softirq+0x5e/0xc3 [<c011f4bd>] do_softirq+0x3a/0x4a [<c0106131>] do_IRQ+0x48/0x53 [<c020c1cc>] evtchn_do_upcall+0x64/0x9b [<c0104a51>] hypervisor_callback+0x3d/0x48 [<c0107342>] raw_safe_halt+0x8c/0xaf [<c0102c5f>] xen_idle+0x22/0x2e [<c0102d7e>] cpu_idle+0x91/0xab [<c03236fc>] start_kernel+0x378/0x37f (XEN) Domain 0 crashed: rebooting machine in 5 seconds. Then a reboot. ksymoops doesn't want to read the symbols from /proc/kallsyms (and /proc/ksyms) doesn't exist, so the decoding isn't complete: ksymoops 2.4.11 on i686 2.6.18-5-xen-686. Options used -V (default) -k /proc/ksyms (default) -l /proc/modules (default) -o /lib/modules/2.6.18-5-xen-686/ (default) -m /boot/System.map-2.6.18-5-xen-686 (default) Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options. Error (regular_file): read_ksyms stat /proc/ksyms failed No modules in ksyms, skipping objects No ksyms, skipping lsmod BUG: unable to handle kernel paging request at virtual address c081e000 c01bb090 19529000 -> *pde = 00000001:19f48001 27948000 -> *pme = 00000000:07099067 01099000 -> *pte = 00000000:00000000 Oops: 0000 [#1] CPU: 0 EIP: 0061:[<c01bb090>] Not tainted VLI Using defaults from ksymoops -t elf32-i386 -a i386 EFLAGS: 00010202 (2.6.18-5-xen-686 #1) eax: 00000000 ebx: c01bb0f0 ecx: 00000007 edx: 00000400 esi: c081e080 edi: 00000408 ebp: 00000064 esp: c031fd54 ds: 007b es: 007b ss: 0069 Stack: c081e000 00000064 c022d626 c081e000 00000400 00000000 00000010 e4f00d2c 00000000 00000050 00000464 c911cef4 c022e532 c911c800 00000400 e4f00d2c e58258ec e5825900 c031fe54 c0231f33 92a6a1ff c83c4a00 00000003 c031fe54 Call Trace: [<c022d626>] skb_checksum+0x112/0x27e [<c022e532>] pskb_expand_head+0xce/0x112 [<c0231f33>] skb_checksum_help+0x5d/0xac [<e93ef2ea>] ip_nat_fn+0x42/0x184 [iptable_nat] [<e93f8092>] ipt_local_hook+0x76/0xcc [iptable_mangle] [<e93ef61e>] ip_nat_local_fn+0x34/0xaa [iptable_nat] [<c024def0>] dst_output+0x0/0x7 [<c0246e28>] nf_iterate+0x30/0x61 [<c024def0>] dst_output+0x0/0x7 [<c0246f4e>] nf_hook_slow+0x3a/0x90 [<c024def0>] dst_output+0x0/0x7 [<c02500e8>] ip_queue_xmit+0x35f/0x3b3 [<c024def0>] dst_output+0x0/0x7 [<c0115f49>] rebalance_tick+0x116/0x2ae [<c025dab0>] tcp_transmit_skb+0x604/0x632 [<c025e80c>] tcp_retransmit_skb+0x4e2/0x5c7 [<c012e066>] hrtimer_run_queues+0x147/0x15f [<c0257960>] tcp_enter_loss+0x1a1/0x1fd [<c02608e3>] tcp_write_timer+0x0/0x5c9 [<c0260cdb>] tcp_write_timer+0x3f8/0x5c9 [<c0123440>] run_timer_softirq+0x101/0x15c [<c011f41e>] __do_softirq+0x5e/0xc3 [<c011f4bd>] do_softirq+0x3a/0x4a [<c0106131>] do_IRQ+0x48/0x53 [<c020c1cc>] evtchn_do_upcall+0x64/0x9b [<c0104a51>] hypervisor_callback+0x3d/0x48 [<c0107342>] raw_safe_halt+0x8c/0xaf [<c0102c5f>] xen_idle+0x22/0x2e [<c0102d7e>] cpu_idle+0x91/0xab [<c03236fc>] start_kernel+0x378/0x37f Code: 00 74 b6 83 e9 02 77 cd 74 16 83 c1 02 0f 84 9f 00 00 00 0f b6 1e 01 d8 83 d0 00 e9 92 00 00 00 66 03 06 83 d0 00 e9 87 00 00 00 <03> 46 80 13 46 84 13 46 88 13 46 8c 13 46 90 13 46 94 13 46 98 >>EIP; c01bb090 <csum_partial+88/120> <===== >>ebx; c01bb0f0 <csum_partial+e8/120> >>esp; c031fd54 <init_thread_union+1d54/2000> Trace; c022d626 <skb_checksum+112/27e> Trace; c022e532 <pskb_expand_head+ce/112> Trace; c0231f33 <skb_checksum_help+5d/ac> Trace; e93ef2ea <END_OF_CODE+2905a2ea/????> Trace; e93f8092 <END_OF_CODE+29063092/????> Trace; e93ef61e <END_OF_CODE+2905a61e/????> Trace; c024def0 <dst_output+0/7> Trace; c0246e28 <nf_iterate+30/61> Trace; c024def0 <dst_output+0/7> Trace; c0246f4e <nf_hook_slow+3a/90> Trace; c024def0 <dst_output+0/7> Trace; c02500e8 <ip_queue_xmit+35f/3b3> Trace; c024def0 <dst_output+0/7> Trace; c0115f49 <rebalance_tick+116/2ae> Trace; c025dab0 <tcp_transmit_skb+604/632> Trace; c025e80c <tcp_retransmit_skb+4e2/5c7> Trace; c012e066 <hrtimer_run_queues+147/15f> Trace; c0257960 <tcp_enter_loss+1a1/1fd> Trace; c02608e3 <tcp_write_timer+0/5c9> Trace; c0260cdb <tcp_write_timer+3f8/5c9> Trace; c0123440 <run_timer_softirq+101/15c> Trace; c011f41e <__do_softirq+5e/c3> Trace; c011f4bd <do_softirq+3a/4a> Trace; c0106131 <do_IRQ+48/53> Trace; c020c1cc <evtchn_do_upcall+64/9b> Trace; c0104a51 <hypervisor_callback+3d/48> Trace; c0107342 <raw_safe_halt+8c/af> Trace; c0102c5f <xen_idle+22/2e> Trace; c0102d7e <cpu_idle+91/ab> Trace; c03236fc <start_kernel+378/37f> This architecture has variable length instructions, decoding before eip is unreliable, take these instructions with a pinch of salt. Code; c01bb065 <csum_partial+5d/120> 00000000 <_EIP>: Code; c01bb065 <csum_partial+5d/120> 0: 00 74 b6 83 add %dh,0xffffff83(%esi,%esi,4) Code; c01bb069 <csum_partial+61/120> 4: e9 02 77 cd 74 jmp 74cd770b <_EIP+0x74cd770b> Code; c01bb06e <csum_partial+66/120> 9: 16 push %ss Code; c01bb06f <csum_partial+67/120> a: 83 c1 02 add $0x2,%ecx Code; c01bb072 <csum_partial+6a/120> d: 0f 84 9f 00 00 00 je b2 <_EIP+0xb2> Code; c01bb078 <csum_partial+70/120> 13: 0f b6 1e movzbl (%esi),%ebx Code; c01bb07b <csum_partial+73/120> 16: 01 d8 add %ebx,%eax Code; c01bb07d <csum_partial+75/120> 18: 83 d0 00 adc $0x0,%eax Code; c01bb080 <csum_partial+78/120> 1b: e9 92 00 00 00 jmp b2 <_EIP+0xb2> Code; c01bb085 <csum_partial+7d/120> 20: 66 03 06 add (%esi),%ax Code; c01bb088 <csum_partial+80/120> 23: 83 d0 00 adc $0x0,%eax Code; c01bb08b <csum_partial+83/120> 26: e9 87 00 00 00 jmp b2 <_EIP+0xb2> This decode from eip onwards should be reliable Code; c01bb090 <csum_partial+88/120> 00000000 <_EIP>: Code; c01bb090 <csum_partial+88/120> <===== 0: 03 46 80 add 0xffffff80(%esi),%eax <===== Code; c01bb093 <csum_partial+8b/120> 3: 13 46 84 adc 0xffffff84(%esi),%eax Code; c01bb096 <csum_partial+8e/120> 6: 13 46 88 adc 0xffffff88(%esi),%eax Code; c01bb099 <csum_partial+91/120> 9: 13 46 8c adc 0xffffff8c(%esi),%eax Code; c01bb09c <csum_partial+94/120> c: 13 46 90 adc 0xffffff90(%esi),%eax Code; c01bb09f <csum_partial+97/120> f: 13 46 94 adc 0xffffff94(%esi),%eax Code; c01bb0a2 <csum_partial+9a/120> 12: 13 46 98 adc 0xffffff98(%esi),%eax EIP: [<c01bb090>] csum_partial+0x88/0x120 SS:ESP 0069:c031fd54 <0>Kernel panic - not syncing: Fatal exception in interrupt [<c010cddd>] smp_call_function+0x53/0xf8 [<c011b4be>] printk+0x14/0x18 [<c010ce95>] smp_send_stop+0x13/0x1e [<c011aaeb>] panic+0x45/0xde [<c0105235>] die+0x242/0x276 [<c0111d05>] do_page_fault+0xa53/0xb76 [<c01bb0f0>] csum_partial+0xe8/0x120 [<c01112b2>] do_page_fault+0x0/0xb76 [<c0104a0f>] error_code+0x2b/0x30 [<c01bb0f0>] csum_partial+0xe8/0x120 [<c01bb090>] csum_partial+0x88/0x120 [<c022d626>] skb_checksum+0x112/0x27e [<c022e532>] pskb_expand_head+0xce/0x112 [<c0231f33>] skb_checksum_help+0x5d/0xac [<e93ef2ea>] ip_nat_fn+0x42/0x184 [iptable_nat] [<e93f8092>] ipt_local_hook+0x76/0xcc [iptable_mangle] [<e93ef61e>] ip_nat_local_fn+0x34/0xaa [iptable_nat] [<c024def0>] dst_output+0x0/0x7 [<c0246e28>] nf_iterate+0x30/0x61 [<c024def0>] dst_output+0x0/0x7 [<c0246f4e>] nf_hook_slow+0x3a/0x90 [<c024def0>] dst_output+0x0/0x7 [<c02500e8>] ip_queue_xmit+0x35f/0x3b3 [<c024def0>] dst_output+0x0/0x7 [<c0115f49>] rebalance_tick+0x116/0x2ae [<c025dab0>] tcp_transmit_skb+0x604/0x632 [<c025e80c>] tcp_retransmit_skb+0x4e2/0x5c7 [<c012e066>] hrtimer_run_queues+0x147/0x15f [<c0257960>] tcp_enter_loss+0x1a1/0x1fd [<c02608e3>] tcp_write_timer+0x0/0x5c9 [<c0260cdb>] tcp_write_timer+0x3f8/0x5c9 [<c0123440>] run_timer_softirq+0x101/0x15c [<c011f41e>] __do_softirq+0x5e/0xc3 [<c011f4bd>] do_softirq+0x3a/0x4a [<c0106131>] do_IRQ+0x48/0x53 [<c020c1cc>] evtchn_do_upcall+0x64/0x9b [<c0104a51>] hypervisor_callback+0x3d/0x48 [<c0107342>] raw_safe_halt+0x8c/0xaf [<c0102c5f>] xen_idle+0x22/0x2e [<c0102d7e>] cpu_idle+0x91/0xab [<c03236fc>] start_kernel+0x378/0x37f Warning (Oops_read): Code line not seen, dumping what data is available >>EIP; c01bb090 <csum_partial+88/120> <===== These are production boxes, so I should avoid rebooting them too often for debugging purposes. Any hints on how to approach and solve this issue would be highly appreciated. with best regards, Rene -- ------------------------------------------------- Gibraltar firewall http://www.gibraltar.at/ Attachment:
signature.asc _______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-users
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |