Re: [Xen-devel] bug: slab corruption (net backend?)

[Gerd Knorr <kraxel@xxxxxxx>]

  Hi,

Maybe related:  I also see tcp connection stalls.  Just booting domain0
is enougth for that, whereas I see the slab corruption stuff only
after trying to boot some domU.

As the tcpdump below shows the xen machine sends the same packet over
and over again.  I'd guess some kind of memory corruption which kills
the packet checksum and makes eskarina drop the packet.  Maybe just
use-after-free as the slabdebug stuff will fill released memory blocks
with some pattern.

  Gerd

14:56:41.445799 IP eskarina.40106 > master-xen.ssh: S 4050091082:4050091082(0) 
win 5840 <mss 1460,sackOK,timestamp 2080736744 0,nop,wscale 2>
14:56:41.446106 IP master-xen.ssh > eskarina.40106: S 892717242:892717242(0) 
ack 4050091083 win 5792 <mss 1460,sackOK,timestamp 5230 2080736744,nop,wscale 2>
14:56:41.446129 IP eskarina.40106 > master-xen.ssh: . ack 1 win 1460 
<nop,nop,timestamp 2080736744 5230>
14:56:41.461316 IP master-xen.ssh > eskarina.40106: P 1:24(23) ack 1 win 1448 
<nop,nop,timestamp 5232 2080736744>
14:56:41.461720 IP eskarina.40106 > master-xen.ssh: . ack 24 win 1460 
<nop,nop,timestamp 2080736760 5232>
14:56:41.462250 IP eskarina.40106 > master-xen.ssh: P 1:23(22) ack 24 win 1460 
<nop,nop,timestamp 2080736760 5232>
14:56:41.462641 IP master-xen.ssh > eskarina.40106: . ack 23 win 1448 
<nop,nop,timestamp 5232 2080736760>
14:56:41.463196 IP eskarina.40106 > master-xen.ssh: P 23:663(640) ack 24 win 
1460 <nop,nop,timestamp 2080736761 5232>
14:56:41.463988 IP master-xen.ssh > eskarina.40106: . ack 663 win 1768 
<nop,nop,timestamp 5232 2080736761>
14:56:41.464835 IP master-xen.ssh > eskarina.40106: P 24:664(640) ack 663 win 
1768 <nop,nop,timestamp 5232 2080736761>
14:56:41.465183 IP eskarina.40106 > master-xen.ssh: P 663:687(24) ack 664 win 
1780 <nop,nop,timestamp 2080736763 5232>
14:56:41.469605 IP master-xen.ssh > eskarina.40106: P 664:816(152) ack 687 win 
1768 <nop,nop,timestamp 5233 2080736763>
14:56:41.485010 IP eskarina.40106 > master-xen.ssh: P 687:831(144) ack 816 win 
2100 <nop,nop,timestamp 2080736783 5233>
14:56:41.493626 IP master-xen.ssh > eskarina.40106: P 816:1280(464) ack 831 win 
1768 <nop,nop,timestamp 5235 2080736783>
14:56:41.495546 IP eskarina.40106 > master-xen.ssh: P 831:847(16) ack 1280 win 
2420 <nop,nop,timestamp 2080736793 5235>
14:56:41.529119 IP master-xen.ssh > eskarina.40106: . ack 847 win 1768 
<nop,nop,timestamp 5239 2080736793>
14:56:41.529143 IP eskarina.40106 > master-xen.ssh: P 847:895(48) ack 1280 win 
2420 <nop,nop,timestamp 2080736827 5239>
14:56:41.529488 IP master-xen.ssh > eskarina.40106: . ack 895 win 1768 
<nop,nop,timestamp 5239 2080736827>
14:56:41.529642 IP master-xen.ssh > eskarina.40106: P 1280:1328(48) ack 895 win 
1768 <nop,nop,timestamp 5239 2080736827>
14:56:41.530430 IP eskarina.40106 > master-xen.ssh: P 895:959(64) ack 1328 win 
2420 <nop,nop,timestamp 2080736828 5239>
14:56:41.534746 IP master-xen.ssh > eskarina.40106: P 1328:1392(64) ack 959 win 
1768 <nop,nop,timestamp 5239 2080736828>
14:56:41.534918 IP eskarina.40106 > master-xen.ssh: P 959:1199(240) ack 1392 
win 2420 <nop,nop,timestamp 2080736833 5239>
14:56:41.536297 IP master-xen.ssh > eskarina.40106: P 1392:1584(192) ack 1199 
win 2088 <nop,nop,timestamp 5239 2080736833>
14:56:41.538960 IP eskarina.40106 > master-xen.ssh: P 1199:1583(384) ack 1584 
win 2420 <nop,nop,timestamp 2080736837 5239>
14:56:41.541838 IP master-xen.ssh > eskarina.40106: P 1584:1616(32) ack 1583 
win 2408 <nop,nop,timestamp 5240 2080736837>
14:56:41.542436 IP eskarina.40106 > master-xen.ssh: P 1583:1647(64) ack 1616 
win 2420 <nop,nop,timestamp 2080736840 5240>
14:56:41.579115 IP master-xen.ssh > eskarina.40106: . ack 1647 win 2408 
<nop,nop,timestamp 5244 2080736840>
14:56:41.580513 IP master-xen.ssh > eskarina.40106: P 1616:1664(48) ack 1647 
win 2408 <nop,nop,timestamp 5244 2080736840>
14:56:41.581070 IP eskarina.40106 > master-xen.ssh: P 1647:2287(640) ack 1664 
win 2420 <nop,nop,timestamp 2080736879 5244>
14:56:41.581859 IP master-xen.ssh > eskarina.40106: . ack 2287 win 2728 
<nop,nop,timestamp 5244 2080736879>
14:56:41.613880 IP master-xen.ssh > eskarina.40106: P 1664:1712(48) ack 2287 
win 2728 <nop,nop,timestamp 5247 2080736879>
14:56:41.614052 IP master-xen.ssh > eskarina.40106: P 1712:1840(128) ack 2287 
win 2728 <nop,nop,timestamp 5247 2080736879>
14:56:41.653594 IP eskarina.40106 > master-xen.ssh: . ack 1840 win 2740 
<nop,nop,timestamp 2080736952 5247>
14:56:41.662633 IP master-xen.ssh > eskarina.40106: P 1840:1904(64) ack 2287 
win 2728 <nop,nop,timestamp 5252 2080736952>
14:56:41.662697 IP eskarina.40106 > master-xen.ssh: . ack 1904 win 2740 
<nop,nop,timestamp 2080736961 5252>
14:56:41.668714 IP master-xen.ssh > eskarina.40106: P 1904:1984(80) ack 2287 
win 2728 <nop,nop,timestamp 5252 2080736961>
14:56:41.668798 IP eskarina.40106 > master-xen.ssh: . ack 1984 win 2740 
<nop,nop,timestamp 2080736967 5252>
14:56:41.668836 IP master-xen.ssh > eskarina.40106: P 1984:2048(64) ack 2287 
win 2728 <nop,nop,timestamp 5252 2080736961>
14:56:41.668874 IP eskarina.40106 > master-xen.ssh: . ack 2048 win 2740 
<nop,nop,timestamp 2080736967 5252>
14:56:41.669064 IP master-xen.ssh > eskarina.40106: P 2048:2096(48) ack 2287 
win 2728 <nop,nop,timestamp 5252 2080736961>
14:56:41.669254 IP eskarina.40106 > master-xen.ssh: . ack 2096 win 2740 
<nop,nop,timestamp 2080736967 5252>
14:56:41.670240 IP master-xen.ssh > eskarina.40106: P 2096:2160(64) ack 2287 
win 2728 <nop,nop,timestamp 5253 2080736967>
14:56:41.671049 IP eskarina.40106 > master-xen.ssh: . ack 2160 win 2740 
<nop,nop,timestamp 2080736969 5253>
14:56:43.375106 IP eskarina.40106 > master-xen.ssh: P 2287:2335(48) ack 2160 
win 2740 <nop,nop,timestamp 2080738673 5253>
14:56:43.376098 IP master-xen.ssh > eskarina.40106: P 2160:2208(48) ack 2335 
win 2728 <nop,nop,timestamp 5423 2080738673>
14:56:43.376119 IP eskarina.40106 > master-xen.ssh: . ack 2208 win 2740 
<nop,nop,timestamp 2080738674 5423>
14:56:43.518162 IP eskarina.40106 > master-xen.ssh: P 2335:2383(48) ack 2208 
win 2740 <nop,nop,timestamp 2080738816 5423>
14:56:43.518924 IP master-xen.ssh > eskarina.40106: P 2208:2256(48) ack 2383 
win 2728 <nop,nop,timestamp 5437 2080738816>
14:56:43.518942 IP eskarina.40106 > master-xen.ssh: . ack 2256 win 2740 
<nop,nop,timestamp 2080738817 5437>
14:56:43.599847 IP eskarina.40106 > master-xen.ssh: P 2383:2431(48) ack 2256 
win 2740 <nop,nop,timestamp 2080738898 5437>
14:56:43.600591 IP master-xen.ssh > eskarina.40106: P 2256:2304(48) ack 2431 
win 2728 <nop,nop,timestamp 5446 2080738898>
14:56:43.600607 IP eskarina.40106 > master-xen.ssh: . ack 2304 win 2740 
<nop,nop,timestamp 2080738899 5446>
14:56:43.773975 IP eskarina.40106 > master-xen.ssh: P 2431:2479(48) ack 2304 
win 2740 <nop,nop,timestamp 2080739072 5446>
14:56:43.774815 IP master-xen.ssh > eskarina.40106: P 2304:2352(48) ack 2479 
win 2728 <nop,nop,timestamp 5463 2080739072>
14:56:43.774834 IP eskarina.40106 > master-xen.ssh: . ack 2352 win 2740 
<nop,nop,timestamp 2080739073 5463>
14:56:43.843676 IP eskarina.40106 > master-xen.ssh: P 2479:2527(48) ack 2352 
win 2740 <nop,nop,timestamp 2080739142 5463>
14:56:43.844417 IP master-xen.ssh > eskarina.40106: P 2352:2400(48) ack 2527 
win 2728 <nop,nop,timestamp 5470 2080739142>
14:56:43.844601 IP eskarina.40106 > master-xen.ssh: . ack 2400 win 2740 
<nop,nop,timestamp 2080739143 5470>
14:56:44.200677 IP eskarina.40106 > master-xen.ssh: P 2527:2575(48) ack 2400 
win 2740 <nop,nop,timestamp 2080739499 5470>
14:56:44.201595 IP master-xen.ssh > eskarina.40106: P 2400:2448(48) ack 2575 
win 2728 <nop,nop,timestamp 5506 2080739499>
14:56:44.201616 IP eskarina.40106 > master-xen.ssh: . ack 2448 win 2740 
<nop,nop,timestamp 2080739500 5506>
14:56:44.207227 IP master-xen.ssh > eskarina.40106: . 2448:3896(1448) ack 2575 
win 2728 <nop,nop,timestamp 5506 2080739500>
14:56:44.207255 IP eskarina.40106 > master-xen.ssh: . ack 3896 win 3464 
<nop,nop,timestamp 2080739506 5506>
14:56:44.208455 IP master-xen.ssh > eskarina.40106: . 3896:5344(1448) ack 2575 
win 2728 <nop,nop,timestamp 5506 2080739500>
14:56:44.208475 IP eskarina.40106 > master-xen.ssh: . ack 5344 win 4188 
<nop,nop,timestamp 2080739507 5506>
14:56:44.209923 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 
win 2728 <nop,nop,timestamp 5506 2080739506>
14:56:44.211154 IP master-xen.ssh > eskarina.40106: . 6792:8240(1448) ack 2575 
win 2728 <nop,nop,timestamp 5506 2080739506>
14:56:44.211174 IP eskarina.40106 > master-xen.ssh: . ack 5344 win 4188 
<nop,nop,timestamp 2080739510 5506,nop,nop,sack sack 1 {6792:8240} >
14:56:44.212383 IP master-xen.ssh > eskarina.40106: . 8240:9688(1448) ack 2575 
win 2728 <nop,nop,timestamp 5506 2080739507>
14:56:44.213790 IP master-xen.ssh > eskarina.40106: . 9688:11136(1448) ack 2575 
win 2728 <nop,nop,timestamp 5506 2080739507>
14:56:44.213798 IP eskarina.40106 > master-xen.ssh: . ack 5344 win 4188 
<nop,nop,timestamp 2080739512 5506,nop,nop,sack sack 2 {9688:11136}{6792:8240} >
14:56:44.215022 IP master-xen.ssh > eskarina.40106: . 11136:12584(1448) ack 
2575 win 2728 <nop,nop,timestamp 5507 2080739510>
14:56:44.215034 IP eskarina.40106 > master-xen.ssh: . ack 5344 win 4188 
<nop,nop,timestamp 2080739513 5506,nop,nop,sack sack 2 {9688:12584}{6792:8240} >
14:56:44.216438 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 
win 2728 <nop,nop,timestamp 5507 2080739512>
14:56:44.420361 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 
win 2728 <nop,nop,timestamp 5528 2080739513>
14:56:44.840346 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 
win 2728 <nop,nop,timestamp 5570 2080739513>
14:56:45.680357 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 
win 2728 <nop,nop,timestamp 5654 2080739513>
14:56:47.360382 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 
win 2728 <nop,nop,timestamp 5822 2080739513>
14:56:50.720720 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 
win 2728 <nop,nop,timestamp 6158 2080739513>
14:56:57.440528 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 
win 2728 <nop,nop,timestamp 6830 2080739513>
14:57:10.880688 IP master-xen.ssh > eskarina.40106: . 5344:6792(1448) ack 2575 
win 2728 <nop,nop,timestamp 8174 2080739513>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel