|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] refcount errors then crash on XenoLinux with the latest source
It happened again. Is it possible that Xen isn't disabling network
interrupts while it is "auditing all pages"?
-Kip
Killing domain 1
Releasing task 1
DOM0: INIT: Id "x1" respawning too fast: disabled for 5 minutes
DOM0: INIT: Id "x1" respawning too fast: disabled for 5 minutes
DOM0: INIT: Id "x1" respawning too fast: disabled for 5 minutes
audit_all_pages
refcount error: pfn=000247 cf=00000001 refcount=0
audit page: pfn=247 info: cf=1 tf=f0000001 ts=0 dom=fc648040
refcount error: pfn=00024d cf=00000001 refcount=0
audit page: pfn=24d info: cf=1 tf=f0000001 ts=0 dom=fc648040
refcount error: pfn=00036f cf=40000002 refcount=1
audit page: pfn=36f info: cf=40000002 tf=f0000002 ts=0 dom=fc648be0
pte_pfn=002207 cf=10000023 tf=30000021 dom=fc648be0
pte_idx=3f9 *pte_idx=0036f063
refcount error: pfn=000371 cf=40000002 refcount=1
audit page: pfn=371 info: cf=40000002 tf=f0000002 ts=0 dom=fc648be0
pte_pfn=002207 cf=10000023 tf=30000021 dom=fc648be0
pte_idx=3fe *pte_idx=00371063
refcount error: pfn=000372 cf=40000002 refcount=1
audit page: pfn=372 info: cf=40000002 tf=f0000002 ts=0 dom=fc648be0
pte_pfn=002207 cf=10000023 tf=30000021 dom=fc648be0
pte_idx=3fd *pte_idx=00372063
CPU: 1
EIP: 0808:[<fc532ddf>]
EFLAGS: 00010206
eax: 06081012 ebx: 00000010 ecx: fc657560 edx: fc650da0
esi: fc650da0 edi: fc657540 ebp: 00000000 esp: fc64fd70
ds: 0810 es: 0810 fs: 0810 gs: 0810 ss: 0810
Stack trace from ESP=fc64fd70:
ff803012 00000008 [fc51302a] fc76bb40 0000003c fc657540 fc657540
[fc5328c2]
fc657540 fc657400 00000017 ffffffff 00000017 fc648040 00000040
0000003e
fc657400 fc76bb40 00000040 fc76a740 04000001 fc657540 00005048
[fc5323a8]
fc657540 fc5ebc80 fc5d1f1c 00000001 00000046 00000004 [fc509639]
fc76a740
04000001 fc64fe60 00000010 [fc5b1f2d] 00000010 fc657400 fc64fe60
3d6e6670
33303030 63203237 00000001 fc76a740 fc600bc0 00000010 fc64fe60
[fc5b20f3]
00000010 fc64fe60 fc76a740 00002207 00372063 34429e8f 00000001
0007fff0
0007fff0 00000002 00000004 [fc5af970] 0007fff0 fd800000 00000001
0007fff0
00000002 00000004 00000040 00010810 00000810 00000810 fc500810
ffffff10
[fc50d0e5] 00000808 00000202 0000004d fc64ff6c fc64ff6c
[fc509f22] 0000004d
00000000 fc64ff6c [fc5162b6] 00000003 00000040 fc5ebc80
[fc517c4e] 0000004d
fc64ff6c fc64ff6c [fc512529] 00000003 fc6501e0 02000001
[fc517ca4] fc5ebc80
fc64ff6c 0092578a 00000000 fc651200 00000006 00000006 [fc5b1f2d]
00000004
fc5ebc80 fc64ff6c [fc5b151c] 00000004 00000001 00000001 fc6501e0
fc6008c0
00000004 fc64ff6c [fc5b20f3] 00000004 fc64ff6c fc6501e0
[fc511e2e] fc624494
431ea128 00000001 00000040 fc648040 00000040 fc649780 [fc5af970]
00000040
00000001 00000040 fc648040 00000040 fc649780 00000040 fc640810
fc640810
00000810 fc640810 ffffff04 [fc5b5e04] 00000808 00000246
[fc5b5e40] fc648040
004c4b40 ffffffff 655f6464 7972746e 5f636100 656d6974 61007372
69745f63
5f72656d 74666f73 5f717269 69746361 64006e6f 5f706d75 656d6974
62007172
On Mon, 23 Feb 2004, Kip Macy wrote:
>
> >
> > This is a Xen crash dump. ksymoops won't help -- you'll need to map
> > the crash dump to Xen code by hand. It doesn't take long. The
> > addresses in the stack trace that are enclosed in square brackets are
> > likely to be return addresses in the function-call trace.
>
> This is sufficiently tedious that if this happens again I'm going to
> either run screaming or write a ksymoops for xen.
>
> >
> > 'objdump -d xen >xen.s'. Then you can search in xen.s with a text
> > editor to find the call-trace addresses.
>
> I did this and got what you see below. It looks like to backtraces
> interleaved. All of the values in brackets are legitimate return
> addresses (they immediately follow a call instruction). "function addr"
> is the address of the function itself and "ret addr" is the address
> taken from the oops.
>
> function function addr ret addr
> ================================================
> putchar fc5095be fc5095ef
> e100_rx_srv fc532048 fc53240a
> printf fc5095f7 fc509664
> putchar_serial fc50927c fc509299
> e100intr fc531d8f fc531ef0
> handle_IRQ_event fc5b1a25 fc5b1a7d
> do_IRQ fc5b1bbb fc5b1c43
> call_do_IRQ fc5af4bb fc5af4c0
> serial_rx_int fc51801d fc518078
> serial_rx_int fc51801d fc518046
> handle_IRQ_event fc5b1a25 fc5b1a7d
> reprogram_ac_timer fc5af087 fc5af0aa
> do_IRQ fc5b1bbb fc5b1c43
> ac_timer_softirq_action fc50455c fc50465b
> call_do_IRQ fc5af4bb fc5af4c0
> default_idle fc5b585c fc5b582e
> continue_cpu_idle_loop fc5b585f fc5b5898
>
>
> The fault instruction is this:
> fc532927: 66 83 38 00 cmpw $0x0,(%eax)
> It is in e100_start_ru. Obviously eax is pointing at some piece of
> unmapped memory. I'm not sufficiently versed in assembler, particularly
> optimized, to tell where in we are going wrong:
>
>
> list_for_each(entry_ptr, &(bdp->active_rx_list)) {
> rx_struct =
> list_entry(entry_ptr, struct rx_list_elem, list_elem);
> pci_dma_sync_single(bdp->pdev, rx_struct->dma_addr,
> bdp->rfd_size, PCI_DMA_FROMDEVICE);
> if (!((SKB_RFD_STATUS(rx_struct->skb, bdp) &
> __constant_cpu_to_le16(RFD_STATUS_COMPLETE)))) {
> buffer_found = 1;
> break;
> }
> }
>
> Could the list have been corrupted?
>
>
> -Kip
>
>
> >
> > -- Keir
> >
> > > After a few more minutes the following popped out on the console:
> > >
> > > CPU: 1
> > > EIP: 0808:[<fc532927>]
> > > EFLAGS: 00010206
> > > eax: 0a725012 ebx: 00000010 ecx: fc657560 edx: fc76a460
> > > esi: fc76a460 edi: fc657540 ebp: 00000000 esp: fc64fda0
> > > ds: 0810 es: 0810 fs: 0810 gs: 0810 ss: 0810
> > > Stack trace from ESP=fc64fda0:
> > > ff865012 0000000a [fc5095ef] fc780140 0000003c fc657540 fc657540
> > > [fc53240a]
> > > fc657540 fc657400 [fc509664] 0000000a [fc509299] fc648040 00000040
> > > 0000003e
> > > fc657400 fc780140 00000040 fc76a740 04000001 fc657540 00005048
> > > [fc531ef0]
> > > fc657540 00000046 [fc509664] 0000000a 30303030 74203130 00000046
> > > fc76a740
> > > 04000001 fc64fe90 00000010 [fc5b1a7d] 00000010 fc657400 fc64fe90
> > > 3d6e6670
> > > 33303030 63203462 00000001 fc76a740 fc600200 00000010 fc64fe90
> > > [fc5b1c43]
> > > 00000010 fc64fe90 fc76a740 0007fff0 000003b4 25c4fe2d 00000001
> > > 0007fff0
> > > 0007fff0 00000000 00000000 [fc5af4c0] 0007fff0 fd800000 00000001
> > > 0007fff0
> > > 00000000 00000000 00000040 00010810 00000810 00000810 fc500810
> > > ffffff10
> > > [fc50cff5] 00000808 00000202 fc654d4d 0000004d fc64ff6c [fc518078]
> > > 0000004d
> > > 00000000 fc64ff6c [fc518046] 0036bfec 00000000 00000292 fc654d00
> > > 02000001
> > > fc64ff6c 00000004 [fc5b1a7d] 00000004 00000000 fc64ff6c [fc5af0aa]
> > > fc650200
> > > 00000086 00000001 fc654d00 fc5fff00 00000004 fc64ff6c [fc5b1c43]
> > > 00000004
> > > fc64ff6c fc654d00 [fc50465b] 35c9c161 50d04d38 00000001 00000040
> > > fc648040
> > > 00000040 fc7b8080 [fc5af4c0] 00000040 00000028 00000040 fc648040
> > > 00000040
> > > fc7b8080 00000040 fc640810 fc640810 00000810 fc7b0810 ffffff04
> > > [fc5b585c]
> > > 00000808 00000246 [fc5b5898] fc648040 004c4b40 ffffffff 61007372
> > > 69745f63
> > > 5f72656d 74666f73 5f717269 69746361 64006e6f 5f706d75 656d6974
> > > 62007172
> > > 636f6c72 00632e6b 736e6f63 2e656c6f 65640063 2e677562 65640063
> > > fc648040
> > >
> > > ****************************************
> > > CPU1 FATAL PAGE FAULT
> > > [error_code=00000000]
> > > Faulting linear address might be 0a725012
> > > Aieee! CPU1 is toast...
> > > ****************************************
> > >
> > > Is this oops from Xen or from XenoLinux? I downloaded the latest
> > > ksymoops and did the following:
> > > kmacy@xentap ./ksymoops -v ../xenolinux-2.4.25/vmlinux -m
> > > ../xenolinux-2.4.25/System.map <
> > > ../xeno-unstable.bk.home/tools/xc/lib/crash1.txt
> > > ksymoops 2.4.9 on i686 2.4.25-xeno. Options used
> > > -v ../xenolinux-2.4.25/vmlinux (specified)
> > > -k /proc/ksyms (default)
> > > -l /proc/modules (default)
> > > -o /lib/modules/2.4.25-xeno/ (default)
> > > -m ../xenolinux-2.4.25/System.map (specified)
> > >
> > > No modules in ksyms, skipping objects
> > > Warning (read_lsmod): no symbols in lsmod, is /proc/modules a valid
> > > lsmod file?
> > > Warning (compare_maps): mismatch on symbol state d, System.map says
> > > c0175ca8, vmlinux says 0. Ignoring System.map entry
> > > Warning (compare_maps): mismatch on symbol state a, vmlinux says 0,
> > > System.map says c0175ca8. Ignoring System.map entry
> > > CPU: 1
> > > EIP: 0808:[<fc532927>]
> > > Using defaults from ksymoopsSegmentation fault
> > >
> > >
> > > -Kip
> > >
> > > On Mon, 23 Feb 2004, Kip Macy wrote:
> > >
> > > > I had just tested my domain builder for the nth time on xeno-unstable
> > > > (very latest source), when I saw the messages below on the console.
> > > > DOM0 no longer responds to ping - I'm hoping that it will recover,
> > > > however, in all likelihood I will be hitting the rpb in a few minutes.
> > > >
> > > > audit_all_pages
> > > > zombie: pfn=00000000 cf=fffffffd tf=fffffffd dom=00000000
> > > > refcount error: pfn=000000 cf=fffffffd refcount=1
> > > > audit page: pfn=0 info: cf=fffffffd tf=fffffffd ts=0 dom=0
> > > >
> > > > refcount error: pfn=000247 cf=00000001 refcount=0
> > > > audit page: pfn=247 info: cf=1 tf=f0000001 ts=0 dom=fc648040
> > > >
> > > > refcount error: pfn=00024d cf=00000001 refcount=0
> > > > audit page: pfn=24d info: cf=1 tf=f0000001 ts=0 dom=fc648040
> > > >
> > > > refcount error: pfn=00036f cf=40000002 refcount=1
> > > > audit page: pfn=36f info: cf=40000002 tf=f0000002 ts=0 dom=fc648be0
> > > > pte_pfn=002207 cf=10000026 tf=30000024 dom=fc648be0
> > > > pte_idx=3f9 *pte_idx=0036f063
> > > >
> > > > refcount error: pfn=000371 cf=40000002 refcount=1
> > > > audit page: pfn=371 info: cf=40000002 tf=f0000002 ts=0 dom=fc648be0
> > > > pte_pfn=002207 cf=10000026 tf=30000024 dom=fc648be0
> > > > pte_idx=3fe *pte_idx=00371063
> > > >
> > > > refcount error: pfn=000372 cf=40000002 refcount=1
> > > > audit page: pfn=372 info: cf=40000002 tf=f0000002 ts=0 dom=fc648be0
> > > > pte_pfn=002207 cf=10000026 tf=30000024 dom=fc648be0
> > > > pte_idx=3fd *pte_idx=00372063
> > > >
> > > > refcount error: pfn=000390 cf=00000001 refcount=0
> > > > audit page: pfn=390 info: cf=1 tf=f0000001 ts=4ddca dom=fc649780
> > > >
> > > > refcount error: pfn=000392 cf=00000001 refcount=0
> > > > audit page: pfn=392 info: cf=1 tf=f0000001 ts=4ddca dom=fc649780
> > > >
> > > > refcount error: pfn=000393 cf=00000001 refcount=0
> > > > audit page: pfn=393 info: cf=1 tf=f0000001 ts=4ae4c dom=fc64a320
> > > >
> > > > refcount error: pfn=000395 cf=00000001 refcount=0
> > > > audit page: pfn=395 info: cf=1 tf=f0000001 ts=0 dom=fc64a320
> > > >
> > > > refcount error: pfn=00039f cf=00000001 refcount=0
> > > > audit page: pfn=39f info: cf=1 tf=f0000001 ts=0 dom=fc64aec0
> > > >
> > > > refcount error: pfn=0003a1 cf=00000001 refcount=0
> > > > audit page: pfn=3a1 info: cf=1 tf=f0000001 ts=0 dom=fc64aec0
> > > >
> > > > refcount error: pfn=0003a2 cf=00000001 refcount=0
> > > > audit page: pfn=3a2 info: cf=1 tf=f0000001 ts=0 dom=fc7a4060
> > > >
> > > > refcount error: pfn=0003a8 cf=00000001 refcount=0
> > > > audit page: pfn=3a8 info: cf=1 tf=f0000001 ts=0 dom=fc7a4060
> > > >
> > > > refcount error: pfn=0003a9 cf=00000001 refcount=0
> > > > audit page: pfn=3a9 info: cf=1 tf=f0000001 ts=0 dom=fc7a4c00
> > > >
> > > > refcount error: pfn=0003ab cf=00000001 refcount=0
> > > > audit page: pfn=3ab info: cf=1 tf=f0000001 ts=0 dom=fc7a4c00
> > > >
> > > > refcount error: pfn=0003ac cf=00000001 refcount=0
> > > > audit page: pfn=3ac info: cf=1 tf=f0000001 ts=191ab3 dom=fc7a57a0
> > > >
> > > > refcount error: pfn=0003ae cf=00000001 refcount=0
> > > > audit page: pfn=3ae info: cf=1 tf=f0000001 ts=191ab3 dom=fc7a57a0
> > > >
> > > > refcount error: pfn=0003af cf=00000001 refcount=0
> > > > audit page: pfn=3af info: cf=1 tf=f0000001 ts=191ab2 dom=fc7a6340
> > > >
> > > > refcount error: pfn=0003b1 cf=00000001 refcount=0
> > > > audit page: pfn=3b1 info: cf=1 tf=f0000001 ts=0 dom=fc7a6340
> > > >
> > > > refcount error: pfn=0003b2 cf=00000001 refcount=0
> > > > audit page: pfn=3b2 info: cf=1 tf=f0000001 ts=0 dom=fc7a6ee0
> > > >
> > > > refcount error: pfn=0003b4 cf=00000001 refcount=0
> > > > audit page: pfn=3b4 info: cf=1 tf=f0000001 ts=0 dom=fc7a6ee0
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > -------------------------------------------------------
> > > > SF.Net is sponsored by: Speed Start Your Linux Apps Now.
> > > > Build and deploy apps & Web services for Linux with
> > > > a free DVD software kit from IBM. Click Now!
> > > > http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
> > > > _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@xxxxxxxxxxxxxxxxxxxxx
> > > > https://lists.sourceforge.net/lists/listinfo/xen-devel
> > > >
> > >
> > >
> > > -------------------------------------------------------
> > > SF.Net is sponsored by: Speed Start Your Linux Apps Now.
> > > Build and deploy apps & Web services for Linux with
> > > a free DVD software kit from IBM. Click Now!
> > > http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@xxxxxxxxxxxxxxxxxxxxx
> > > https://lists.sourceforge.net/lists/listinfo/xen-devel
> >
> >
> >
> > -------------------------------------------------------
> > SF.Net is sponsored by: Speed Start Your Linux Apps Now.
> > Build and deploy apps & Web services for Linux with
> > a free DVD software kit from IBM. Click Now!
> > http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxxxxxxxxxx
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
> >
>
>
> -------------------------------------------------------
> SF.Net is sponsored by: Speed Start Your Linux Apps Now.
> Build and deploy apps & Web services for Linux with
> a free DVD software kit from IBM. Click Now!
> http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxxxx
> https://lists.sourceforge.net/lists/listinfo/xen-devel
>
-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.sourceforge.net/lists/listinfo/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |