[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] 4.11.0 RC1 panic



On Sun, Jun 10, 2018 at 03:54:45AM -0600, Jan Beulich wrote:
> [...]
> 
> So I've been trying to look into this some more, and I've noticed an oddity in
> the raw stack dump you had provided with the first report. Unfortunately you
> didn't include that part for either of the above (the first one as being on a
> different path would be of particular interest).
> Additionally, to be able to check
> whether the (type_info) values on the stack really point at some anomaly, I'd
> need the xen-syms (or xen.efi) from that same build of Xen. That'll allow me
> to determine whether the values are simply leftovers from prior function
> invocations. Otherwise, if they're live values, it'd be suspicious for
> free_page_type() to be called on a page the type refcount of which is still 2.
> 
> As I assume that you don't have recorded/stored the additional bits I'm asking
> for, I do realize that this means that unfortunately you'll have to obtain 
> the data
> another time. I'm sorry for that.
> 

Actually I have them: I have complete logs of the serial console,
and I still have the built directory.

This one is from a newer build (4.11.0rc4). 
I've put the binary files at ftp://asim.lip6.fr/outgoing/bouyer/xen-debug/

(XEN) Xen version 4.11-rcnb0 (bouyer@) (gcc (nb2 20150115) 4.8.5) debug=y  Tue M
ay 15 17:21:40 MEST 2018
[...]
(XEN) Assertion 'oc > 0' failed at mm.c:681
(XEN) ----[ Xen-4.11-rcnb0  x86_64  debug=y   Not tainted ]----
(XEN) CPU:    4
(XEN) RIP:    e008:[<ffff82d080284bd2>] mm.c#dec_linear_entries+0x12/0x20
(XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor (d0v0)
(XEN) rax: ffffffffffff0000   rbx: 4400000000000001   rcx: 00000000001a3200
(XEN) rdx: 0400000000000000   rsi: 000000000000002e   rdi: ffff82e003465040
(XEN) rbp: ffff82e003464000   rsp: ffff8301bf127bb0   r8:  0000000000000000
(XEN) r9:  0000000000000200   r10: 4000000000000000   r11: ffff82e003465040
(XEN) r12: ffff82e003465040   r13: 0000000000000000   r14: 10ffffffffffffff
(XEN) r15: 1000000000000000   cr0: 000000008005003b   cr4: 0000000000002660
(XEN) cr3: 00000001b9096000   cr2: 00007f7ff60ce790
(XEN) fsb: 00007f7ff7ff36c0   gsb: ffffffff80cc8500   gss: 0000000000000000
(XEN) ds: 003f   es: 003f   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen code around <ffff82d080284bd2> (mm.c#dec_linear_entries+0x12/0x20):
(XEN)  c1 47 1e 66 85 c0 7f 02 <0f> 0b c3 66 66 2e 0f 1f 84 00 00 00 00 00 41 54
(XEN) Xen stack trace from rsp=ffff8301bf127bb0:
(XEN)    ffff82d08028922e 000000000023a00d ffff8301bf127fff ffff82d08023a00d
(XEN)    ffff82e003464000 ffff82e003465040 ffff8301b98b6000 0000000000000001
(XEN)    ffff820040030000 0200000000000000 ffff82d0802898af 00000000000001fc
(XEN)    ffff82e003465040 ffff82d080288c59 00000000001a3282 0000001500000000
(XEN)    ffff8301bf127fff 4400000000000001 ffff82e003465040 0000000000000000
(XEN)    00ffffffffffffff 10ffffffffffffff 1000000000000000 ffff82d0802891f7
(XEN)    000000010122471f ffff8301bf127fff ffff8301bf127d10 ffff82e003465040
(XEN)    ffff8301bf127d10 ffff8301b98b6028 ffff8301b98b6000 ffff8301b98b6020
(XEN)    ffff82e003465050 ffff82d0802898ef ffff82d080272adb 0000000000000068
(XEN)    e400000000000000 ffff8301bf127fff 8000000000000000 ffff8301b98b6000
(XEN)    0000000000000000 ffff8301b98b6018 deadbeefdeadf00d 0000000000000001
(XEN)    00007f7ff7b1b004 ffff82d080276ae3 ffff8301b98b6000 00007f7ff7b1b004
(XEN)    ffff82d0802068bd ffff8301b98b6000 00007f7ff7b1b004 0000000000000000
(XEN)    ffff82d0802039e3 ffff82d080457640 ffff8301bf0e3fa0 0000000000000002
(XEN)    00000000000000f0 0000000000000000 0000000000000000 0000000000000000
(XEN)    000000000000000f 0000000000000000 0000001000000002 00007f7ff7c00016
(XEN)    00007f7ff7ffe400 0000000000000016 00007f7fffffd510 00007f7ff74c97d1
(XEN)    00007f7ff7c02d7d 0000000000000246 0000000000000000 0000000000000000
(XEN)    0000000000000000 00007f7ff7b10800 0000000000000016 0000000000000016
(XEN)    0000000000000000 00007f7ff7b10800 0000000000000202 00007f7ff7ffa800
(XEN) Xen call trace:
(XEN)    [<ffff82d080284bd2>] mm.c#dec_linear_entries+0x12/0x20
(XEN)    [<ffff82d08028922e>] mm.c#_put_page_type+0x13e/0x350
(XEN)    [<ffff82d08023a00d>] _spin_lock+0xd/0x50
(XEN)    [<ffff82d0802898af>] mm.c#put_page_from_l2e+0xdf/0x110
(XEN)    [<ffff82d080288c59>] free_page_type+0x2f9/0x790
(XEN)    [<ffff82d0802891f7>] mm.c#_put_page_type+0x107/0x350
(XEN)    [<ffff82d0802898ef>] put_page_type_preemptible+0xf/0x10
(XEN)    [<ffff82d080272adb>] domain.c#relinquish_memory+0xab/0x460
(XEN)    [<ffff82d080276ae3>] domain_relinquish_resources+0x203/0x290
(XEN)    [<ffff82d0802068bd>] domain_kill+0xbd/0x150
(XEN)    [<ffff82d0802039e3>] do_domctl+0x7d3/0x1a90
(XEN)    [<ffff82d080203210>] do_domctl+0/0x1a90
(XEN)    [<ffff82d080367b95>] pv_hypercall+0x1f5/0x430
(XEN)    [<ffff82d08036e422>] lstar_enter+0xa2/0x120
(XEN)    [<ffff82d08036e42e>] lstar_enter+0xae/0x120
(XEN)    [<ffff82d08036e422>] lstar_enter+0xa2/0x120
(XEN)    [<ffff82d08036e42e>] lstar_enter+0xae/0x120
(XEN)    [<ffff82d08036e422>] lstar_enter+0xa2/0x120
(XEN)    [<ffff82d08036e42e>] lstar_enter+0xae/0x120
(XEN)    [<ffff82d08036e48c>] lstar_enter+0x10c/0x120


The second one is from the same build:
(XEN) Xen version 4.11-rcnb0 (bouyer@) (gcc (nb2 20150115) 4.8.5) debug=y  Tue M
ay 15 17:21:40 MEST 2018
[...]
(XEN) Assertion 'oc > 0' failed at mm.c:681
(XEN) ----[ Xen-4.11-rcnb0  x86_64  debug=y   Not tainted ]----
(XEN) CPU:    3
(XEN) RIP:    e008:[<ffff82d080284bd2>] mm.c#dec_linear_entries+0x12/0x20
(XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor (d2v0)
(XEN) rax: ffffffffffff0000   rbx: 4400000000000001   rcx: 00000000001a7d29
(XEN) rdx: 0400000000000000   rsi: 0000000000000011   rdi: ffff82e0034f5540
(XEN) rbp: ffff82e0034fa520   rsp: ffff8301bf13fc08   r8:  0000000000000000
(XEN) r9:  0000000000000200   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: ffff82e0034f5540   r13: 0000000000000000   r14: 10ffffffffffffff
(XEN) r15: 1000000000000000   cr0: 000000008005003b   cr4: 00000000000026e4
(XEN) cr3: 00000001b98b6000   cr2: 00000000bb418010
(XEN) fsb: 00000000c0636dc0   gsb: 0000000000000000   gss: 0000000000000000
(XEN) ds: 0011   es: 0011   fs: 0031   gs: 0011   ss: 0000   cs: e008
(XEN) Xen code around <ffff82d080284bd2> (mm.c#dec_linear_entries+0x12/0x20):
(XEN)  c1 47 1e 66 85 c0 7f 02 <0f> 0b c3 66 66 2e 0f 1f 84 00 00 00 00 00 41 54
(XEN) Xen stack trace from rsp=ffff8301bf13fc08:
(XEN)    ffff82d08028922e 0000000000800063 ffff8301bf13ffff 4c00000000000002
(XEN)    ffff82e0034fa520 ffff82e0034f5540 ffff8301b98f5000 0000000000000001
(XEN)    ffff820040018000 0200000000000000 ffff82d0802898af 00000000000001fd
(XEN)    ffff82e0034f5540 ffff82d080288c59 00000000001a7aaa 0000000000000000
(XEN)    ffff8301bf13ffff 4400000000000001 ffff82e0034f5540 0000000000000000
(XEN)    00ffffffffffffff 10ffffffffffffff 1000000000000000 ffff82d0802891f7
(XEN)    0000000101000206 ffff8301bf13ffff 4400000000000002 00000000001a7aaa
(XEN)    ffff82e0034f5540 0000000000000000 ffff8301b98f5000 ffff820080000000
(XEN)    0000000000000000 ffff82d0802898ef ffff82d080290b6d ffff8300bff40000
(XEN)    00000001802a8302 ffff8301b98f5000 0000000000000000 ffff8301b98f5000
(XEN)    0000000000000007 ffff8300bff40000 00007ff000000000 0000000000000000
(XEN)    ffff8301b98f5000 ffff82e00350bf80 ffff82d0804b0058 ffff82d0804b0060
(XEN)    00000000001a85fc 00000000001a85fc 0000000100000004 00000000001a7aaa
(XEN)    00000000001aaaa7 ffff820080000018 0000000000000001 00000000cdb0db7c
(XEN)    ffff82d080387b90 0000000000000001 ffff8301bf13ffff ffff82d080295630
(XEN)    ffff8301bf13fe14 00000001bf13ffff ffff820080000000 0000000000000000
(XEN)    00007ff000000000 000000048036bbe8 cdb0dbb0001a7aaa ffff8301bf13fef8
(XEN)    ffff8300bff40000 00000000000001a0 00000000deadf00d 0000000000000004
(XEN)    00000000deadf00d ffff82d080367d4a ffff82d000007ff0 ffff82d000000000
(XEN)    ffff82d000000001 ffff82d0cdb0db70 ffff82d08036bbf4 ffff82d08036bbe8
(XEN) Xen call trace:
(XEN) Xen call trace:
(XEN)    [<ffff82d080284bd2>] mm.c#dec_linear_entries+0x12/0x20
(XEN)    [<ffff82d08028922e>] mm.c#_put_page_type+0x13e/0x350
(XEN)    [<ffff82d0802898af>] mm.c#put_page_from_l2e+0xdf/0x110
(XEN)    [<ffff82d080288c59>] free_page_type+0x2f9/0x790
(XEN)    [<ffff82d0802891f7>] mm.c#_put_page_type+0x107/0x350
(XEN)    [<ffff82d0802898ef>] put_page_type_preemptible+0xf/0x10
(XEN)    [<ffff82d080290b6d>] do_mmuext_op+0x73d/0x1810
(XEN)    [<ffff82d080295630>] compat_mmuext_op+0x430/0x450
(XEN)    [<ffff82d080367d4a>] pv_hypercall+0x3aa/0x430
(XEN)    [<ffff82d08036bbf4>] entry_int82+0x74/0xc0
(XEN)    [<ffff82d08036bbe8>] entry_int82+0x68/0xc0
(XEN)    [<ffff82d08036bbf4>] entry_int82+0x74/0xc0
(XEN)    [<ffff82d08036bbe8>] entry_int82+0x68/0xc0
(XEN)    [<ffff82d08036bbf4>] entry_int82+0x74/0xc0
(XEN)    [<ffff82d08036bbe8>] entry_int82+0x68/0xc0
(XEN)    [<ffff82d08036bbf4>] entry_int82+0x74/0xc0
(XEN)    [<ffff82d08036957e>] do_entry_int82+0x1e/0x20
(XEN)    [<ffff82d08036bc31>] entry_int82+0xb1/0xc0


> 
> Two other questions on the internals of NetBSD page table management: Are all
> updates to a given (set of) page table(s) fully serialized, e.g. via a 
> respective
> spin lock?

In the general case they're done using atomic operations (pmap_pte_cas()).
For xenPV, the update is protected by a global lock, so page table updates
are serialized (globally).

> Could you additionally point me at that code, or give an outline of
> how the (un)pinning of L2 tables in the 32-bit case works?

When a new set of page tables is needed (this is pmap_create()), a pdp is
requested from a cache.  If the cache is empty, pages are allocated in
pmap_pdp_ctor(), which is going to also pin the L2 pages.
When the page table is not needed any more (this is pmap_destroy()),
the pdp is returned to the cache.  L2 pages remains pinned, with pointers to
the kernel L1 pages.  If memory needs to be reclaimed from the cache,
or is an explicit call to pool_cache_destruct_object() is done, 
the L2 pages are unpinned, but they are not explicitely zeroed out before
(can this be a problem ?).

The code for this is in
http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/x86/x86/pmap.c

Some helper functions are in
http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/xen/x86/xen_pmap.c
Some #defines and inline are in
http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/xen/include/xenpmap.h
http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/i386/include/pmap.h

-- 
Manuel Bouyer <bouyer@xxxxxxxxxxxxxxx>
     NetBSD: 26 ans d'experience feront toujours la difference
--

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.