[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-devel] 4.10.1 Xen crash and reboot
Hi, Up front information: Today one of my Xen hosts crashed with this logging on the serial: (XEN) ----[ Xen-4.10.1 x86_64 debug=n Not tainted ]---- (XEN) CPU: 15 (XEN) RIP: e008:[<ffff82d08033db45>] guest_4.o#shadow_set_l1e+0x75/0x6a0 (XEN) RFLAGS: 0000000000010246 CONTEXT: hypervisor (d31v1) (XEN) rax: ffff82e01ecfae80 rbx: 0000000f67d74025 rcx: 0000000000000000 (XEN) rdx: ffff82e000000000 rsi: ffff81bfd79f12d8 rdi: 00000000ffffffff (XEN) rbp: 0000000000f67d74 rsp: ffff83202628fbd8 r8: 00000000010175c6 (XEN) r9: 0000000000000000 r10: ffff830079592000 r11: 0000000000000000 (XEN) r12: 0000000f67d74025 r13: ffff832020549000 r14: 0000000000f67d74 (XEN) r15: ffff81bfd79f12d8 cr0: 0000000080050033 cr4: 0000000000372660 (XEN) cr3: 0000001fd5b8d001 cr2: ffff81bfd79f12d8 (XEN) fsb: 00007faf3e71f700 gsb: 0000000000000000 gss: ffff88007f300000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008 (XEN) Xen code around <ffff82d08033db45> (guest_4.o#shadow_set_l1e+0x75/0x6a0): (XEN) 0f 20 0f 85 23 01 00 00 <4d> 8b 37 4c 39 f3 0f 84 97 01 00 00 49 89 da 89 (XEN) Xen stack trace from rsp=ffff83202628fbd8: (XEN) 0000000f67d74000 00000000010175c6 0000000000000000 ffff832000000002 (XEN) ffff830079592000 ffff832020549000 ffff81bfd79f12d8 ffff83202628fef8 (XEN) 00000000010175c6 0000000000f67d74 ffff830079592000 ffff82d08033fc82 (XEN) 8000000fad0dc125 00007faf3e25bba0 ffff832020549600 0000000000f67d74 (XEN) 0000000000f67d74 0000000000f67d74 ffff83202628fd70 ffff83202628fd20 (XEN) 00000007faf3e25b 00000000000000c0 ffff82d0805802c0 0000000220549000 (XEN) 00000000000007f8 00000000000005e0 0000000000000f88 ffff82d0805802c0 (XEN) 00000000010175c6 00007faf3e25bba0 00000000000002d8 000000000000005b (XEN) ffff81c0dfebcf88 01ff82d000000000 0000000f67d74025 ffff82d000000000 (XEN) ffff832020549000 000000010000000d ffff83202628ffff ffff83202628fd20 (XEN) 00000000000000e9 00007faf3e25bba0 0000000f472df067 0000000f49296067 (XEN) 0000000f499f1067 0000000f67d74125 0000000000f498cf 0000000000f472df (XEN) 0000000000f49296 0000000000f499f1 0000000000000015 ffffffffffffffff (XEN) ffff82e03fab71a0 ffff830079593000 ffff82d0803557eb ffff82d08020bf4a (XEN) 0000000000000000 ffff830079592000 ffff832020549000 ffff83202628fef8 (XEN) 0000000000000002 ffff82d08034e9b0 0000000000633400 ffff82d08034a330 (XEN) ffff830079592000 ffff83202628ffff ffff830079592000 ffff82d08034eaae (XEN) ffff82d080355913 ffff82d080355907 ffff82d080355913 ffff82d080355907 (XEN) ffff82d080355913 ffff82d080355907 ffff82d080355913 ffff82d080355907 (XEN) ffff82d080355913 ffff82d080355907 ffff82d080355913 ffff82d080355907 (XEN) Xen call trace: (XEN) [<ffff82d08033db45>] guest_4.o#shadow_set_l1e+0x75/0x6a0 (XEN) [<ffff82d08033fc82>] guest_4.o#sh_page_fault__guest_4+0x8f2/0x2060 (XEN) [<ffff82d0803557eb>] common_interrupt+0x9b/0x120 (XEN) [<ffff82d08020bf4a>] evtchn_check_pollers+0x1a/0xb0 (XEN) [<ffff82d08034e9b0>] do_iret+0/0x1a0 (XEN) [<ffff82d08034a330>] toggle_guest_pt+0x30/0x160 (XEN) [<ffff82d08034eaae>] do_iret+0xfe/0x1a0 (XEN) [<ffff82d080355913>] handle_exception+0x9b/0xf9 (XEN) [<ffff82d080355907>] handle_exception+0x8f/0xf9 (XEN) [<ffff82d080355913>] handle_exception+0x9b/0xf9 (XEN) [<ffff82d080355907>] handle_exception+0x8f/0xf9 (XEN) [<ffff82d080355913>] handle_exception+0x9b/0xf9 (XEN) [<ffff82d080355907>] handle_exception+0x8f/0xf9 (XEN) [<ffff82d080355913>] handle_exception+0x9b/0xf9 (XEN) [<ffff82d080355907>] handle_exception+0x8f/0xf9 (XEN) [<ffff82d080355913>] handle_exception+0x9b/0xf9 (XEN) [<ffff82d080355907>] handle_exception+0x8f/0xf9 (XEN) [<ffff82d080355913>] handle_exception+0x9b/0xf9 (XEN) [<ffff82d080355907>] handle_exception+0x8f/0xf9 (XEN) [<ffff82d080355913>] handle_exception+0x9b/0xf9 (XEN) [<ffff82d0802a16b2>] do_page_fault+0x1a2/0x4e0 (XEN) [<ffff82d080355913>] handle_exception+0x9b/0xf9 (XEN) [<ffff82d080355907>] handle_exception+0x8f/0xf9 (XEN) [<ffff82d080355913>] handle_exception+0x9b/0xf9 (XEN) [<ffff82d080355907>] handle_exception+0x8f/0xf9 (XEN) [<ffff82d080355913>] handle_exception+0x9b/0xf9 (XEN) [<ffff82d0803559d9>] x86_64/entry.S#handle_exception_saved+0x68/0x94 (XEN) (XEN) Pagetable walk from ffff81bfd79f12d8: (XEN) L4[0x103] = 8000001fd5b8d063 ffffffffffffffff (XEN) L3[0x0ff] = 0000000000000000 ffffffffffffffff (XEN) (XEN) Reboot in five seconds... (XEN) Resetting with ACPI MEMORY or I/O RESET_REG. The same host also crashed about 2 weeks ago but I had nothing in place to record the serial console so I have no logs. There has also been one other host crash on a different host but again no information collected. Longer background: Around the weekend of 18 November I deployed a hypervisor built from staging-4.10 plus the outstanding XSA patches including XSA-273 which I had up until then held off on. As described in: https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg02811.html within a few days I began noticing sporadic memory corruption issues in some guests, we established there was a bug in the L1TF fixes, and I was able to avoid the problem in affected guests by making sure to upgrade their guest kernels so they have Linux's L1TF fixes. During first reboot into that hypervisor one of my hosts crashed and rebooted, but it went by too fast for me to get any information and there wasn't enough scrollback on the serial console. Since then, a different host has crashed and rebooted twice. The first time I have managed to log it is above. I don't think it's a hardware fault, or at least if it is it is only being tickled by something added recently. I have absolutely no idea it is the case but I can't help feeling it's going to be related to L1TF again. Do my logs above help at all? Is it worth me trying to work out what d31 was at the time and taking a closer look at that? Production system, problem that occurs weeks apart… could be a bit tricky to get to the bottom of. The host is a Debian jessie dom0 running kernel version linux-image-3.16.0-7-amd64 3.16.59-1. The hardware is a single socket Xeon D-1540. The xl info is: host : hobgoblin release : 3.16.0-7-amd64 version : #1 SMP Debian 3.16.59-1 (2018-10-03) machine : x86_64 nr_cpus : 16 max_cpu_id : 15 nr_nodes : 1 cores_per_socket : 8 threads_per_core : 2 cpu_mhz : 2000 hw_caps : bfebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100 virt_caps : hvm hvm_directio total_memory : 130969 free_memory : 4646 sharing_freed_memory : 0 sharing_used_memory : 0 outstanding_claims : 0 free_cpus : 0 xen_major : 4 xen_minor : 10 xen_extra : .1 xen_version : 4.10.1 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : fe50b33b07fd447949-x86: write to correct variable in parse_pv_l xen_commandline : placeholder dom0_mem=2048M dom0_max_vcpus=2 com1=115200,8n1,0x2f8,10 console=com1,vga ucode=scan serial_tx_buffer=256k cc_compiler : gcc (Debian 4.9.2-10+deb8u1) 4.9.2 cc_compile_by : andy cc_compile_domain : prymar56.org cc_compile_date : Wed Nov 7 16:52:19 UTC 2018 build_id : 091f7ab43ab0b6ef9208a2e593c35496517fbe91 xend_config_format : 4 Are there any other hypervisor command line options that would be beneficial to set for next time? Unfortunately unless we are very sure to get somewhere, or I can isolate a guest that is triggering this and put it on test hardware, I don't really want to keep rebooting this system. But I can set something so it boots into it next time. Thanks, Andy _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxxxxxxxxx https://lists.xenproject.org/mailman/listinfo/xen-devel
|
![]() |
Lists.xenproject.org is hosted with RackSpace, monitoring our |