Re: [Xen-devel] Xen crash with mem-sharing and cloning

[Andres Lagar Cavilla <andres@xxxxxxxxxxxxxxxx>]

On Mon, Mar 23, 2015 at 9:10 AM, Tamas K Lengyel <tklengyel@xxxxxxxxxxxxx> wrote:

Hello everyone,

I'm trying to chase down a bug that reproducibly crashes Xen (tested with 4.4.1). The problem is somewhere within the mem-sharing subsystem and how that interacts with domains that are being actively saved. In my setup I use the xl toolstack to rapidly create clones of HVM domains by piping "xl save -c" into xl restore with a modified domain config which updates the name/disk/vif. However, during such an operation Xen crashes with the following log if there are already active clones.

IMHO there should be no conflict between saving the domain and memsharing, as long as the domain is actually just being checkpointed "-c" - it's memory should remain as is. This is however clearly not the case. Any ideas?

Tamas, I'm not clear on the use of memsharing in this workflow. As described, you pipe save into restore, but the internal magic is lost on me. Are you fanning out to multiple restores? That would seem to be the case, given the need to update name/disk/vif.

Anyway, I'm inferring. Instead, could you elaborate?

Thanks

Andres

Â

Thanks,

Tamas

----------------------------------------------------------
(XEN) HVM1 save: CPU
(XEN) HVM1 save: PIC
(XEN) HVM1 save: IOAPIC
(XEN) HVM1 save: LAPIC
(XEN) HVM1 save: LAPIC_REGS
(XEN) HVM1 save: PCI_IRQ
(XEN) HVM1 save: ISA_IRQ
(XEN) HVM1 save: PCI_LINK
(XEN) HVM1 save: PIT
(XEN) HVM1 save: RTC
(XEN) HVM1 save: HPET
(XEN) HVM1 save: PMTIMER
(XEN) HVM1 save: MTRR
(XEN) HVM1 save: VIRIDIAN_DOMAIN
(XEN) HVM1 save: CPU_XSAVE
(XEN) HVM1 save: VIRIDIAN_VCPU
(XEN) HVM1 save: VMCE_VCPU
(XEN) HVM1 save: TSC_ADJUST
(XEN) HVM19 restore: CPU 0
(XEN) HVM19 restore: PIC 0
(XEN) HVM19 restore: PIC 1
(XEN) HVM19 restore: IOAPIC 0
(XEN) HVM19 restore: LAPIC 0
(XEN) HVM19 restore: LAPIC_REGS 0
(XEN) HVM19 restore: PCI_IRQ 0
(XEN) HVM19 restore: ISA_IRQ 0
(XEN) HVM19 restore: PCI_LINK 0
(XEN) HVM19 restore: PIT 0
(XEN) HVM19 restore: RTC 0
(XEN) HVM19 restore: HPET 0
(XEN) HVM19 restore: PMTIMER 0
(XEN) HVM19 restore: MTRR 0
(XEN) HVM19 restore: CPU_XSAVE 0
(XEN) HVM19 restore: VMCE_VCPU 0
(XEN) HVM19 restore: TSC_ADJUST 0
(XEN) ----[ Xen-4.4.1 x86_64 debug=n Not tainted ]----
(XEN) CPU:ÂÂÂ 2
(XEN) RIP:ÂÂÂ e008:[<ffff82d0801f2beb>] __mem_sharing_unshare_page+0x1ab/0xb10
(XEN) RFLAGS: 0000000000010283ÂÂ CONTEXT: hypervisor
(XEN) rax: 0000000000000000ÂÂ rbx: ffff83021fa92000ÂÂ rcx: 000000000000000d
(XEN) rdx: ffff8302ea96cb90ÂÂ rsi: 0000000000000001ÂÂ rdi: 00000000002acd06
(XEN) rbp: 0000000000006306ÂÂ rsp: ffff83042fdffca8ÂÂ r8:Â 007fffffffffffff
(XEN) r9:Â 0400000000000000ÂÂ r10: 0080000000000001ÂÂ r11: 0000000000000002
(XEN) r12: 0000000000000000ÂÂ r13: ffff82e00559a0c0ÂÂ r14: 00000000002acd06
(XEN) r15: 0000000000000000ÂÂ cr0: 0000000080050033ÂÂ cr4: 00000000000426f0
(XEN) cr3: 0000000253b45000ÂÂ cr2: 0000000000000000
(XEN) ds: 0000ÂÂ es: 0000ÂÂ fs: 0000ÂÂ gs: 0000ÂÂ ss: e010ÂÂ cs: e008
(XEN) Xen stack trace from rsp=ffff83042fdffca8:
(XEN)ÂÂÂ ffff830400000001 ffff83031010e810 ffff83042fdf8000 ffff82d08030110c
(XEN)ÂÂÂ 00000033b2f75d80 00000000002acd06 0000000100000000 0000000000000000
(XEN)ÂÂÂ ffff830227b19000 000000070000000c 0000000000000000 0000000000000003
(XEN)ÂÂÂ ffff83031010e810 0000000000006306 ffff83042fdffec0 ffff83042fdffdb0
(XEN)ÂÂÂ 0000000000000000 ffff82d0801e45b3 ffff82d080301108 ffff83042fdf8000
(XEN)ÂÂÂ 0000000000000000 ffff83042fdffec0 ffff83031010e810 ffff83042fdffdb0
(XEN)ÂÂÂ 0000000000006306 0000000000000002 ffff83021fa92000 ffff82d0801e483b
(XEN)ÂÂÂ ffff830200000001 ffff82d080173f5d 000000fd000000ff 00000000002acd06
(XEN)ÂÂÂ 0000000000000100 ffff830400000007 ffff83042fdb77e0 0000000000000007
(XEN)ÂÂÂ ffff83027f86d1a8 0000000000000000 0000000000000000 000000000027f86d
(XEN)ÂÂÂ ffff880056313bd8 ffff82d08017e190 000000000000000a 0000000000000000
(XEN)ÂÂÂ 0000000000000000 ffff82d08017bc23 ffff82d08012aa4b ffff830400000002
(XEN)ÂÂÂ ffff83021fa92000 ffff83042fdaf000 ffff8300aecea000 000000012fdaf0b0
(XEN)ÂÂÂ ffff83042fdf8000 ffff82e004ff0da0 ffff83042fdaf000 0000000000000007
(XEN)ÂÂÂ ffff83042fdf8000 8000000006306627 ffff83027f86d000 ffff8300aecea000
(XEN)ÂÂÂ 000000000027f86d 0000000d00000003 0000000000000000 0000000000006306
(XEN)ÂÂÂ 0000000000253b45 0000000000000000 0000000000000206 00007f8f0000000c
(XEN)ÂÂÂ 000000027f86d1a8 8000000006306627 0000000000000033 ffff8300aecea000
(XEN)ÂÂÂ 0000000000000001 00007f8f84a35000 0000000000000001 ffff880056313bd8
(XEN)ÂÂÂ ffff880061468d48 ffff82d080220019 00000091c9d82304 0000000000000003
(XEN) Xen call trace:
(XEN)ÂÂÂ [<ffff82d0801f2beb>] __mem_sharing_unshare_page+0x1ab/0xb10
(XEN)ÂÂÂ [<ffff82d0801e45b3>] __get_gfn_type_access+0xd3/0x200
(XEN)ÂÂÂ [<ffff82d0801e483b>] get_page_from_gfn_p2m+0xfb/0x2a0
(XEN)ÂÂÂ [<ffff82d080173f5d>] get_page+0x2d/0x100
(XEN)ÂÂÂ [<ffff82d08017e190>] do_mmu_update+0x1080/0x1bb0
(XEN)ÂÂÂ [<ffff82d08017bc23>] do_mmuext_op+0x803/0x1940
(XEN)ÂÂÂ [<ffff82d08012aa4b>] add_entry+0x4b/0xb0
(XEN)ÂÂÂ [<ffff82d080220019>] syscall_enter+0xa9/0xae
(XEN)
(XEN) Pagetable walk from 0000000000000000:
(XEN)Â L4[0x000] = 0000000000000000 ffffffffffffffff
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 2:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=0000]
(XEN) Faulting linear address: 0000000000000000
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...
The system is powered off.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel