Re: [Xen-devel] bug in _shadow_prealloc during migration of PV domU

[Juergen Gross <jgross@xxxxxxxx>]

On 12/04/18 10:01, Juergen Gross wrote:
> On 11/04/18 22:32, Olaf Hering wrote:
>> I was testing 'virsh migrate domU host' and did some libvirtd debugging
>> on 'host'. This means the migration was attempted a few times, but did
>> not actually start because libvirtd was in gdb. Not sure if libvirt on
>> the sender does anything with the domU before a connection to the remote
>> host is fully established.
>>
>> Finally I installed the fixed libvirtd on 'host' and started the
>> migration again. This time the sender died like this:
> Unfortunately I can reproduce that easily.
> 
> Unfortunately because this happens with my XPTI series after the first
> failed migration when I try to migrate again.
> 
> I guess this is related to some missing cleanup when suspending the
> guest failed (e.g. due to a timeout).

Here is more data:

The first migrate attempt failed with:



# xl migrate 1 localhost
Password:
migration target: Ready to receive domain.
Saving to migration stream new xl format (info 0x3/0x0/1218)
Loading new save file <incoming migration stream> (new xl fmt info
0x3/0x0/1218)
 Savefile contains xl domain config in JSON format
Parsing config from <saved>
xc: info: Saving domain 1, type x86 PV
xc: info: Found x86 PV domain from Xen 4.11
xc: info: Restoring domain
libxl: error:
libxl_dom_suspend.c:262:domain_suspend_common_pvcontrol_suspending:
Domain 1:guest didn't acknowledge suspend, cancelling request
xc: error: Domain has not been suspended: shutdown 0, reason 255:
Internal error
xc: error: Save failed (0 = Success): Internal error
libxl: error: libxl_stream_write.c:350:libxl__xc_domain_save_done:
Domain 1:saving domain: domain did not respond to suspend request: Success
migration sender: libxl_domain_suspend failed (rc=-8)
xc: error: Failed to read Record Header from stream (0 = Success):
Internal error
xc: error: Restore failed (0 = Success): Internal error
libxl: error: libxl_stream_read.c:850:libxl__xc_domain_restore_done:
restoring domain: Success
libxl: error: libxl_create.c:1265:domcreate_rebuild_done: Domain
2:cannot (re-)build domain: -3
libxl: error: libxl_domain.c:1034:libxl__destroy_domid: Domain
2:Non-existant domain
libxl: error: libxl_domain.c:993:domain_destroy_callback: Domain
2:Unable to destroy guest
libxl: error: libxl_domain.c:920:domain_destroy_cb: Domain 2:Destruction
of domain failed
migration target: Domain creation failed (code -3).
libxl: info: libxl_exec.c:118:libxl_report_child_exitstatus: migration
transport process [2393] exited with error status 1
Migration failed, failed to suspend at sender.


The second attempt immediately produced:

(XEN) sh error: _shadow_prealloc(): Can't pre-allocate 1 shadow pages!
(XEN)   shadow pages total = 6, free = 0, p2m=0
(XEN) Xen BUG at common.c:1315
(XEN) ----[ Xen-4.11-unstable  x86_64  debug=y   Tainted:  C   ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82d08033f8ff>]
common.c#_shadow_prealloc+0x5b1/0x638
(XEN) RFLAGS: 0000000000010292   CONTEXT: hypervisor (d0v2)
(XEN) rax: 0000000000000200   rbx: ffff83021e125000   rcx: 0000000000000000
(XEN) rdx: ffff8300dba87fff   rsi: 000000000000000a   rdi: ffff82d0804766b8
(XEN) rbp: ffff8300dba87af8   rsp: ffff8300dba87aa8   r8:  ffff830217f78000
(XEN) r9:  0000000000000001   r10: 0000000000000000   r11: 0000000000000001
(XEN) r12: 0000000000000020   r13: 0000000000000000   r14: ffff82d0805bfff8
(XEN) r15: ffff8300dba87fff   cr0: 0000000080050033   cr4: 00000000001526e0
(XEN) cr3: 00000000d0747000   cr2: 00007f231d31f272
(XEN) fsb: 00007f231d93f700   gsb: ffff88020f700000   gss: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen code around <ffff82d08033f8ff>
(common.c#_shadow_prealloc+0x5b1/0x638):
(XEN)  00 00 00 e8 f4 29 f1 ff <0f> 0b f6 40 10 40 0f 85 2d fb ff ff e9
21 fb ff
(XEN) Xen stack trace from rsp=ffff8300dba87aa8:
(XEN)    ffff8300dba87ac0 000000011e125000 0000000000000000 0000000000000000
(XEN)    0000000000000000 ffff8300db7fe000 ffff83021e125000 0000000000000002
(XEN)    ffff82d0803e77c0 0000000000000195 ffff8300dba87b08 ffff82d08033f9a6
(XEN)    ffff8300dba87b48 ffff82d08034325b 00007f231d960004 ffff8300db7fe000
(XEN)    0000000000001000 0000000000000000 00007f231d960004 ffff830217f3b000
(XEN)    ffff8300dba87b68 ffff82d0803435a2 ffff8300dba87b78 ffff83021e125000
(XEN)    ffff8300dba87b88 ffff82d08034391a ffff83021e125000 ffff83021e125650
(XEN)    ffff8300dba87bb8 ffff82d080343a33 0000000000000001 ffff83021e125000
(XEN)    0000000000000001 0000000000000000 ffff8300dba87bd8 ffff82d080321f47
(XEN)    ffff83021e125000 ffff8300dba87d98 ffff8300dba87c68 ffff82d080322943
(XEN)    ffff8300dba87c98 ffff82d08022df69 80100000c06e5067 ffff8300dba87fff
(XEN)    ffff8300d8dfe000 0000000000000001 ffff830217f3b000 0000000400000025
(XEN)    0000000000000008 0000000000000000 0000000000000000 ffff8300dba87d88
(XEN)    ffff83021e125000 00007f231d960004 00007f231d960004 ffff830217f3b000
(XEN)    ffff8300dba87d28 ffff82d080272a4c ffff8300dba87d08 ffff82d080290269
(XEN)    0000000000000206 00000000000c0733 0000000000000000 ffff8300dba87cd0
(XEN)    0000000000000000 ffff830217f3b000 ffff8200400088f8 0000000000000000
(XEN)    ffff8300dba87d08 0000000000000000 0000000000000009 0000000000000206
(XEN)    ffff820040008000 0000000000000000 0000000000000292 ffff83021e125000
(XEN)    0000000000000000 00007f231d960004 0000000000000000 deadbeefdeadf00d
(XEN) Xen call trace:
(XEN)    [<ffff82d08033f8ff>] common.c#_shadow_prealloc+0x5b1/0x638
(XEN)    [<ffff82d08033f9a6>] shadow_prealloc+0x20/0x22
(XEN)    [<ffff82d08034325b>] common.c#sh_update_paging_modes+0xf5/0x3de
(XEN)    [<ffff82d0803435a2>] common.c#sh_new_mode+0x5e/0x6e
(XEN)    [<ffff82d08034391a>] common.c#shadow_one_bit_enable+0xd3/0xf2
(XEN)    [<ffff82d080343a33>] common.c#sh_enable_log_dirty+0xfa/0x14d
(XEN)    [<ffff82d080321f47>] paging_log_dirty_enable+0x47/0x61
(XEN)    [<ffff82d080322943>] paging_domctl+0x1cc/0xaca
(XEN)    [<ffff82d080272a4c>] arch_do_domctl+0x219/0x2648
(XEN)    [<ffff82d080206cb4>] do_domctl+0x1872/0x1bce
(XEN)    [<ffff82d08036c2aa>] pv_hypercall+0x1f4/0x43e
(XEN)    [<ffff82d0803734a5>] lstar_enter+0x115/0x120
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Xen BUG at common.c:1315
(XEN) ****************************************


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel