[MirageOS-devel] Xen, MirageOS, and Unikernel Migration

[Vittorio Cozzolino <vittorio.cozzolino@xxxxxxxxx>]

Hi,

I'm experimenting a bit with VM migration in XEN. Right now I'm trying 
to migrate a MirageOS Unikernel from one server to another without 
success. The image that I'm migrating is a customized version of the 
simple "hello" example that can be found in the mirage-skeleton repo. 
Both my server machines run Ubuntu 14.04 and have Xen 4.6, moreover, 
migration works for a normal VM image (like Alpine Linux).
Anyway, this is the trace generated by the migrate command:

Saving to migration stream new xl format (info 0x3/0x0/966)
libxl: debug: libxl.c:968:libxl_domain_suspend: ao 0x207cbb0: create: 
how=(nil) callback=(nil) poller=0x207b9e0
libxl: debug: libxl.c:6537:libxl__fd_flags_modify_save: fnctl F_GETFL 
flags for fd 8 are 0x1
libxl: debug: libxl.c:6545:libxl__fd_flags_modify_save: fnctl F_SETFL of 
fd 8 to 0x1
libxl: debug: libxl.c:995:libxl_domain_suspend: ao 0x207cbb0: 
inprogress: poller=0x207b9e0, flags=i
Loading new save file <incoming migration stream> (new xl fmt info 
0x3/0x0/966)
 Savefile contains xl domain config in JSON format
Parsing config from <saved>
libxl-save-helper: debug: starting save: Success
xc: detail: fd 8, dom 3, max_iters 0, max_factor 0, flags 1, hvm 0
xc: info: Saving domain 3, type x86 PV
xc: detail: 64 bits, 4 levels
xc: detail: max_pfn 0xffff, p2m_frames 128
xc: detail: max_mfn 0x2080000
libxl: debug: libxl_create.c:1557:do_domain_create: ao 0x2485260: 
create: how=(nil) callback=(nil) poller=0x2485300
libxl: debug: libxl.c:6537:libxl__fd_flags_modify_save: fnctl F_GETFL 
flags for fd 0 are 0x0
libxl: debug: libxl.c:6545:libxl__fd_flags_modify_save: fnctl F_SETFL of 
fd 0 to 0x0
libxl: debug: libxl_create.c:942:initiate_domain_create: restoring, not 
running bootloader
libxl: notice: libxl_numa.c:499:libxl__get_numa_candidate: NUMA 
placement failed, performance might be affected
libxl: debug: libxl_create.c:1580:do_domain_create: ao 0x2485260: 
inprogress: poller=0x2485300, flags=i
libxl: debug: libxl_stream_read.c:314:stream_header_done: Stream v2
libxl: debug: libxl_stream_read.c:529:process_record: Record: 1, length 0
libxl-save-helper: debug: starting restore: Success
xc: detail: fd 6, dom 4, hvm 0, pae 0, superpages 0, checkpointed_stream 0
xc: info: Found x86 PV domain from Xen 4.6
xc: info: Restoring domain
xc: detail: 64 bits, 4 levels
xc: detail: max_mfn 0x2080000
xc: detail: Changed max_pfn from 0 to 0xffff
libxl: debug: libxl_dom_suspend.c:144:domain_suspend_callback_common: 
issuing PV suspend request via XenBus control node
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch 
w=0x207c1d0 wpath=/local/domain/3/control/shutdown token=3/0: register 
slotnum=3
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x207c1d0 
wpath=/local/domain/3/control/shutdown token=3/0: event 
epath=/local/domain/3/control/shutdown
libxl: debug: libxl_aoutils.c:88:xswait_timeout_callback: guest 
acknowledgement of suspend request: xswait timeout 
(path=/local/domain/3/control/shutdown)
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch 
w=0x207c1d0 wpath=/local/domain/3/control/shutdown token=3/0: deregister 
slotnum=3
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
w=0x207c1d0: deregister unregistered
libxl: error: 
libxl_dom_suspend.c:215:domain_suspend_common_pvcontrol_suspending: 
guest didn't acknowledge suspend, cancelling request
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
w=0x207c1e8: deregister unregistered
xc: error: Domain has not been suspended: shutdown 0, reason 255: 
Internal error
xc: error: Save failed (0 = Success): Internal error
libxl-save-helper: debug: complete r=-1: Success
libxl: error: libxl_stream_write.c:329:libxl__xc_domain_save_done: 
saving domain: domain did not respond to suspend request: Success
libxl: debug: libxl.c:6564:libxl__fd_flags_restore: fnctl F_SETFL of fd 
8 to 0x1
libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x207cbb0: 
complete, rc=-8
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x207cbb0: destroy
migration sender: libxl_domain_suspend failed (rc=-8)
xc: error: Failed to read Record Header from stream (0 = Success): 
Internal error
xc: error: Restore failed (0 = Success): Internal error
libxl-save-helper: debug: complete r=-1: Success
libxl: error: libxl_stream_read.c:749:libxl__xc_domain_restore_done: 
restoring domain: Success
libxl: error: libxl_create.c:1142:domcreate_rebuild_done: cannot 
(re-)build domain: -3
libxl: debug: libxl.c:1719:devices_destroy_cb: forked pid 5584 for 
destroy of domain 4
libxl: debug: libxl.c:6564:libxl__fd_flags_restore: fnctl F_SETFL of fd 
0 to 0x0
libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x2485260: 
complete, rc=-3
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x2485260: destroy
libxl: debug: libxl.c:1458:libxl_domain_destroy: ao 0x2485260: create: 
how=(nil) callback=(nil) poller=0x2485300
libxl: error: libxl.c:1591:libxl__destroy_domid: non-existant domain 4
libxl: error: libxl.c:1549:domain_destroy_callback: unable to destroy 
guest with domid 4
libxl: error: libxl.c:1476:domain_destroy_cb: destruction of domain 4 failed
libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x2485260: 
complete, rc=-21
libxl: debug: libxl.c:1467:libxl_domain_destroy: ao 0x2485260: 
inprogress: poller=0x2485300, flags=ic
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x2485260: destroy
migration target: Domain creation failed (code -3).
xc: debug: hypercall buffer: total allocations:254 total releases:254
xc: debug: hypercall buffer: current allocations:0 maximum allocations:3
xc: debug: hypercall buffer: cache current size:3
xc: debug: hypercall buffer: cache hits:236 misses:3 toobig:15
libxl: info: libxl_exec.c:118:libxl_report_child_exitstatus: migration 
transport process [38632] exited with error status 3
Migration failed, failed to suspend at sender.
xc: debug: hypercall buffer: total allocations:29 total releases:29
xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
xc: debug: hypercall buffer: cache current size:2
xc: debug: hypercall buffer: cache hits:21 misses:2 toobig:6

I suppose the migration fails due to a problem with the shutdown 
command? I'm not an expert of XEN, I just started learning, so any 
remark/comment/suggestion is welcome.
Best regards,
Vittorio

Attachment: smime.p7s
Description: Firma crittografica S/MIME

_______________________________________________
MirageOS-devel mailing list
MirageOS-devel@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel