[Xen-users] Xen 4.6 - VM Migration Issue

[Vittorio Cozzolino <vittorio.cozzolino@xxxxxxxxx>]

I'm trying to migrate a VM from Server A to Server B. Both servers have as OS Ubuntu and Xen version 4.6 (same configuration, network is routed). If I migrate the VM with the following parameter:

vif = []

it works. If I enable the network on the VM:

vif = [xxx.xxx.xxx.xxx]

the migration fails and this is the log trace I get:

cozzolin@emu01:~$ sudo xl -vvvv migrate alpine 131.159.24.150
libxl: debug: libxl.c:6744:libxl_retrieve_domain_configuration: no vtpm from xenstore for domain 2
libxl: debug: libxl.c:6746:libxl_retrieve_domain_configuration: no pci from xenstore for domain 2
root@131.159.24.150's password:
migration target: Ready to receive domain.
Saving to migration stream new xl format (info 0x3/0x0/1739)
libxl: debug: libxl.c:968:libxl_domain_suspend: ao 0x1c10600: create: how=(nil) callback=(nil) poller=0x1c0e9e0
libxl: debug: libxl.c:6537:libxl__fd_flags_modify_save: fnctl F_GETFL flags for fd 8 are 0x1
libxl: debug: libxl.c:6545:libxl__fd_flags_modify_save: fnctl F_SETFL of fd 8 to 0x1
libxl: debug: libxl.c:995:libxl_domain_suspend: ao 0x1c10600: inprogress: poller=0x1c0e9e0, flags=i
Loading new save file  (new xl fmt info 0x3/0x0/1739)
 Savefile contains xl domain config in JSON format
Parsing config from
libxl-save-helper: debug: starting save: Success
xc: detail: fd 8, dom 2, max_iters 0, max_factor 0, flags 1, hvm 0
xc: info: Saving domain 2, type x86 PV
xc: detail: 64 bits, 4 levels
xc: detail: max_pfn 0x3ffff, p2m_frames 512
xc: detail: max_mfn 0x2080000
libxl: debug: libxl_create.c:1557:do_domain_create: ao 0x1520820: create: how=(nil) callback=(nil) poller=0x1522520
libxl: debug: libxl.c:6537:libxl__fd_flags_modify_save: fnctl F_GETFL flags for fd 0 are 0x0
libxl: debug: libxl.c:6545:libxl__fd_flags_modify_save: fnctl F_SETFL of fd 0 to 0x0
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=unknown
libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk vdev=xvda, using backend phy
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk vdev=xvdc spec.backend=unknown
libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk vdev=xvdc, using backend phy
libxl: debug: libxl_create.c:942:initiate_domain_create: restoring, not running bootloader
libxl: notice: libxl_numa.c:499:libxl__get_numa_candidate: NUMA placement failed, performance might be affected
libxl: debug: libxl_create.c:1580:do_domain_create: ao 0x1520820: inprogress: poller=0x1522520, flags=i
libxl: debug: libxl_stream_read.c:314:stream_header_done: Stream v2
libxl: debug: libxl_stream_read.c:529rocess_record: Record: 1, length 0
libxl-save-helper: debug: starting restore: Success
xc: detail: fd 6, dom 2, hvm 0, pae 0, superpages 0, checkpointed_stream 0
xc: info: Found x86 PV domain from Xen 4.6
xc: info: Restoring domain
xc: detail: 64 bits, 4 levels
xc: detail: max_mfn 0x2080000
xc: detail: Changed max_pfn from 0 to 0x3ffff
libxl: debug: libxl_dom_suspend.c:144:domain_suspend_callback_common: issuing PV suspend request via XenBus control node
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x1c0fea0 wpath=/local/domain/2/control/shutdown token=3/0: register slotnum=3
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1c0fea0 wpath=/local/domain/2/control/shutdown token=3/0: event epath=/local/domain/2/control/shutdown
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1c0fea0 wpath=/local/domain/2/control/shutdown token=3/0: event epath=/local/domain/2/control/shutdown
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch w=0x1c0fea0 wpath=/local/domain/2/control/shutdown token=3/0: deregister slotnum=3
libxl: debug: libxl_dom_suspend.c:226:domain_suspend_common_pvcontrol_suspending: guest acknowledged suspend request
libxl: debug: libxl_dom_suspend.c:245:domain_suspend_common_wait_guest: wait for the guest to suspend
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x1c0feb8 wpath=@releaseDomain token=3/1: register slotnum=3
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1c0feb8 wpath=@releaseDomain token=3/1: event epath=@releaseDomain
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1c0feb8 wpath=@releaseDomain token=3/1: event epath=@releaseDomain
libxl: debug: libxl_dom_suspend.c:304:suspend_common_wait_guest_check: guest has suspended
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch w=0x1c0feb8 wpath=@releaseDomain token=3/1: deregister slotnum=3
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1c0feb8: deregister unregistered
xc: info: Restore successful
libxl-save-helper: debug: complete r=0: Success
libxl: debug: libxl.c:6564:libxl__fd_flags_restore: fnctl F_SETFL of fd 8 to 0x1
libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x1c10600: complete, rc=0
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x1c10600: destroy
xc: info: XenStore: mfn 0x10dd330, dom 0, evt 1
xc: info: Console: mfn 0x119ef2, dom 0, evt 2
libxl-save-helper: debug: complete r=0: Success
libxl: debug: libxl_stream_read.c:529rocess_record: Record: 0, length 0
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=phy
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x1521400 wpath=/local/domain/0/backend/vbd/2/51712/state token=3/0: register slotnum=3
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk vdev=xvdc spec.backend=phy
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk vdev=xvdc spec.backend=phy
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x15243a0 wpath=/local/domain/0/backend/vbd/2/51744/state token=2/1: register slotnum=2
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1521400 wpath=/local/domain/0/backend/vbd/2/51712/state token=3/0: event epath=/local/domain/0/backend/vbd/2/51712/state
libxl: debug: libxl_event.c:880:devstate_callback: backend /local/domain/0/backend/vbd/2/51712/state wanted state 2 ok
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch w=0x1521400 wpath=/local/domain/0/backend/vbd/2/51712/state token=3/0: deregister slotnum=3
libxl: debug: libxl_device.c:937:device_backend_callback: calling device_backend_cleanup
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1521400: deregister unregistered
libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and environment ready
libxl: debug: libxl_device.c:1034:device_hotplug: calling hotplug script: /etc/xen/scripts/block add
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/block add
libxl: debug: libxl_event.c:545:watchfd_callback: watch epath=/local/domain/0/backend/vbd/2/51712/state token=3/0: empty slot
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x15243a0 wpath=/local/domain/0/backend/vbd/2/51744/state token=2/1: event epath=/local/domain/0/backend/vbd/2/51744/state
libxl: debug: libxl_event.c:880:devstate_callback: backend /local/domain/0/backend/vbd/2/51744/state wanted state 2 ok
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch w=0x15243a0 wpath=/local/domain/0/backend/vbd/2/51744/state token=2/1: deregister slotnum=2
libxl: debug: libxl_device.c:937:device_backend_callback: calling device_backend_cleanup
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x15243a0: deregister unregistered
libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and environment ready
libxl: debug: libxl_device.c:1034:device_hotplug: calling hotplug script: /etc/xen/scripts/block add
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/block add
libxl: debug: libxl_event.c:545:watchfd_callback: watch epath=/local/domain/0/backend/vbd/2/51744/state token=2/1: empty slot
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1521500: deregister unregistered
libxl: debug: libxl_linux.c:246:libxl__get_hotplug_script_info: num_exec 1, not running hotplug scripts
libxl: debug: libxl_device.c:1021:device_hotplug: No hotplug script to execute
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1521500: deregister unregistered
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x15244a0: deregister unregistered
libxl: debug: libxl_linux.c:246:libxl__get_hotplug_script_info: num_exec 1, not running hotplug scripts
libxl: debug: libxl_device.c:1021:device_hotplug: No hotplug script to execute
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x15244a0: deregister unregistered
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x1526fa0 wpath=/local/domain/0/backend/vif/2/0/state token=2/2: register slotnum=2
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1526fa0 wpath=/local/domain/0/backend/vif/2/0/state token=2/2: event epath=/local/domain/0/backend/vif/2/0/state
libxl: debug: libxl_event.c:884:devstate_callback: backend /local/domain/0/backend/vif/2/0/state wanted state 2 still waiting state 1
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1526fa0 wpath=/local/domain/0/backend/vif/2/0/state token=2/2: event epath=/local/domain/0/backend/vif/2/0/state
libxl: debug: libxl_event.c:880:devstate_callback: backend /local/domain/0/backend/vif/2/0/state wanted state 2 ok
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch w=0x1526fa0 wpath=/local/domain/0/backend/vif/2/0/state token=2/2: deregister slotnum=2
libxl: debug: libxl_device.c:937:device_backend_callback: calling device_backend_cleanup
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1526fa0: deregister unregistered
libxl: debug: libxl_device.c:1034:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-route online
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/vif-route online
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-route online [28689] exited with error status 1
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x15270a0: deregister unregistered
libxl: error: libxl_device.c:1084:device_hotplug_child_death_cb: script: /etc/xen/scripts/vif-route failed; error detected.
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x15270a0: deregister unregistered
libxl: error: libxl_create.c:1381:domcreate_attach_vtpms: unable to add nic devices
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x15255d0 wpath=/local/domain/0/backend/vbd/2/51712/state token=2/3: register slotnum=2
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x1525990 wpath=/local/domain/0/backend/vbd/2/51744/state token=3/4: register slotnum=3
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x1529d20 wpath=/local/domain/0/backend/vif/2/0/state token=1/5: register slotnum=1
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x15255d0 wpath=/local/domain/0/backend/vbd/2/51712/state token=2/3: event epath=/local/domain/0/backend/vbd/2/51712/state
libxl: debug: libxl_event.c:880:devstate_callback: backend /local/domain/0/backend/vbd/2/51712/state wanted state 6 ok
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch w=0x15255d0 wpath=/local/domain/0/backend/vbd/2/51712/state token=2/3: deregister slotnum=2
libxl: debug: libxl_device.c:937:device_backend_callback: calling device_backend_cleanup
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x15255d0: deregister unregistered
libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and environment ready
libxl: debug: libxl_device.c:1034:device_hotplug: calling hotplug script: /etc/xen/scripts/block remove
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/block remove
libxl: debug: libxl_event.c:545:watchfd_callback: watch epath=/local/domain/0/backend/vbd/2/51712/state token=2/3: empty slot
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1525990 wpath=/local/domain/0/backend/vbd/2/51744/state token=3/4: event epath=/local/domain/0/backend/vbd/2/51744/state
libxl: debug: libxl_event.c:880:devstate_callback: backend /local/domain/0/backend/vbd/2/51744/state wanted state 6 ok
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch w=0x1525990 wpath=/local/domain/0/backend/vbd/2/51744/state token=3/4: deregister slotnum=3
libxl: debug: libxl_device.c:937:device_backend_callback: calling device_backend_cleanup
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1525990: deregister unregistered
libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and environment ready
libxl: debug: libxl_device.c:1034:device_hotplug: calling hotplug script: /etc/xen/scripts/block remove
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/block remove
libxl: debug: libxl_event.c:545:watchfd_callback: watch epath=/local/domain/0/backend/vbd/2/51744/state token=3/4: empty slot
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1529d20 wpath=/local/domain/0/backend/vif/2/0/state token=1/5: event epath=/local/domain/0/backend/vif/2/0/state
libxl: debug: libxl_event.c:880:devstate_callback: backend /local/domain/0/backend/vif/2/0/state wanted state 6 ok
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch w=0x1529d20 wpath=/local/domain/0/backend/vif/2/0/state token=1/5: deregister slotnum=1
libxl: debug: libxl_device.c:937:device_backend_callback: calling device_backend_cleanup
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1529d20: deregister unregistered
libxl: debug: libxl_device.c:1034:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-route offline
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/vif-route offline
libxl: debug: libxl_event.c:545:watchfd_callback: watch epath=/local/domain/0/backend/vif/2/0/state token=1/5: empty slot
libxl: debug: libxl_event.c:545:watchfd_callback: watch epath=/local/domain/0/backend/vif/2/0/state token=1/5: empty slot
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x15256d0: deregister unregistered
libxl: debug: libxl_linux.c:246:libxl__get_hotplug_script_info: num_exec 1, not running hotplug scripts
libxl: debug: libxl_device.c:1021:device_hotplug: No hotplug script to execute
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x15256d0: deregister unregistered
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-route offline [28761] exited with error status 1
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1529e20: deregister unregistered
libxl: error: libxl_device.c:1084:device_hotplug_child_death_cb: script: /etc/xen/scripts/vif-route failed; error detected.
libxl: debug: libxl_device.c:1021:device_hotplug: No hotplug script to execute
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1529e20: deregister unregistered
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1525a90: deregister unregistered
libxl: debug: libxl_linux.c:246:libxl__get_hotplug_script_info: num_exec 1, not running hotplug scripts
libxl: debug: libxl_device.c:1021:device_hotplug: No hotplug script to execute
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1525a90: deregister unregistered
libxl: debug: libxl.c:1719:devices_destroy_cb: forked pid 28874 for destroy of domain 2
libxl: debug: libxl.c:6564:libxl__fd_flags_restore: fnctl F_SETFL of fd 0 to 0x0
libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x1520820: complete, rc=-3
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x1520820: destroy
libxl: debug: libxl.c:1458:libxl_domain_destroy: ao 0x1521650: create: how=(nil) callback=(nil) poller=0x1522520
libxl: error: libxl.c:1591:libxl__destroy_domid: non-existant domain 2
libxl: error: libxl.c:1549:domain_destroy_callback: unable to destroy guest with domid 2
libxl: error: libxl.c:1476:domain_destroy_cb: destruction of domain 2 failed
libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x1521650: complete, rc=-21
libxl: debug: libxl.c:1467:libxl_domain_destroy: ao 0x1521650: inprogress: poller=0x1522520, flags=ic
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x1521650: destroy
migration target: Domain creation failed (code -3).
xc: debug: hypercall buffer: total allocations:571 total releases:571
xc: debug: hypercall buffer: current allocations:0 maximum allocations:3
xc: debug: hypercall buffer: cache current size:3
xc: debug: hypercall buffer: cache hits:552 misses:3 toobig:16
libxl: error: libxl_utils.c:430:libxl_read_exactly: file/stream truncated reading ready message from migration receiver stream
libxl: info: libxl_exec.c:118:libxl_report_child_exitstatus: migration transport process [29185] exited with error status 3
Migration failed, resuming at sender.
libxl: debug: libxl.c:526:libxl_domain_resume: ao 0x1c0ef20: create: how=(nil) callback=(nil) poller=0x1c0e9e0
libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x1c0ef20: complete, rc=0
libxl: debug: libxl.c:529:libxl_domain_resume: ao 0x1c0ef20: inprogress: poller=0x1c0e9e0, flags=ic
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x1c0ef20: destroy
xc: debug: hypercall buffer: total allocations:42 total releases:42
xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
xc: debug: hypercall buffer: cache current size:2
xc: debug: hypercall buffer: cache hits:32 misses:2 toobig:8


It seems that something gets messed up during the migration or the network configuration at the destination server fails.

Best regards,
Vittorio

Attachment: smime.p7s
Description: Firma crittografica S/MIME

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users