[Xen-users] Live migration fails (ERROR Restore failed)

[Hitesh Khandelwal <hitesh.iitk@xxxxxxxxx>]

I compiled xen-4.0.1 (kernel 2.6.32.27) on Ubuntu 10.10 as Dom0. Ubuntu 10.10 with stock kernel is used as DomU. I tried live migration by following instructions on http://www.virtuatopia.com/index.php/Migrating_Xen_domainU_Guests_Between_Host_Systems , but it failed with the following logs.

Both the machines have same configuration and same amount of memory, and there is plenty of memory available on target host. VM guest runs successfully on both the source and target hosts individually. Relevant bug report: http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1653

Thanks,

Hitesh

Log on source Dom0:

[2011-01-31 14:20:24 1008] DEBUG (XendCheckpoint:124) [xc_save]: /usr/lib/xen/bin/xc_save 46 10 0 0 1

[2011-01-31 14:20:24 1008] INFO (XendCheckpoint:423) /usr/lib/xen/bin/xc_save: symbol lookup error: /usr/lib/xen/bin/xc_save: undefined symbol: xs_suspend_evtchn_port

[2011-01-31 14:20:24 1008] ERROR (XendCheckpoint:178) Save failed on domain cloud-vm01 (10) - resuming.

Traceback (most recent call last):

  File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py", line 146, in save

    forkHelper(cmd, fd, saveInputHandler, False)

  File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py", line 409, in forkHelper

    raise XendError("%s failed: popen failed" % string.join(cmd))

XendError: /usr/lib/xen/bin/xc_save 46 10 0 0 1 failed: popen failed

[2011-01-31 14:20:24 1008] DEBUG (XendDomainInfo:3117) XendDomainInfo.resumeDomain(10)

[2011-01-31 14:20:27 1008] ERROR (xmlrpclib2:178) Internal error handling xend.domain.migrate

Traceback (most recent call last):

  File "/usr/local/lib/python2.6/dist-packages/xen/util/xmlrpclib2.py", line 131, in _marshaled_dispatch

    response = self._dispatch(method, params)

  File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 418, in _dispatch

    return func(*params)

  File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomain.py", line 1366, in domain_migrate

    self._domain_migrate(dominfo, dst, live, port, node)

  File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomain.py", line 1458, in _domain_migrate

    dsterr = sock.recv(1024)

timeout: timed out

Log on target Dom0:

[2011-01-31 14:20:23 12024] DEBUG (XendDomainInfo:235) XendDomainInfo.restore([\047domain\047, [\047domid\047, \04710\047], [\047cpu_weight\047, \047256\047], [\047cpu_cap\047, \0470\047], [\047bootloader\047, \047/usr/bin/pygrub\047], [\047on_crash\047, \047restart\047], [\047uuid\047, \047f32d195c-1fed-a9ad-5d7e-ff0456f79505\047], [\047bootloader_args\047], [\047vcpus\047, \0471\047], [\047description\047], [\047name\047, \047cloud-vm01\047], [\047cpus\047, [[]]], [\047on_reboot\047, \047restart\047], [\047on_poweroff\047, \047destroy\047], [\047maxmem\047, \047512\047], [\047memory\047, \047512\047], [\047shadow_memory\047, \0470\047], [\047vcpu_avail\047, \0471\047], [\047features\047], [\047on_xend_start\047, \047ignore\047], [\047on_xend_stop\047, \047ignore\047], [\047start_time\047, \0471296428689.01\047], [\047cpu_time\047, \04711.77212655\047], [\047online_vcpus\047, \0471\047], [\047image\047, [\047linux\047, [\047kernel\047], [\047args\047, \047root=/dev/sda2 ro \047], [\047superpages\047, \0470\047], [\047videoram\047, \0474\047], [\047pci\047, []], [\047nomigrate\047, \0470\047], [\047tsc_mode\047, \0470\047], [\047notes\047, [\047HV_START_LOW\047, \0474118806528\047], [\047FEATURES\047, \047!writable_page_tables|pae_pgdir_above_4gb\047], [\047VIRT_BASE\047, \0473221225472\047], [\047GUEST_VERSION\047, \0472.6\047], [\047PADDR_OFFSET\047, \0470\047], [\047GUEST_OS\047, \047linux\047], [\047HYPERCALL_PAGE\047, \0473222282240\047], [\047LOADER\047, \047generic\047], [\047SUSPEND_CANCEL\047, \0471\047], [\047PAE_MODE\047, \047yes\047], [\047ENTRY\047, \0473229974528\047], [\047XEN_VERSION\047, \047xen-3.0\047]]]], [\047status\047, \0472\047], [\047state\047, \047-b----\047], [\047store_mfn\047, \0472262985\047], [\047console_mfn\047, \0472262984\047], [\047device\047, [\047vif\047, [\047bridge\047, \047eth2\047], [\047uuid\047, \0477c22b250-2997-9b91-e6f0-2ad17a7e00db\047], [\047script\047, \047/etc/xen/scripts/vif-bridge\047], [\047ip\047, \047128.10.130.186\047], [\047mac\047, \04700:16:3E:C3:D2:A8\047], [\047backend\047, \0470\047]]], [\047device\047, [\047console\047, [\047protocol\047, \047vt100\047], [\047location\047, \0472\047], [\047uuid\047, \047b250cf24-1551-eaa0-cfdf-ea14e0d47934\047]]], [\047device\047, [\047vbd\047, [\047protocol\047, \047x86_32-abi\047], [\047uuid\047, \047f522f311-68bf-4358-216d-b8f81488a31b\047], [\047bootable\047, \0471\047], [\047dev\047, \047sda2:disk\047], [\047uname\047, \047file:/home/hkhande/xen-image/domains/cloud-vm01/disk.img\047], [\047mode\047, \047w\047], [\047backend\047, \0470\047], [\047VDI\047]]], [\047device\047, [\047vbd\047, [\047protocol\047, \047x86_32-abi\047], [\047uuid\047, \047a7f4ba1e-84e7-f104-a5f3-cd607b83a09f\047], [\047bootable\047, \0470\047], [\047dev\047, \047sda1:disk\047], [\047uname\047, \047file:/home/hkhande/xen-image/domains/cloud-vm01/swap.img\047], [\047mode\047, \047w\047], [\047backend\047, \0470\047], [\047VDI\047]]], [\047change_home_server\047, \047False\047]])

[2011-01-31 14:20:23 12024] DEBUG (XendDomainInfo:2508) XendDomainInfo.constructDomain

[2011-01-31 14:20:23 12024] DEBUG (balloon:220) Balloon: 1125176 KiB free; need 16384; done.

[2011-01-31 14:20:23 12024] DEBUG (XendDomain:464) Adding Domain: 8

[2011-01-31 14:20:23 12024] DEBUG (XendDomainInfo:3400) Storing VM details: {\047on_xend_stop\047: \047ignore\047, \047shadow_memory\047: \0470\047, \047uuid\047: \047f32d195c-1fed-a9ad-5d7e-ff0456f79505\047, \047on_reboot\047: \047restart\047, \047start_time\047: \0471296428689.01\047, \047on_poweroff\047: \047destroy\047, \047bootloader_args\047: \047\047, \047on_xend_start\047: \047ignore\047, \047on_crash\047: \047restart\047, \047xend/restart_count\047: \0470\047, \047vcpus\047: \0471\047, \047vcpu_avail\047: \0471\047, \047bootloader\047: \047/usr/bin/pygrub\047, \047image\047: "(linux (kernel ) (args \047root=/dev/sda2 ro \047) (superpages 0) (videoram 4) (pci ()) (nomigrate 0) (tsc_mode 0) (notes (HV_START_LOW 4118806528) (FEATURES \047!writable_page_tables|pae_pgdir_above_4gb\047) (VIRT_BASE 3221225472) (GUEST_VERSION 2.6) (PADDR_OFFSET 0) (GUEST_OS linux) (HYPERCALL_PAGE 3222282240) (LOADER generic) (SUSPEND_CANCEL 1) (PAE_MODE yes) (ENTRY 3229974528) (XEN_VERSION xen-3.0)))", \047name\047: \047cloud-vm01\047}

[2011-01-31 14:20:23 12024] INFO (XendDomainInfo:2367) createDevice: console : {\047protocol\047: \047vt100\047, \047location\047: \0472\047, \047uuid\047: \047b250cf24-1551-eaa0-cfdf-ea14e0d47934\047}

[2011-01-31 14:20:23 12024] DEBUG (DevController:95) DevController: writing {\047state\047: \0471\047, \047backend-id\047: \0470\047, \047backend\047: \047/local/domain/0/backend/console/8/0\047} to /local/domain/8/device/console/0.

[2011-01-31 14:20:23 12024] DEBUG (DevController:97) DevController: writing {\047domain\047: \047cloud-vm01\047, \047frontend\047: \047/local/domain/8/device/console/0\047, \047uuid\047: \047b250cf24-1551-eaa0-cfdf-ea14e0d47934\047, \047frontend-id\047: \0478\047, \047state\047: \0471\047, \047location\047: \0472\047, \047online\047: \0471\047, \047protocol\047: \047vt100\047} to /local/domain/0/backend/console/8/0.

[2011-01-31 14:20:23 12024] INFO (XendDomainInfo:2367) createDevice: vbd : {\047protocol\047: \047x86_32-abi\047, \047uuid\047: \047f522f311-68bf-4358-216d-b8f81488a31b\047, \047bootable\047: 1, \047driver\047: \047paravirtualised\047, \047dev\047: \047sda2:disk\047, \047uname\047: \047file:/home/hkhande/xen-image/domains/cloud-vm01/disk.img\047, \047mode\047: \047w\047, \047backend\047: \0470\047}

[2011-01-31 14:20:23 12024] DEBUG (DevController:95) DevController: writing {\047virtual-device\047: \0472050\047, \047protocol\047: \047x86_32-abi\047, \047device-type\047: \047disk\047, \047backend-id\047: \0470\047, \047state\047: \0471\047, \047backend\047: \047/local/domain/0/backend/vbd/8/2050\047} to /local/domain/8/device/vbd/2050.

[2011-01-31 14:20:23 12024] DEBUG (DevController:97) DevController: writing {\047domain\047: \047cloud-vm01\047, \047frontend\047: \047/local/domain/8/device/vbd/2050\047, \047uuid\047: \047f522f311-68bf-4358-216d-b8f81488a31b\047, \047bootable\047: \0471\047, \047dev\047: \047sda2\047, \047state\047: \0471\047, \047params\047: \047/home/hkhande/xen-image/domains/cloud-vm01/disk.img\047, \047mode\047: \047w\047, \047online\047: \0471\047, \047frontend-id\047: \0478\047, \047type\047: \047file\047} to /local/domain/0/backend/vbd/8/2050.

[2011-01-31 14:20:24 12024] INFO (XendDomainInfo:2367) createDevice: vbd : {\047protocol\047: \047x86_32-abi\047, \047uuid\047: \047a7f4ba1e-84e7-f104-a5f3-cd607b83a09f\047, \047bootable\047: 0, \047driver\047: \047paravirtualised\047, \047dev\047: \047sda1:disk\047, \047uname\047: \047file:/home/hkhande/xen-image/domains/cloud-vm01/swap.img\047, \047mode\047: \047w\047, \047backend\047: \0470\047}

[2011-01-31 14:20:24 12024] DEBUG (DevController:95) DevController: writing {\047virtual-device\047: \0472049\047, \047protocol\047: \047x86_32-abi\047, \047device-type\047: \047disk\047, \047backend-id\047: \0470\047, \047state\047: \0471\047, \047backend\047: \047/local/domain/0/backend/vbd/8/2049\047} to /local/domain/8/device/vbd/2049.

[2011-01-31 14:20:24 12024] DEBUG (DevController:97) DevController: writing {\047domain\047: \047cloud-vm01\047, \047frontend\047: \047/local/domain/8/device/vbd/2049\047, \047uuid\047: \047a7f4ba1e-84e7-f104-a5f3-cd607b83a09f\047, \047bootable\047: \0470\047, \047dev\047: \047sda1\047, \047state\047: \0471\047, \047params\047: \047/home/hkhande/xen-image/domains/cloud-vm01/swap.img\047, \047mode\047: \047w\047, \047online\047: \0471\047, \047frontend-id\047: \0478\047, \047type\047: \047file\047} to /local/domain/0/backend/vbd/8/2049.

[2011-01-31 14:20:24 12024] INFO (XendDomainInfo:2367) createDevice: vif : {\047bridge\047: \047eth2\047, \047uuid\047: \0477c22b250-2997-9b91-e6f0-2ad17a7e00db\047, \047script\047: \047/etc/xen/scripts/vif-bridge\047, \047ip\047: \047128.10.130.186\047, \047mac\047: \04700:16:3E:C3:D2:A8\047, \047backend\047: \0470\047}

[2011-01-31 14:20:24 12024] DEBUG (DevController:95) DevController: writing {\047backend-id\047: \0470\047, \047mac\047: \04700:16:3E:C3:D2:A8\047, \047handle\047: \0470\047, \047state\047: \0471\047, \047backend\047: \047/local/domain/0/backend/vif/8/0\047} to /local/domain/8/device/vif/0.

[2011-01-31 14:20:24 12024] DEBUG (DevController:97) DevController: writing {\047bridge\047: \047eth2\047, \047domain\047: \047cloud-vm01\047, \047handle\047: \0470\047, \047uuid\047: \0477c22b250-2997-9b91-e6f0-2ad17a7e00db\047, \047script\047: \047/etc/xen/scripts/vif-bridge\047, \047ip\047: \047128.10.130.186\047, \047mac\047: \04700:16:3E:C3:D2:A8\047, \047frontend-id\047: \0478\047, \047state\047: \0471\047, \047online\047: \0471\047, \047frontend\047: \047/local/domain/8/device/vif/0\047} to /local/domain/0/backend/vif/8/0.

[2011-01-31 14:20:24 12024] DEBUG (XendDomainInfo:1804) Storing domain details: {\047image/entry\047: \0473229974528\047, \047console/port\047: \0472\047, \047image/loader\047: \047generic\047, \047vm\047: \047/vm/f32d195c-1fed-a9ad-5d7e-ff0456f79505\047, \047control/platform-feature-multiprocessor-suspend\047: \0471\047, \047image/hv-start-low\047: \0474118806528\047, \047image/guest-os\047: \047linux\047, \047image/virt-base\047: \0473221225472\047, \047memory/target\047: \047524288\047, \047image/guest-version\047: \0472.6\047, \047image/pae-mode\047: \047yes\047, \047description\047: \047\047, \047console/limit\047: \0471048576\047, \047image/paddr-offset\047: \0470\047, \047image/hypercall-page\047: \0473222282240\047, \047image/suspend-cancel\047: \0471\047, \047cpu/0/availability\047: \047online\047, \047image/features/pae-pgdir-above-4gb\047: \0471\047, \047image/features/writable-page-tables\047: \0470\047, \047console/type\047: \047xenconsoled\047, \047name\047: \047cloud-vm01\047, \047domid\047: \0478\047, \047image/xen-version\047: \047xen-3.0\047, \047store/port\047: \0471\047}

[2011-01-31 14:20:24 12024] DEBUG (XendCheckpoint:286) restore:shadow=0x0, _static_max=0x20000000, _static_min=0x0, 

[2011-01-31 14:20:24 12024] DEBUG (XendCheckpoint:305) [xc_restore]: /usr/lib/xen/bin/xc_restore 16 8 1 2 0 0 0 0

[2011-01-31 14:20:26 12024] INFO (XendCheckpoint:423) ERROR Internal error: read: p2m_size

[2011-01-31 14:20:26 12024] INFO (XendCheckpoint:423) Restore exit with rc=1

[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:3053) XendDomainInfo.destroy: domid=8

[2011-01-31 14:20:26 12024] ERROR (XendDomainInfo:3067) XendDomainInfo.destroy: domain destruction failed.

Traceback (most recent call last):

  File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomainInfo.py", line 3060, in destroy

    xc.domain_pause(self.domid)

Error: (3, \047No such process\047)

[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:2416) No device model

[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:2418) Releasing devices

[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:2424) Removing vif/0

[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:1286) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0

[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:2424) Removing console/0

[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:1286) XendDomainInfo.destroyDevice: deviceClass = console, device = console/0

[2011-01-31 14:20:27 12024] DEBUG (XendDomainInfo:2424) Removing vbd/2050

[2011-01-31 14:20:27 12024] DEBUG (XendDomainInfo:1286) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/2050

[2011-01-31 14:20:27 12024] DEBUG (XendDomainInfo:2424) Removing vbd/2049

[2011-01-31 14:20:27 12024] DEBUG (XendDomainInfo:1286) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/2049

[2011-01-31 14:20:27 12024] INFO (XendDomain:1107) Domain cloud-vm01 (f32d195c-1fed-a9ad-5d7e-ff0456f79505) deleted.

[2011-01-31 14:20:27 12024] ERROR (XendCheckpoint:357) /usr/lib/xen/bin/xc_restore 16 8 1 2 0 0 0 0 failed

Traceback (most recent call last):

  File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py", line 309, in restore

    forkHelper(cmd, fd, handler.handler, True)

  File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py", line 411, in forkHelper

    raise XendError("%s failed" % string.join(cmd))

XendError: /usr/lib/xen/bin/xc_restore 16 8 1 2 0 0 0 0 failed

[2011-01-31 14:20:27 12024] ERROR (XendDomain:1175) Restore failed

Traceback (most recent call last):

  File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomain.py", line 1159, in domain_restore_fd

    dominfo = XendCheckpoint.restore(self, fd, paused=paused, relocating=relocating)

  File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py", line 358, in restore

    raise exn

XendError: /usr/lib/xen/bin/xc_restore 16 8 1 2 0 0 0 0 failed

Hitesh Khandelwal | Graduate Student | Purdue University | http://www.hiteshkhandelwal.com

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users