[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] [BUG] complicated migration problem


  • To: xen-devel@xxxxxxxxxxxxxxxxxxx
  • From: Michal Purzynski <michal@xxxxxxxxx>
  • Date: Mon, 13 Aug 2007 17:01:02 +0200
  • Delivery-date: Fri, 17 Aug 2007 09:54:44 -0700
  • List-id: Xen developer discussion <xen-devel.lists.xensource.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

first of all, thank you for a great project i really enjoy!

ok, now here comes the bug report...

i'm trying to test xen migration capabilities, with xen-3.1, hand compilled, fetched from mercurial repository.

my setup looks like this - x86_64 dom0, x86_32p dom0, x86_32p domU.
when trying to migrate from 64 to 32 bit dom0 i'm getting following error on x86_64 dom0

[2007-08-13 16:53:18 3247] DEBUG (XendDomainInfo:824) Storing domain details: {'console/ring-ref': '753585', 'image/entry': '3223322624', 'console/port': '2', 'store/ring-ref': '753586', 'image/loader': 'generic', 'vm': '/vm/fee7f9d5-2934-7c54-01ff-cb475482540c', 'control/platform-feature-multiprocessor-suspend': '1', 'image/hv-start-low': '4118806528', 'image/guest-os': 'linux', 'image/features/writable-descriptor-tables': '1', 'image/virt-base': '3221225472', 'memory/target': '65536', 'image/guest-version': '2.6', 'image/features/supervisor-mode-kernel': '1', 'image/pae-mode': 'yes', 'console/limit': '1048576', 'image/paddr-offset': '0', 'image/hypercall-page': '3223326720', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '1', 'image/features/auto-translated-physmap': '1', 'name': 'migrating-Etch', 'domid': '1', 'image/xen-version': 'xen-3.0', 'store/port': '1'} [2007-08-13 16:53:18 3247] DEBUG (XendCheckpoint:88) [xc_save]: /usr/lib64/xen/bin/xc_save 24 1 0 0 0
[2007-08-13 16:53:18 3247] DEBUG (XendCheckpoint:320) suspend
[2007-08-13 16:53:18 3247] DEBUG (XendCheckpoint:91) In saveInputHandler suspend
[2007-08-13 16:53:18 3247] DEBUG (XendCheckpoint:93) Suspending 1 ...
[2007-08-13 16:53:18 3247] DEBUG (XendDomainInfo:443) XendDomainInfo.shutdown(suspend) [2007-08-13 16:53:18 3247] DEBUG (XendDomainInfo:908) XendDomainInfo.handleShutdownWatch [2007-08-13 16:53:18 3247] DEBUG (XendDomainInfo:908) XendDomainInfo.handleShutdownWatch [2007-08-13 16:53:18 3247] INFO (XendDomainInfo:1090) Domain has shutdown: name=migrating-Etch id=1 reason=suspend.
[2007-08-13 16:53:18 3247] INFO (XendCheckpoint:98) Domain 1 suspended.
[2007-08-13 16:53:18 3247] DEBUG (XendCheckpoint:107) Written done
[2007-08-13 16:53:19 3247] INFO (XendCheckpoint:349) ERROR Internal error: Timed out waiting for frame list updated. [2007-08-13 16:53:19 3247] INFO (XendCheckpoint:349) ERROR Internal error: Failed to map/save the p2m frame list
[2007-08-13 16:53:19 3247] INFO (XendCheckpoint:349) Save exit rc=1
[2007-08-13 16:53:19 3247] ERROR (XendCheckpoint:140) Save failed on domain Etch (1).
Traceback (most recent call last):
File "//usr/lib64/python/xen/xend/XendCheckpoint.py", line 109, in save
    forkHelper(cmd, fd, saveInputHandler, False)
File "//usr/lib64/python/xen/xend/XendCheckpoint.py", line 337, in forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib64/xen/bin/xc_save 24 1 0 0 0 failed
[2007-08-13 16:53:19 3247] DEBUG (XendDomainInfo:1699) XendDomainInfo.resumeDomain(1) [2007-08-13 16:53:19 3247] DEBUG (XendCheckpoint:143) XendCheckpoint.save: resumeDomain

and following one on x86_32p dom0

[2007-08-13 17:32:21 3195] DEBUG (XendDomainInfo:210) XendDomainInfo.restore(['domain', ['domid', '1'], ['on_crash', 'restart'], ['uuid', 'fee7f9d5-2934-7c54-01ff-cb475482540c'], ['bootloader_args'], ['vcpus', '1'], ['name', 'Etch'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['bootloader'], ['maxmem', '64'], ['memory', '64'], ['shadow_memory', '0'], ['vcpu_avail', '1'], ['cpu_weight', '256'], ['cpu_cap', '0'], ['features'], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['start_time', '1187016349.76'], ['cpu_time', '3.379816355'], ['online_vcpus', '1'], ['image', ['linux', ['kernel', '/boot/vmlinux-2.6.18-xen-3.1-domU_32'], ['args', 'root=/dev/xvda1 ro 3'], ['rtc_timeoffset', '0'], ['notes', ['HV_START_LOW', '4118806528'], ['FEATURES', 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap |pae_pgdir_above_4gb|supervisor_mode_kernel'], ['VIRT_BASE', '3221225472'], ['GUEST_VERSION', '2.6'], ['PADDR_OFFSET', '0'], ['GUEST_OS', 'linux'], ['HYPERCALL_PAGE', '3223326720'], ['LOADER', 'generic'], ['SUSPEND_CANCEL', '1'], ['PAE_MODE', 'yes'], ['ENTRY', '3223322624'], ['XEN_VERSION', 'xen-3.0']]]], ['status', '2'], ['state', '-b----'], ['store_mfn', '753586'], ['console_mfn', '753585'], ['device', ['vif', ['mac', '00:16:3e:37:b9:24'], ['script', 'vif-bridge'], ['type', 'netfront'], ['uuid', '75f6bbf5-9bc2-4c42-af5c-13d0d85b99b3'], ['backend', '0']]], ['device', ['vbd', ['uname', 'phy:/dev/disk/by-path/ip-192.168.1.4:3260-iscsi-iqn.2007-08.eu.nusec: xen.etch.xvda'], ['uuid', 'ade9f8ea-c9ea-7de0-3fac-0a2d2bc8f050'], ['mode', 'w'], ['dev', 'xvda1:disk'], ['backend', '0'], ['bootable', '1']]], ['device', ['console', ['protocol', 'vt100'], ['location', '2'], ['uuid', '405936bb-c9d1-cdc0-de34-3e25a2b454fb']]]]) [2007-08-13 17:32:21 3195] DEBUG (XendDomainInfo:1414) XendDomainInfo.constructDomain [2007-08-13 17:32:21 3195] DEBUG (balloon:113) Balloon: 2716848 KiB free; need 2048; done.
[2007-08-13 17:32:21 3195] DEBUG (XendDomain:434) Adding Domain: 2
[2007-08-13 17:32:21 3195] DEBUG (XendDomainInfo:1973) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': 'fee7f9d5-2934-7c54-01ff-cb475482540c', 'on_reboot': 'restart', 'start_time': '1187016349.76', 'on_poweroff': 'destroy', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'image': "(linux (kernel /boot/vmlinux-2.6.18-xen-3.1-domU_32) (args 'root=/dev/xvda1 ro 3') (rtc_timeoffset 0) (notes (HV_START_LOW 4118806528) (FEATURES 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap |pae_pgdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE 3221225472) (GUEST_VERSION 2.6) (PADDR_OFFSET 0) (GUEST_OS linux) (HYPERCALL_PAGE 3223326720) (LOADER generic) (SUSPEND_CANCEL 1) (PAE_MODE yes) (ENTRY 3223322624) (XEN_VERSION xen-3.0)))", 'name': 'Etch'} [2007-08-13 17:32:21 3195] INFO (XendDomainInfo:1316) createDevice: console : {'protocol': 'vt100', 'location': '2', 'uuid': '405936bb-c9d1-cdc0-de34-3e25a2b454fb'} [2007-08-13 17:32:21 3195] DEBUG (DevController:115) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/2/0'} to /local/domain/2/device/console/0. [2007-08-13 17:32:21 3195] DEBUG (DevController:117) DevController: writing {'domain': 'Etch', 'protocol': 'vt100', 'uuid': '405936bb-c9d1-cdc0-de34-3e25a2b454fb', 'frontend': '/local/domain/2/device/console/0', 'state': '1', 'location': '2', 'online': '1', 'frontend-id': '2'} to /local/domain/0/backend/console/2/0. [2007-08-13 17:32:21 3195] INFO (XendDomainInfo:1316) createDevice: vbd : {'uuid': 'ade9f8ea-c9ea-7de0-3fac-0a2d2bc8f050', 'bootable': 1, 'driver': 'paravirtualised', 'dev': 'xvda1:disk', 'uname': 'phy:/dev/disk/by-path/ip-192.168.1.4:3260-iscsi-iqn.2007-08.eu.nusec: xen.etch.xvda', 'mode': 'w', 'backend': '0'} [2007-08-13 17:32:21 3195] DEBUG (DevController:115) DevController: writing {'backend-id': '0', 'virtual-device': '51713', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/2/51713'} to /local/domain/2/device/vbd/51713. [2007-08-13 17:32:21 3195] DEBUG (DevController:117) DevController: writing {'domain': 'Etch', 'frontend': '/local/domain/2/device/vbd/51713', 'uuid': 'ade9f8ea-c9ea-7de0-3fac-0a2d2bc8f050', 'dev': 'xvda1', 'state': '1', 'params': '/dev/disk/by-path/ip-192.168.1.4:3260-iscsi-iqn.2007-08.eu.nusec: xen.etch.xvda', 'mode': 'w', 'online': '1', 'frontend-id': '2', 'type': 'phy'} to /local/domain/0/backend/vbd/2/51713. [2007-08-13 17:32:21 3195] INFO (XendDomainInfo:1316) createDevice: vif : {'mac': '00:16:3e:37:b9:24', 'script': 'vif-bridge', 'type': 'netfront', 'uuid': '75f6bbf5-9bc2-4c42-af5c-13d0d85b99b3', 'backend': '0'} [2007-08-13 17:32:21 3195] DEBUG (DevController:115) DevController: writing {'backend-id': '0', 'mac': '00:16:3e:37:b9:24', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/2/0'} to /local/domain/2/device/vif/0. [2007-08-13 17:32:21 3195] DEBUG (DevController:117) DevController: writing {'domain': 'Etch', 'handle': '0', 'uuid': '75f6bbf5-9bc2-4c42-af5c-13d0d85b99b3', 'script': '/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': '/local/domain/2/device/vif/0', 'mac': '00:16:3e:37:b9:24', 'online': '1', 'frontend-id': '2', 'type': 'netfront'} to /local/domain/0/backend/vif/2/0. [2007-08-13 17:32:21 3195] DEBUG (XendDomainInfo:824) Storing domain details: {'image/entry': '3223322624', 'console/port': '2', 'image/loader': 'generic', 'vm': '/vm/fee7f9d5-2934-7c54-01ff-cb475482540c', 'control/platform-feature-multiprocessor-suspend': '1', 'image/hv-start-low': '4118806528', 'image/guest-os': 'linux', 'image/features/writable-descriptor-tables': '1', 'image/virt-base': '3221225472', 'memory/target': '65536', 'image/guest-version': '2.6', 'image/features/supervisor-mode-kernel': '1', 'image/pae-mode': 'yes', 'console/limit': '1048576', 'image/paddr-offset': '0', 'image/hypercall-page': '3223326720', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '1', 'image/features/auto-translated-physmap': '1', 'name': 'Etch', 'domid': '2', 'image/xen-version': 'xen-3.0', 'store/port': '1'} [2007-08-13 17:32:21 3195] DEBUG (XendCheckpoint:201) restore:shadow=0x0, _static_max=0x4000000, _static_min=0x0, [2007-08-13 17:32:21 3195] DEBUG (balloon:113) Balloon: 2716848 KiB free; need 65536; done. [2007-08-13 17:32:21 3195] DEBUG (XendCheckpoint:213) [xc_restore]: /usr/lib/xen/bin/xc_restore 16 2 1 2 0 0 0 [2007-08-13 17:32:21 3195] INFO (XendCheckpoint:349) xc_domain_restore start: p2m_size = 4800 [2007-08-13 17:32:22 3195] INFO (XendCheckpoint:349) ERROR Internal error: read p2m_frame_list failed [2007-08-13 17:32:22 3195] INFO (XendCheckpoint:349) Restore exit with rc=1 [2007-08-13 17:32:22 3195] DEBUG (XendDomainInfo:1660) XendDomainInfo.destroy: domid=2 [2007-08-13 17:32:22 3195] DEBUG (XendDomainInfo:1677) XendDomainInfo.destroyDomain(2) [2007-08-13 17:32:22 3195] ERROR (XendDomainInfo:1689) XendDomainInfo.destroy: xc.domain_destroy failed.
Traceback (most recent call last):
File "//usr/lib/python/xen/xend/XendDomainInfo.py", line 1683, in destroyDomain
    xc.domain_destroy(self.domid)
Error: (3, 'No such process')
[2007-08-13 17:32:22 3195] ERROR (XendDomain:1089) Restore failed
Traceback (most recent call last):
File "//usr/lib/python/xen/xend/XendDomain.py", line 1084, in domain_restore_fd
    return XendCheckpoint.restore(self, fd, paused=paused)
File "//usr/lib/python/xen/xend/XendCheckpoint.py", line 217, in restore
    forkHelper(cmd, fd, handler.handler, True)
File "//usr/lib/python/xen/xend/XendCheckpoint.py", line 337, in forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib/xen/bin/xc_restore 16 2 1 2 0 0 0 failed

after looking at it, seems that p2m (whatever it is) is misbehaving.

x86_64 machine has 3 GB of ram, x86_32p also 3 GB, PAE enabled both in dom0 kernel and xen. (and domU too).

looking forward for help with resolving this problem. i can provide any additional details i could be missing out.

Michal Purzynski
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)

iD8DBQFGwHIy+8juDbyM0PsRAqwkAJ9nyOBUaSje63C9qemwct1i3PAiOQCghW4o
6GnNLCRXldtl4FLc6DPTf7E=
=MhKd
-----END PGP SIGNATURE-----


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.