[Xen-devel] [BUG] complicated migration problem

[Michal Purzynski <michal@xxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

first of all, thank you for a great project i really enjoy!

ok, now here comes the bug report...

i'm trying to test xen migration capabilities, with xen-3.1, hand  
compilled, fetched from mercurial repository.
my setup looks like this - x86_64 dom0, x86_32p dom0, x86_32p domU.
when trying to migrate from 64 to 32 bit dom0 i'm getting following  
error on x86_64 dom0
[2007-08-13 16:53:18 3247] DEBUG (XendDomainInfo:824) Storing domain  
details: {'console/ring-ref': '753585', 'image/entry': '3223322624',  
'console/port': '2', 'store/ring-ref': '753586', 'image/loader':  
'generic', 'vm': '/vm/fee7f9d5-2934-7c54-01ff-cb475482540c',  
'control/platform-feature-multiprocessor-suspend': '1',  
'image/hv-start-low': '4118806528', 'image/guest-os': 'linux',  
'image/features/writable-descriptor-tables': '1', 'image/virt-base':  
'3221225472', 'memory/target': '65536', 'image/guest-version': '2.6',  
'image/features/supervisor-mode-kernel': '1', 'image/pae-mode': 'yes',  
'console/limit': '1048576', 'image/paddr-offset': '0',  
'image/hypercall-page': '3223326720', 'image/suspend-cancel': '1',  
'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb':  
'1', 'image/features/writable-page-tables': '1',  
'image/features/auto-translated-physmap': '1', 'name':  
'migrating-Etch', 'domid': '1', 'image/xen-version': 'xen-3.0',  
'store/port': '1'}
[2007-08-13 16:53:18 3247] DEBUG (XendCheckpoint:88) [xc_save]:  
/usr/lib64/xen/bin/xc_save 24 1 0 0 0
[2007-08-13 16:53:18 3247] DEBUG (XendCheckpoint:320) suspend
[2007-08-13 16:53:18 3247] DEBUG (XendCheckpoint:91) In  
saveInputHandler suspend
[2007-08-13 16:53:18 3247] DEBUG (XendCheckpoint:93) Suspending 1 ...
[2007-08-13 16:53:18 3247] DEBUG (XendDomainInfo:443)  
XendDomainInfo.shutdown(suspend)
[2007-08-13 16:53:18 3247] DEBUG (XendDomainInfo:908)  
XendDomainInfo.handleShutdownWatch
[2007-08-13 16:53:18 3247] DEBUG (XendDomainInfo:908)  
XendDomainInfo.handleShutdownWatch
[2007-08-13 16:53:18 3247] INFO (XendDomainInfo:1090) Domain has  
shutdown: name=migrating-Etch id=1 reason=suspend.
[2007-08-13 16:53:18 3247] INFO (XendCheckpoint:98) Domain 1 suspended.
[2007-08-13 16:53:18 3247] DEBUG (XendCheckpoint:107) Written done
[2007-08-13 16:53:19 3247] INFO (XendCheckpoint:349) ERROR Internal  
error: Timed out waiting for frame list updated.
[2007-08-13 16:53:19 3247] INFO (XendCheckpoint:349) ERROR Internal  
error: Failed to map/save the p2m frame list
[2007-08-13 16:53:19 3247] INFO (XendCheckpoint:349) Save exit rc=1
[2007-08-13 16:53:19 3247] ERROR (XendCheckpoint:140) Save failed on  
domain Etch (1).
Traceback (most recent call last):
  File "//usr/lib64/python/xen/xend/XendCheckpoint.py", line 109, in  
save
    forkHelper(cmd, fd, saveInputHandler, False)
  File "//usr/lib64/python/xen/xend/XendCheckpoint.py", line 337, in  
forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib64/xen/bin/xc_save 24 1 0 0 0 failed
[2007-08-13 16:53:19 3247] DEBUG (XendDomainInfo:1699)  
XendDomainInfo.resumeDomain(1)
[2007-08-13 16:53:19 3247] DEBUG (XendCheckpoint:143)  
XendCheckpoint.save: resumeDomain
and following one on x86_32p dom0

[2007-08-13 17:32:21 3195] DEBUG (XendDomainInfo:210)  
XendDomainInfo.restore(['domain', ['domid', '1'], ['on_crash',  
'restart'], ['uuid', 'fee7f9d5-2934-7c54-01ff-cb475482540c'],  
['bootloader_args'], ['vcpus', '1'], ['name', 'Etch'], ['on_poweroff',  
'destroy'], ['on_reboot', 'restart'], ['bootloader'], ['maxmem', '64'],  
['memory', '64'], ['shadow_memory', '0'], ['vcpu_avail', '1'],  
['cpu_weight', '256'], ['cpu_cap', '0'], ['features'],  
['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['start_time',  
'1187016349.76'], ['cpu_time', '3.379816355'], ['online_vcpus', '1'],  
['image', ['linux', ['kernel', '/boot/vmlinux-2.6.18-xen-3.1-domU_32'],  
['args', 'root=/dev/xvda1 ro 3'], ['rtc_timeoffset', '0'], ['notes',  
['HV_START_LOW', '4118806528'], ['FEATURES',  
'writable_page_tables|writable_descriptor_tables|auto_translated_physmap 
|pae_pgdir_above_4gb|supervisor_mode_kernel'], ['VIRT_BASE',  
'3221225472'], ['GUEST_VERSION', '2.6'], ['PADDR_OFFSET', '0'],  
['GUEST_OS', 'linux'], ['HYPERCALL_PAGE', '3223326720'], ['LOADER',  
'generic'], ['SUSPEND_CANCEL', '1'], ['PAE_MODE', 'yes'], ['ENTRY',  
'3223322624'], ['XEN_VERSION', 'xen-3.0']]]], ['status', '2'],  
['state', '-b----'], ['store_mfn', '753586'], ['console_mfn',  
'753585'], ['device', ['vif', ['mac', '00:16:3e:37:b9:24'], ['script',  
'vif-bridge'], ['type', 'netfront'], ['uuid',  
'75f6bbf5-9bc2-4c42-af5c-13d0d85b99b3'], ['backend', '0']]], ['device',  
['vbd', ['uname',  
'phy:/dev/disk/by-path/ip-192.168.1.4:3260-iscsi-iqn.2007-08.eu.nusec: 
xen.etch.xvda'], ['uuid', 'ade9f8ea-c9ea-7de0-3fac-0a2d2bc8f050'],  
['mode', 'w'], ['dev', 'xvda1:disk'], ['backend', '0'], ['bootable',  
'1']]], ['device', ['console', ['protocol', 'vt100'], ['location',  
'2'], ['uuid', '405936bb-c9d1-cdc0-de34-3e25a2b454fb']]]])
[2007-08-13 17:32:21 3195] DEBUG (XendDomainInfo:1414)  
XendDomainInfo.constructDomain
[2007-08-13 17:32:21 3195] DEBUG (balloon:113) Balloon: 2716848 KiB  
free; need 2048; done.
[2007-08-13 17:32:21 3195] DEBUG (XendDomain:434) Adding Domain: 2
[2007-08-13 17:32:21 3195] DEBUG (XendDomainInfo:1973) Storing VM  
details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid':  
'fee7f9d5-2934-7c54-01ff-cb475482540c', 'on_reboot': 'restart',  
'start_time': '1187016349.76', 'on_poweroff': 'destroy',  
'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count':  
'0', 'vcpus': '1', 'vcpu_avail': '1', 'image': "(linux (kernel  
/boot/vmlinux-2.6.18-xen-3.1-domU_32) (args 'root=/dev/xvda1 ro 3')  
(rtc_timeoffset 0) (notes (HV_START_LOW 4118806528) (FEATURES  
'writable_page_tables|writable_descriptor_tables|auto_translated_physmap 
|pae_pgdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE 3221225472)  
(GUEST_VERSION 2.6) (PADDR_OFFSET 0) (GUEST_OS linux) (HYPERCALL_PAGE  
3223326720) (LOADER generic) (SUSPEND_CANCEL 1) (PAE_MODE yes) (ENTRY  
3223322624) (XEN_VERSION xen-3.0)))", 'name': 'Etch'}
[2007-08-13 17:32:21 3195] INFO (XendDomainInfo:1316) createDevice:  
console : {'protocol': 'vt100', 'location': '2', 'uuid':  
'405936bb-c9d1-cdc0-de34-3e25a2b454fb'}
[2007-08-13 17:32:21 3195] DEBUG (DevController:115) DevController:  
writing {'state': '1', 'backend-id': '0', 'backend':  
'/local/domain/0/backend/console/2/0'} to  
/local/domain/2/device/console/0.
[2007-08-13 17:32:21 3195] DEBUG (DevController:117) DevController:  
writing {'domain': 'Etch', 'protocol': 'vt100', 'uuid':  
'405936bb-c9d1-cdc0-de34-3e25a2b454fb', 'frontend':  
'/local/domain/2/device/console/0', 'state': '1', 'location': '2',  
'online': '1', 'frontend-id': '2'} to  
/local/domain/0/backend/console/2/0.
[2007-08-13 17:32:21 3195] INFO (XendDomainInfo:1316) createDevice: vbd  
: {'uuid': 'ade9f8ea-c9ea-7de0-3fac-0a2d2bc8f050', 'bootable': 1,  
'driver': 'paravirtualised', 'dev': 'xvda1:disk', 'uname':  
'phy:/dev/disk/by-path/ip-192.168.1.4:3260-iscsi-iqn.2007-08.eu.nusec: 
xen.etch.xvda', 'mode': 'w', 'backend': '0'}
[2007-08-13 17:32:21 3195] DEBUG (DevController:115) DevController:  
writing {'backend-id': '0', 'virtual-device': '51713', 'device-type':  
'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/2/51713'}  
to /local/domain/2/device/vbd/51713.
[2007-08-13 17:32:21 3195] DEBUG (DevController:117) DevController:  
writing {'domain': 'Etch', 'frontend':  
'/local/domain/2/device/vbd/51713', 'uuid':  
'ade9f8ea-c9ea-7de0-3fac-0a2d2bc8f050', 'dev': 'xvda1', 'state': '1',  
'params':  
'/dev/disk/by-path/ip-192.168.1.4:3260-iscsi-iqn.2007-08.eu.nusec: 
xen.etch.xvda', 'mode': 'w', 'online': '1', 'frontend-id': '2', 'type':  
'phy'} to /local/domain/0/backend/vbd/2/51713.
[2007-08-13 17:32:21 3195] INFO (XendDomainInfo:1316) createDevice: vif  
: {'mac': '00:16:3e:37:b9:24', 'script': 'vif-bridge', 'type':  
'netfront', 'uuid': '75f6bbf5-9bc2-4c42-af5c-13d0d85b99b3', 'backend':  
'0'}
[2007-08-13 17:32:21 3195] DEBUG (DevController:115) DevController:  
writing {'backend-id': '0', 'mac': '00:16:3e:37:b9:24', 'handle': '0',  
'state': '1', 'backend': '/local/domain/0/backend/vif/2/0'} to  
/local/domain/2/device/vif/0.
[2007-08-13 17:32:21 3195] DEBUG (DevController:117) DevController:  
writing {'domain': 'Etch', 'handle': '0', 'uuid':  
'75f6bbf5-9bc2-4c42-af5c-13d0d85b99b3', 'script':  
'/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend':  
'/local/domain/2/device/vif/0', 'mac': '00:16:3e:37:b9:24', 'online':  
'1', 'frontend-id': '2', 'type': 'netfront'} to  
/local/domain/0/backend/vif/2/0.
[2007-08-13 17:32:21 3195] DEBUG (XendDomainInfo:824) Storing domain  
details: {'image/entry': '3223322624', 'console/port': '2',  
'image/loader': 'generic', 'vm':  
'/vm/fee7f9d5-2934-7c54-01ff-cb475482540c',  
'control/platform-feature-multiprocessor-suspend': '1',  
'image/hv-start-low': '4118806528', 'image/guest-os': 'linux',  
'image/features/writable-descriptor-tables': '1', 'image/virt-base':  
'3221225472', 'memory/target': '65536', 'image/guest-version': '2.6',  
'image/features/supervisor-mode-kernel': '1', 'image/pae-mode': 'yes',  
'console/limit': '1048576', 'image/paddr-offset': '0',  
'image/hypercall-page': '3223326720', 'image/suspend-cancel': '1',  
'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb':  
'1', 'image/features/writable-page-tables': '1',  
'image/features/auto-translated-physmap': '1', 'name': 'Etch', 'domid':  
'2', 'image/xen-version': 'xen-3.0', 'store/port': '1'}
[2007-08-13 17:32:21 3195] DEBUG (XendCheckpoint:201)  
restore:shadow=0x0, _static_max=0x4000000, _static_min=0x0,
[2007-08-13 17:32:21 3195] DEBUG (balloon:113) Balloon: 2716848 KiB  
free; need 65536; done.
[2007-08-13 17:32:21 3195] DEBUG (XendCheckpoint:213) [xc_restore]:  
/usr/lib/xen/bin/xc_restore 16 2 1 2 0 0 0
[2007-08-13 17:32:21 3195] INFO (XendCheckpoint:349) xc_domain_restore  
start: p2m_size = 4800
[2007-08-13 17:32:22 3195] INFO (XendCheckpoint:349) ERROR Internal  
error: read p2m_frame_list failed
[2007-08-13 17:32:22 3195] INFO (XendCheckpoint:349) Restore exit with  
rc=1
[2007-08-13 17:32:22 3195] DEBUG (XendDomainInfo:1660)  
XendDomainInfo.destroy: domid=2
[2007-08-13 17:32:22 3195] DEBUG (XendDomainInfo:1677)  
XendDomainInfo.destroyDomain(2)
[2007-08-13 17:32:22 3195] ERROR (XendDomainInfo:1689)  
XendDomainInfo.destroy: xc.domain_destroy failed.
Traceback (most recent call last):
  File "//usr/lib/python/xen/xend/XendDomainInfo.py", line 1683, in  
destroyDomain
    xc.domain_destroy(self.domid)
Error: (3, 'No such process')
[2007-08-13 17:32:22 3195] ERROR (XendDomain:1089) Restore failed
Traceback (most recent call last):
  File "//usr/lib/python/xen/xend/XendDomain.py", line 1084, in  
domain_restore_fd
    return XendCheckpoint.restore(self, fd, paused=paused)
  File "//usr/lib/python/xen/xend/XendCheckpoint.py", line 217, in  
restore
    forkHelper(cmd, fd, handler.handler, True)
  File "//usr/lib/python/xen/xend/XendCheckpoint.py", line 337, in  
forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib/xen/bin/xc_restore 16 2 1 2 0 0 0 failed

after looking at it, seems that p2m (whatever it is) is misbehaving.

x86_64 machine has 3 GB of ram, x86_32p also 3 GB, PAE enabled both in  
dom0 kernel and xen. (and domU too).
looking forward for help with resolving this problem. i can provide any  
additional details i could be missing out.
Michal Purzynski
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)

iD8DBQFGwHIy+8juDbyM0PsRAqwkAJ9nyOBUaSje63C9qemwct1i3PAiOQCghW4o
6GnNLCRXldtl4FLc6DPTf7E=
=MhKd
-----END PGP SIGNATURE-----


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel