[Xen-users] Migration failing (xen 3.0.3 on RHEL 5) with "not a valid guest state file: pfn count read"

["McCool, Scott" <S.McCool@xxxxxxxxxxxx>]

Hi all, I'm having some problem migrating a simple domU between two
hosts.

Short question:  xm migrate (live or not) fails even though either host
is capable of running the VM.  Error seems to be:  "not a valid guest
state file: pfn count read"

Long question:
I have two boxes (devvmdevvmhost1 and devvmhost2) running xen 3.0.3
(RHEL 5), they are attached to a Dell MD3000 unit for shared storage.
I'm working on configuring live migration, and have set up a test VM
(testwww).  Storage is a single LVM partition on the MD3000.

I can start the VM on either physical host (copied /etc/xen/testwww from
devvmhost1 to devvmhost2) and it works fine (obviously only running it
one one physical box at a time).  

I've configured migration in xend-config on both hosts.

When I try to migrate the VM (either live or not) I get an error, no
matter which machine I'm migrating from the error is identical.  The
relevant message would seem to be:  " not a valid guest state file: pfn
count read", and some googling just turns up references to the source.
Can anyone help me debug my migration problem?  Have I done something
wrong in the setup?  

If I have the VM running on devvmhost1, and try to migrate it to
devvmhost2, the following happens:

On devvmhost1 (sender):
[root@devvmhost1 ~]# xm list
Name                                      ID Mem(MiB) VCPUs State
Time(s)
Domain-0                                   0    15649     8 r-----
2447.0
testwww                                   11      513     1 r-----
30.6
[root@devvmhost1 ~]# xm migrate testwww devvmdevvmhost2
Error: /usr/lib64/xen/bin/xc_save 19 11 0 0 0 failed
Usage: xm migrate <Domain> <Host>

Migrate a domain to another machine.

Options:

-h, --help           Print this help.
-l, --live           Use live migration.
-p=portnum, --port=portnum
                     Use specified port for migration.
-r=MBIT, --resource=MBIT
                     Set level of resource usage for migration.




On devvmhost2 (receiver), the following is output to
/var/log/xen/xend.log when the migration starts:

[2007-09-27 09:32:01 xend 27175] DEBUG (tcp:66) Match devvmhost1
^devvmhost1$
[2007-09-27 09:32:01 xend.XendDomainInfo 27175] DEBUG
(XendDomainInfo:265) XendDomainInfo.restore(['domain', ['domid', '11'],
['uuid', 'b9527e38-a857-aa8f-3e81-97c2e92c8ef5'], ['vcpus', '1'],
['vcpu_avail', '1'], ['cpu_weight', '1.0'], ['memory', '500'],
['shadow_memory', '3'], ['maxmem', '500'], ['features'], ['name',
'testwww'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'],
['on_crash', 'restart'], ['image', ['hvm', ['kernel',
'/usr/lib/xen/boot/hvmloader'], ['device_model',
'/usr/lib64/xen/bin/qemu-dm'], ['pae', '1'], ['vcpus', '1'], ['boot',
'c'], ['serial', 'pty'], ['vnc', '1'], ['vncunused', '1'],
['xauthority', '/root/.Xauthority'], ['acpi', '1'], ['apic', '1'],
['vncpasswd', 'None']]], ['device', ['vif', ['backend', '0'], ['script',
'vif-bridge'], ['bridge', 'xenbr0'], ['mac', '00:16:3e:35:f8:cc'],
['type', 'ioemu']]], ['device', ['vbd', ['backend', '0'], ['dev',
'hda:disk'], ['uname', 'phy:/dev/devVMvg/test-www-root'], ['mode',
'w']]], ['device', ['vkbd', ['backend', '0']]], ['device', ['vfb',
['backend', '0'], ['type', 'vnc'], ['vncunused', '1'], ['xauthority',
'/root/.Xauthority']]], ['state', 'r-----'], ['shutdown_reason',
'poweroff'], ['cpu_time', '30.657009805'], ['online_vcpus', '1'],
['up_time', '103.057826042'], ['start_time', '1190888774.28'],
['store_mfn', '134135']])
[2007-09-27 09:32:01 xend.XendDomainInfo 27175] DEBUG
(XendDomainInfo:296) parseConfig: config is ['domain', ['domid', '11'],
['uuid', 'b9527e38-a857-aa8f-3e81-97c2e92c8ef5'], ['vcpus', '1'],
['vcpu_avail', '1'], ['cpu_weight', '1.0'], ['memory', '500'],
['shadow_memory', '3'], ['maxmem', '500'], ['features'], ['name',
'testwww'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'],
['on_crash', 'restart'], ['image', ['hvm', ['kernel',
'/usr/lib/xen/boot/hvmloader'], ['device_model',
'/usr/lib64/xen/bin/qemu-dm'], ['pae', '1'], ['vcpus', '1'], ['boot',
'c'], ['serial', 'pty'], ['vnc', '1'], ['vncunused', '1'],
['xauthority', '/root/.Xauthority'], ['acpi', '1'], ['apic', '1'],
['vncpasswd', 'None']]], ['device', ['vif', ['backend', '0'], ['script',
'vif-bridge'], ['bridge', 'xenbr0'], ['mac', '00:16:3e:35:f8:cc'],
['type', 'ioemu']]], ['device', ['vbd', ['backend', '0'], ['dev',
'hda:disk'], ['uname', 'phy:/dev/devVMvg/test-www-root'], ['mode',
'w']]], ['device', ['vkbd', ['backend', '0']]], ['device', ['vfb',
['backend', '0'], ['type', 'vnc'], ['vncunused', '1'], ['xauthority',
'/root/.Xauthority']]], ['state', 'r-----'], ['shutdown_reason',
'poweroff'], ['cpu_time', '30.657009805'], ['online_vcpus', '1'],
['up_time', '103.057826042'], ['start_time', '1190888774.28'],
['store_mfn', '134135']]
[2007-09-27 09:32:01 xend.XendDomainInfo 27175] DEBUG
(XendDomainInfo:397) parseConfig: result is {'shadow_memory': 3,
'start_time': 1190888774.28, 'uuid':
'b9527e38-a857-aa8f-3e81-97c2e92c8ef5', 'on_crash': 'restart',
'on_reboot': 'restart', 'localtime': None, 'image': ['hvm', ['kernel',
'/usr/lib/xen/boot/hvmloader'], ['device_model',
'/usr/lib64/xen/bin/qemu-dm'], ['pae', '1'], ['vcpus', '1'], ['boot',
'c'], ['serial', 'pty'], ['vnc', '1'], ['vncunused', '1'],
['xauthority', '/root/.Xauthority'], ['acpi', '1'], ['apic', '1'],
['vncpasswd', 'None']], 'on_poweroff': 'destroy', 'bootloader_args':
None, 'cpus': None, 'name': 'testwww', 'backend': [], 'vcpus': 1,
'cpu_weight': 1.0, 'features': None, 'vcpu_avail': 1, 'memory': 500,
'device': [('vif', ['vif', ['backend', '0'], ['script', 'vif-bridge'],
['bridge', 'xenbr0'], ['mac', '00:16:3e:35:f8:cc'], ['type', 'ioemu']]),
('vbd', ['vbd', ['backend', '0'], ['dev', 'hda:disk'], ['uname',
'phy:/dev/devVMvg/test-www-root'], ['mode', 'w']]), ('vkbd', ['vkbd',
['backend', '0']]), ('vfb', ['vfb', ['backend', '0'], ['type', 'vnc'],
['vncunused', '1'], ['xauthority', '/root/.Xauthority']])],
'bootloader': None, 'cpu': None, 'maxmem': 500}
[2007-09-27 09:32:01 xend.XendDomainInfo 27175] DEBUG
(XendDomainInfo:1264) XendDomainInfo.construct: None
[2007-09-27 09:32:01 xend.XendDomainInfo 27175] DEBUG
(XendDomainInfo:715) Storing VM details: {'shadow_memory': '3', 'uuid':
'b9527e38-a857-aa8f-3e81-97c2e92c8ef5', 'on_reboot': 'restart',
'start_time': '1190888774.28', 'on_poweroff': 'destroy', 'name':
'testwww', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1',
'memory': '500', 'on_crash': 'restart', 'image': '(hvm (kernel
/usr/lib/xen/boot/hvmloader) (device_model /usr/lib64/xen/bin/qemu-dm)
(pae 1) (vcpus 1) (boot c) (serial pty) (vnc 1) (vncunused 1)
(xauthority /root/.Xauthority) (acpi 1) (apic 1) (vncpasswd None))',
'maxmem': '500'}
[2007-09-27 09:32:01 xend 27175] DEBUG (DevController:110)
DevController: writing {'state': '1', 'backend-id': '0', 'backend':
'/local/domain/0/backend/vif/5/0'} to /local/domain/5/device/vif/0.
[2007-09-27 09:32:01 xend 27175] DEBUG (DevController:112)
DevController: writing {'bridge': 'xenbr0', 'domain': 'testwww',
'handle': '0', 'script': '/etc/xen/scripts/vif-bridge', 'state': '1',
'frontend': '/local/domain/5/device/vif/0', 'mac': '00:16:3e:35:f8:cc',
'online': '1', 'frontend-id': '5', 'type': 'ioemu'} to
/local/domain/0/backend/vif/5/0.
[2007-09-27 09:32:01 xend 27175] DEBUG (DevController:110)
DevController: writing {'backend-id': '0', 'virtual-device': '768',
'device-type': 'disk', 'state': '1', 'backend':
'/local/domain/0/backend/vbd/5/768'} to /local/domain/5/device/vbd/768.
[2007-09-27 09:32:01 xend 27175] DEBUG (DevController:112)
DevController: writing {'domain': 'testwww', 'frontend':
'/local/domain/5/device/vbd/768', 'dev': 'hda', 'state': '1', 'params':
'/dev/devVMvg/test-www-root', 'mode': 'w', 'online': '1', 'frontend-id':
'5', 'type': 'phy'} to /local/domain/0/backend/vbd/5/768.
[2007-09-27 09:32:01 xend 27175] DEBUG (DevController:110)
DevController: writing {'state': '1', 'backend-id': '0', 'backend':
'/local/domain/0/backend/vkbd/5/0'} to /local/domain/5/device/vkbd/0.
[2007-09-27 09:32:01 xend 27175] DEBUG (DevController:112)
DevController: writing {'frontend-id': '5', 'domain': 'testwww',
'frontend': '/local/domain/5/device/vkbd/0', 'state': '1', 'online':
'1'} to /local/domain/0/backend/vkbd/5/0.
[2007-09-27 09:32:01 xend 27175] DEBUG (DevController:110)
DevController: writing {'state': '1', 'backend-id': '0', 'backend':
'/local/domain/0/backend/vfb/5/0'} to /local/domain/5/device/vfb/0.
[2007-09-27 09:32:01 xend 27175] DEBUG (DevController:112)
DevController: writing {'vncunused': '1', 'domain': 'testwww',
'frontend': '/local/domain/5/device/vfb/0', 'xauthority':
'/root/.Xauthority', 'state': '1', 'online': '1', 'frontend-id': '5',
'type': 'vnc'} to /local/domain/0/backend/vfb/5/0.
[2007-09-27 09:32:01 xend 27175] DEBUG (vfbif:92) No VNC passwd
configured for vfb access
[2007-09-27 09:32:01 xend.XendDomainInfo 27175] DEBUG
(XendDomainInfo:750) Storing domain details: {'console/port': '2',
'name': 'testwww', 'console/limit': '1048576', 'vm':
'/vm/b9527e38-a857-aa8f-3e81-97c2e92c8ef5', 'domid': '5',
'cpu/0/availability': 'online', 'memory/target': '512000', 'store/port':
'1'}
[2007-09-27 09:32:01 xend 27175] ERROR (XendCheckpoint:48) read_exact:
EOF trying to read 8 (buf='')
[2007-09-27 09:32:01 xend.XendDomainInfo 27175] DEBUG
(XendDomainInfo:1463) XendDomainInfo.destroy: domid=5
[2007-09-27 09:32:01 xend.XendDomainInfo 27175] DEBUG
(XendDomainInfo:1471) XendDomainInfo.destroyDomain(5)
[2007-09-27 09:32:01 xend 27175] ERROR (XendDomain:268) Restore failed
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line
263, in domain_restore_fd
    return XendCheckpoint.restore(self, fd)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py",
line 144, in restore
    "not a valid guest state file: pfn count read")
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py",
line 50, in read_exact
    raise XendError(errmsg)
XendError: not a valid guest state file: pfn count read

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users