[Xen-users] Problems with xm migrate --live

[Matthew Crocker <mcrocker@xxxxxxxxxxx>]

Hello,

 I have 2 Dell 1955 blade servers, running RHEL5-Xen.  I'm testing  
the migrate functionality from one blade to another.  I can start the  
domain,  move it to one blade (minor delay/packet loss) and everything  
is fine.  When I try to move it back to the original blade the  
migration fails and the DomU crashes
c1b1 = Blade 1  (192.168.131.201)
c1b2 = Blade 2  (192.168.131.202)
DomU = Primus

I 'xm create Primus' on c1b2,  wait until it is up and running.

Then I do 'xm migrate --live Primus 192.168.131.201`.  The migration  
takes about 10 seconds and everything over to c1b1
When I do a 'xm migrate --live Primus 192.168.131.202' the migration  
starts,  I see the DomU in 'xm list' , pings stop.  It sits there for  
a bit then the DomU drops off the list on c1b2 and stays on c1b1
Primus lives on an iscsi device

/etc/xen/Primus is the same on both machines

name = "Primus"
uuid = "59c6af94-1336-b3f3-aa6a-e5caa919701c"
maxmem = 2000
memory = 2000
vcpus = 1
bootloader = "/usr/bin/pygrub"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
vfb = [  ]
disk = [ "phy:/dev/iscsi/36001c23000d621530000375147fe5837:,xvda,w" ]
vif = [ "mac=00:16:3e:2c:c9:83,bridge=vlan70" ]

Do you think it could be the : in the iscsi device name screwing  
something up in the disk = line?
[root@c1b1 ~]# ls -al /dev/iscsi/36001c23000d621530000375147fe5837\:*
lrwxrwxrwx 1 root root 6 Apr 11 13:18 /dev/iscsi/ 
36001c23000d621530000375147fe5837: -> ../sdd
lrwxrwxrwx 1 root root 7 Apr 11 13:18 /dev/iscsi/ 
36001c23000d621530000375147fe5837:1 -> ../sdd1
lrwxrwxrwx 1 root root 7 Apr 11 13:18 /dev/iscsi/ 
36001c23000d621530000375147fe5837:2 -> ../sdd2
[root@c1b2 ~]# ls -al /dev/iscsi/36001c23000d621530000375147fe5837\:*
lrwxrwxrwx 1 root root 6 Apr 12 16:24 /dev/iscsi/ 
36001c23000d621530000375147fe5837: -> ../sdd
lrwxrwxrwx 1 root root 7 Apr 12 16:24 /dev/iscsi/ 
36001c23000d621530000375147fe5837:1 -> ../sdd1
lrwxrwxrwx 1 root root 7 Apr 12 16:24 /dev/iscsi/ 
36001c23000d621530000375147fe5837:2 -> ../sdd2
c1b2  xend.log shows

[2008-04-12 21:20:58 xend 9579] DEBUG (tcp:66) Match 192.168.131.201  
192.168.131.*[2008-04-12 21:20:58 xend.XendDomainInfo 9579] DEBUG  
(XendDomainInfo:275) XendDomainInfo.restore(['domain', ['domid', '5'],  
['uuid', '59c6af94-1336-b3f3-aa6a-e5caa919701c'], ['vcpus', '1'],  
['vcpu_ava
il', '1'], ['cpu_weight', '1.0'], ['memory', '2000'],  
['shadow_memory', '0'], ['maxmem', '2000'], ['bootloader', '/usr/bin/ 
pygrub'], ['features'], ['name', 'Primus'], ['on_poweroff',  
'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'],  
['image', ['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.pXPTz6'],  
['kernel', '/var/lib/xen/boot_kernel.pRGBF7'], ['args', 'ro root=/dev/ 
VolGroup00/LogV
ol00 console=xvc0 rhgb quiet']]], ['device', ['vif', ['backend', '0'],  
['script', 'vif-bridge'], ['bridge', 'vlan70'], ['mac', '00:16:3e: 
2c:c9:83']]], ['device', ['vbd', ['backend', '0'], ['dev',  
'xvda:disk'], ['uname', 'phy:/dev/iscsi/ 
36001c23000d621530000375147fe5837:'], ['mode', 'w']]], ['state', '- 
b----'], ['shutdown_reason', 'poweroff'], ['cpu_time', '1.213463491'],  
['online_vcpus', '1'], ['up_time', '-81398.7237072'], ['start_time',  
'1208048492.91'], ['store_mfn', '1618905'], ['console_mfn',  
'1618606']])[2008-04-12 21:20:58 xend.XendDomainInfo 9579] DEBUG  
(XendDomainInfo:306) parseConfig: config is ['domain', ['domid', '5'],  
['uuid', '59c6af94-1336-b3f3-aa6a-e5caa919701c'], ['vcpus', '1'],  
['vcpu_avail', '1'], ['cpu_weight', '1.0'], ['memory', '2000'],  
['shadow_memory', '0'], ['maxmem', '2000'], ['bootloader', '/usr/bin/ 
pygrub'], ['features'], ['name', 'Primus'], ['on_poweroff',  
'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'],  
['image', ['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.pXPTz6'],  
['kernel', '/var/lib/xen/boot_kernel.pRGBF7'], ['args', 'ro root=/dev/ 
VolGroup00/LogV
ol00 console=xvc0 rhgb quiet']]], ['device', ['vif', ['backend', '0'],  
['script', 'vif-bridge'], ['bridge', 'vlan70'], ['mac', '00:16:3e: 
2c:c9:83']]], ['device', ['vbd', ['backend', '0'], ['dev',  
'xvda:disk'], ['uname', 'phy:/dev/iscsi/ 
36001c23000d621530000375147fe5837:'], ['mode', 'w']]], ['state', '- 
b----'], ['shutdown_reason', 'poweroff'], ['cpu_time', '1.213463491'],  
['online_vcpus', '1'], ['up_time', '-81398.7237072'], ['start_time',  
'1208048492.91'], ['store_mfn', '1618905'], ['console_mfn', '1618606']]
[2008-04-12 21:20:58 xend.XendDomainInfo 9579] DEBUG (XendDomainInfo: 
411) parseConfig: result is {'shadow_memory': 0, 'start_time':  
1208048492.9100001, 'uuid': '59c6af94-1336-b3f3-aa6a-e5caa919701c',  
'on_crash': 'restart', 'on_reboot': 'restart', 'localtime': None,  
'image': ['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.pXPTz6'],  
['kernel', '/var/lib/xen/boot_kernel.pRGBF7'], ['args', 'ro root=/
dev/VolGroup00/LogVol00 console=xvc0 rhgb quiet']], 'on_poweroff':  
'destroy', 'bootloader_args': None, 'cpus': None, 'name': 'Primus',  
'backend': [], 'vcpus': 1, 'cpu_weight': 1.0, 'features': None,  
'vcpu_avail': 1, 'memory': 2000, 'device': [('vif', ['vif',  
['backend', '0'], ['script', 'vif-bridge'], ['bridge', 'vlan70'],  
['mac', '00:16:3e:2c:c9:83']]), ('vbd', ['vbd', ['backend', '0'],  
['dev', 'xvda:disk'], ['uname', 'phy:/dev/iscsi/ 
36001c23000d621530000375147fe5837:'], ['mode', 'w']])], 'bootloader':  
'/usr/bin/pygrub', 'cpu': None, 'maxmem': 2000}[2008-04-12 21:20:58  
xend.XendDomainInfo 9579] DEBUG (XendDomainInfo:1349)  
XendDomainInfo.construct: None
[2008-04-12 21:20:58 xend 9579] DEBUG (balloon:127) Balloon: 2057220  
KiB free; need 2048; done.[2008-04-12 21:20:58 xend.XendDomainInfo  
9579] DEBUG (XendDomainInfo:756) Storing VM details: {'shadow_memory':  
'0', 'uuid': '59c6af94-1336-b3f3-aa6a-e5caa919701c', 'on_reboot':  
'restart', 'start_time
': '1208048492.91', 'on_poweroff': 'destroy', 'name': 'Primus',  
'vcpus': '1', 'vcpu_avail': '1', 'memory': '2000', 'on_crash':  
'restart', 'image': "(linux (ramdisk /var/lib/xen/boot_ramdisk.pXPTz6)  
(k
ernel /var/lib/xen/boot_kernel.pRGBF7) (args 'ro root=/dev/VolGroup00/ 
LogVol00 console=xvc0 rhgb quiet'))", 'maxmem': '2000'}[2008-04-12  
21:20:58 xend 9579] DEBUG (DevController:110) DevController: writing  
{'backend-id': '0', 'mac': '00:16:3e:2c:c9:83', 'handle': '0',  
'state': '1', 'backend': '/local/domain/0/backend/vif/6/
0'} to /local/domain/6/device/vif/0.[2008-04-12 21:20:58 xend 9579]  
DEBUG (DevController:112) DevController: writing {'bridge': 'vlan70',  
'domain': 'Primus', 'handle': '0', 'script': '/etc/xen/scripts/vif- 
bridge', 'state': '1', 'fronten
d': '/local/domain/6/device/vif/0', 'mac': '00:16:3e:2c:c9:83',  
'online': '1', 'frontend-id': '6'} to /local/domain/0/backend/vif/6/0.
[2008-04-12 21:20:58 xend 9579] DEBUG (blkif:24) exception looking up  
device number for xvda: [Errno 2] No such file or directory: '/dev/ 
xvda'[2008-04-12 21:20:58 xend 9579] DEBUG (DevController:110)  
DevController: writing {'backend-id': '0', 'virtual-device': '51712',  
'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend
/vbd/6/51712'} to /local/domain/6/device/vbd/51712.
[2008-04-12 21:20:58 xend 9579] DEBUG (DevController:112)  
DevController: writing {'domain': 'Primus', 'frontend': '/local/domain/ 
6/device/vbd/51712', 'dev': 'xvda', 'state': '1', 'params': '/dev/ 
iscsi/36001c23000d621530000375147fe5837:', 'mode': 'w', 'online': '1',  
'frontend-id': '6', 'type': 'phy'} to /local/domain/0/backend/vbd/ 
6/51712.
[2008-04-12 21:20:58 xend.XendDomainInfo 9579] DEBUG (XendDomainInfo: 
791) Storing domain details: {'console/port': '2', 'name': 'Primus',  
'console/limit': '1048576', 'vm': '/vm/59c6af94-1336-b3f3-aa6a
-e5caa919701c', 'domid': '6', 'cpu/0/availability': 'online', 'memory/ 
target': '2048000', 'store/port': '1'}
[2008-04-12 21:20:58 xend 9579] DEBUG (XendCheckpoint:198)  
restore:shadow=0x0, _static_max=0x7d0, _static_min=0x7d0, [2008-04-12  
21:20:58 xend 9579] DEBUG (balloon:127) Balloon: 2057212 KiB free;  
need 2048000; done.
[2008-04-12 21:20:58 xend 9579] DEBUG (XendCheckpoint:215)  
[xc_restore]: /usr/lib64/xen/bin/xc_restore 19 6 1 2 0 0 0
[2008-04-12 21:21:29 xend 9579] INFO (XendCheckpoint:351) ERROR  
Internal error: read: p2m_size[2008-04-12 21:21:29 xend 9579] INFO  
(XendCheckpoint:351) Restore exit with rc=1
[2008-04-12 21:21:29 xend.XendDomainInfo 9579] DEBUG (XendDomainInfo: 
1557) XendDomainInfo.destroy: domid=6
[2008-04-12 21:21:29 xend.XendDomainInfo 9579] DEBUG (XendDomainInfo: 
1566) XendDomainInfo.destroyDomain(6)[2008-04-12 21:21:29  
xend.XendDomainInfo 9579] ERROR (XendDomainInfo:1572)  
XendDomainInfo.destroy: xc.domain_destroy failed.
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/xend/ 
XendDomainInfo.py", line 1570, in destroyDomain
    xc.domain_destroy(self.domid)
Error: (3, 'No such process')
[2008-04-12 21:21:29 xend 9579] ERROR (XendDomain:268) Restore failed
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py",  
line 263, in domain_restore_fd
    return XendCheckpoint.restore(self, fd)
  File "/usr/lib64/python2.4/site-packages/xen/xend/ 
XendCheckpoint.py", line 219, in restore
    forkHelper(cmd, fd, handler.handler, True)
  File "/usr/lib64/python2.4/site-packages/xen/xend/ 
XendCheckpoint.py", line 339, in forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib64/xen/bin/xc_restore 19 6 1 2 0 0 0 failed


root@c1b2 ~]# rpm -qa | grep xen
xen-3.0.3-41.el5
xen-libs-3.0.3-41.el5
xen-libs-3.0.3-41.el5
kernel-xen-devel-2.6.18-53.1.14.el5
kernel-xen-2.6.18-53.1.14.el5
[root@c1b2 ~]# uname -a
Linux c1b2.crocker.com 2.6.18-53.1.14.el5xen #1 SMP Tue Feb 19  
07:33:17 EST 2008 x86_64 x86_64 x86_64 GNU/Linux

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users