Re: [Xen-users] Live migration issues

[austin trask <austin.trask@xxxxxxxxxxxx>]

well I think I figured it out.   It was a matter of having the xen 
images in different directories on the different hosts.  for some reason 
it did not like it if the directories were:
/xen-images/xen-image.img   on xen900  and
/var/xen/xen-image.img on xen901  
once I created similar directories on both machines live migration 
worked like a charm.  IE
/var/xen/xen-image.img on both xen901 and xen900

thanks for xen
- austin




Mark Nielsen wrote:
> looks to me like the destination can't see the guest's storage device
>
> [2007-09-20 10:15:45 xend 6492] DEBUG (blkif:24) exception looking up 
> device number for hda1: [Errno 2] No such file or directory: '/dev/hda1'
> Mark
>
>
>
> austin trask wrote:
> > I have been having a issue recently with the xm migrate feature.  It 
> > seems to me that the migration begins to take place however somewhere 
> > in the middle of the migration things seem to fail.
> > I run this command..
> > root@xen900:/xen-images# xm migrate --live feisty_base 192.168.0.1
> >
> >
> > and to check and see if the system has made the connection I did
> > root@xen901:/var/log/xen# netstat -an|grep 8002
> > tcp        0      0 0.0.0.0:8002            0.0.0.0:*               
> > LISTEN    tcp        0      0 192.168.0.1:8002    192.168.0.2:60671   
> > ESTABLISHED
> > this is from the other xen host.
> >
> > after a few moments the xm migrate command completes.   Reviewing the 
> > logs it seems to be a issue on the receiving host.  I am running the 
> > host machines with ubuntu 7.04.
> > there are a few logs on both machines.. from xen900
> > /var/log/xen/xen-hotplug.log reads
> > > Nothing to flush.
> > > Nothing to flush.
> > > Nothing to flush.
> > > Nothing to flush.
> > > SIOCGIFFLAGS: No such device
> > > xenstore-read: couldn't read path backend/vbd/5/769/node
> > > xenstore-read: couldn't read path backend/vbd/5/770/node
> > > Nothing to flush.
> > > Nothing to flush.
> > > xenstore-read: couldn't read path backend/vbd/7/769/node
> > > xenstore-read: couldn't read path backend/vbd/7/770/node
> > > Nothing to flush.
> > > Nothing to flush.
> > > xenstore-read: couldn't read path backend/vbd/9/770/node
> > > xenstore-read: couldn't read path backend/vbd/9/769/node
> > > Nothing to flush.
> > > Nothing to flush.
> > (xen900)/var/xen/xend.log reads
> >
> > [2007-09-20 10:16:32 xend 15789] INFO (XendCheckpoint:239) 4: sent 
> > 81, skipped 0, delta 9ms, dom0 100%, target 0%, sent 294Mb/s, dirtied 
> > 294Mb/s $
> > [2007-09-20 10:16:32 xend 15789] INFO (XendCheckpoint:239) Total 
> > pages sent= 130884 (0.98x)
> > [2007-09-20 10:16:32 xend 15789] INFO (XendCheckpoint:239) (of which 
> > 0 were fixups)
> > [2007-09-20 10:16:32 xend 15789] INFO (XendCheckpoint:239) All memory 
> > is saved
> > [2007-09-20 10:16:32 xend 15789] INFO (XendCheckpoint:239) Save exit 
> > rc=0
> > [2007-09-20 10:16:32 xend.XendDomainInfo 15789] DEBUG 
> > (XendDomainInfo:1466) XendDomainInfo.destroyDomain(11)
> > [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO 
> > (XendDomainInfo:1564) Dev 769 still active, looping...
> > [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO 
> > (XendDomainInfo:1564) Dev 770 still active, looping...
> > [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO 
> > (XendDomainInfo:1564) Dev 769 still active, looping...
> > [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO 
> > (XendDomainInfo:1564) Dev 770 still active, looping...
> > [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO 
> > (XendDomainInfo:1564) Dev 769 still active, looping...
> > [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO 
> > (XendDomainInfo:1564) Dev 770 still active, looping...
> >
> > (xen900)/var/xen/xend-debug.log reads(these logs are not timestamped 
> > so I am not sure if they are from this run)
> > Traceback (most recent call last):
> >  File "SocketServer.py", line 464, in process_request_thread
> >    self.finish_request(request, client_address)
> >  File "SocketServer.py", line 254, in finish_request
> >    self.RequestHandlerClass(request, client_address, self)
> >  File "SocketServer.py", line 522, in __init__
> >    self.handle()
> >  File "BaseHTTPServer.py", line 316, in handle
> >    self.handle_one_request()
> >  File "BaseHTTPServer.py", line 310, in handle_one_request
> >    method()
> >  File "/usr/lib/python2.5/site-packages/xen/util/xmlrpclib2.py", line 
> > 66, in do_POST
> >    self.send_response(200)
> >  File "BaseHTTPServer.py", line 368, in send_response
> >    (self.protocol_version, code, message))
> >  File "socket.py", line 262, in write
> >    self.flush()
> >  File "socket.py", line 249, in flush
> >    self._sock.sendall(buffer)
> > error: (32, 'Broken pipe')
> >
> >
> >
> > on xen901 /var/xen/xend.log reads
> >
> > [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG 
> > (XendDomainInfo:265) XendDomainInfo.restore(['domain', ['domid', 
> > '11'], ['uuid', '9e934f3b-1$
> > [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG 
> > (XendDomainInfo:296) parseConfig: config is ['domain', ['domid', 
> > '11'], ['uuid', '9e934f3b-1$
> > [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG 
> > (XendDomainInfo:395) parseConfig: result is {'shadow_memory': 0, 
> > 'uuid': '9e934f3b-10b2-a40f$
> > [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG 
> > (XendDomainInfo:1262) XendDomainInfo.construct: None
> > [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG 
> > (XendDomainInfo:713) Storing VM details: {'shadow_memory': '0', 
> > 'uuid': '9e934f3b-10b2-a40f-$
> > [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:110) 
> > DevController: writing {'backend-id': '0', 'mac': 
> > '00:16:3e:5b:e6:66', 'handle': '0', '$
> > [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:112) 
> > DevController: writing {'bridge': 'xenbr0', 'domain': 'feisty_base', 
> > 'handle': '0', 'sc$
> > [2007-09-20 10:15:45 xend 6492] DEBUG (blkif:24) exception looking up 
> > device number for hda1: [Errno 2] No such file or directory: '/dev/hda1'
> > [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:110) 
> > DevController: writing {'backend-id': '0', 'virtual-device': '769', 
> > 'device-type': 'dis$
> > [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:112) 
> > DevController: writing {'domain': 'feisty_base', 'frontend': 
> > '/local/domain/6/device/vb$
> > [2007-09-20 10:15:45 xend 6492] DEBUG (blkif:24) exception looking up 
> > device number for hda2: [Errno 2] No such file or directory: '/dev/hda2'
> > [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:110) 
> > DevController: writing {'backend-id': '0', 'virtual-device': '770', 
> > 'device-type': 'dis$
> > [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:112) 
> > DevController: writing {'domain': 'feisty_base', 'frontend': 
> > '/local/domain/6/device/vb$
> > [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG 
> > (XendDomainInfo:748) Storing domain details: {'console/port': '2', 
> > 'name': 'feisty_base', 'c$
> > [2007-09-20 10:15:45 xend 6492] DEBUG (balloon:127) Balloon: 533004 
> > KiB free; need 532480; done.
> > [2007-09-20 10:15:45 xend 6492] DEBUG (XendCheckpoint:155) 
> > [xc_restore]: /usr/lib/xen/bin/xc_restore 18 6 133120 1 2
> > [2007-09-20 10:15:45 xend 6492] INFO (XendCheckpoint:239) 
> > xc_linux_restore start: max_pfn = 20800
> > [2007-09-20 10:15:45 xend 6492] INFO (XendCheckpoint:239) Increased 
> > domain reservation by 82000 KB
> > [2007-09-20 10:15:45 xend 6492] INFO (XendCheckpoint:239) Reloading 
> > memory pages:   0%
> > [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) Received 
> > all pages (0 races)
> > [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) ^H^H^H^H100%
> > [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) Memory 
> > reloaded.
> > [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) Decreased 
> > reservation by 2409 pages
> > [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) Domain 
> > ready to be built.
> > [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) Restore 
> > exit with rc=0
> > [2007-09-20 10:16:30 xend 6492] DEBUG (XendCheckpoint:209) store-mfn 
> > 565003
> > [2007-09-20 10:16:30 xend 6492] DEBUG (XendCheckpoint:209) 
> > console-mfn 561602
> > [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:143) Waiting for 
> > devices vif.
> > [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:149) Waiting for 0.
> > [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:464) 
> > hotplugStatusCallback /local/domain/0/backend/vif/6/0/hotplug-status.
> > [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:478) 
> > hotplugStatusCallback 1.
> > [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:143) Waiting for 
> > devices usb.
> > [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:143) Waiting for 
> > devices vbd.
> > [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:149) Waiting for 
> > 769.
> > [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:464) 
> > hotplugStatusCallback /local/domain/0/backend/vbd/6/769/hotplug-status.
> > [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:478) 
> > hotplugStatusCallback 2.
> > [2007-09-20 10:16:31 xend.XendDomainInfo 6492] DEBUG 
> > (XendDomainInfo:1458) XendDomainInfo.destroy: domid=6
> > [2007-09-20 10:16:31 xend.XendDomainInfo 6492] DEBUG 
> > (XendDomainInfo:1466) XendDomainInfo.destroyDomain(6)
> > [2007-09-20 10:16:31 xend 6492] ERROR (__init__:1021) Restore failed
> > Traceback (most recent call last):
> >  File "/usr/lib/python2.5/site-packages/xen/xend/XendDomain.py", line 
> > 263, in domain_restore_fd
> >    return XendCheckpoint.restore(self, fd)
> >  File "/usr/lib/python2.5/site-packages/xen/xend/XendCheckpoint.py", 
> > line 165, in restore
> >    dominfo.waitForDevices() # Wait for backends to set up
> >  File "/usr/lib/python2.5/site-packages/xen/xend/XendDomainInfo.py", 
> > line 1605, in waitForDevices
> >    self.waitForDevices_(c)
> >  File "/usr/lib/python2.5/site-packages/xen/xend/XendDomainInfo.py", 
> > line 1099, in waitForDevices_
> >    return self.getDeviceController(deviceClass).waitForDevices()
> >  File 
> > "/usr/lib/python2.5/site-packages/xen/xend/server/DevController.py", 
> > line 145, in waitForDevices
> >    return map(self.waitForDevice, self.deviceIDs())
> >  File 
> > "/usr/lib/python2.5/site-packages/xen/xend/server/DevController.py", 
> > line 163, in waitForDevice
> >    (devid, self.deviceClass))
> > VmError: Device 769 (vbd) could not be connected. Backend device not 
> > found.
> >
> > there is nothing notable in the other logs on xen901. 
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@xxxxxxxxxxxxxxxxxxx
> > http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users