Re: [Xen-users] Re: XEN live migration: cannot console or ssh to the migrated guest VM (domU)

[Hitesh Khandelwal <hitesh.iitk@xxxxxxxxx>]

The problem is unrelated to networking. I cannot even "xm console" to the VM migrated, from the final dom0. I suspect the problem is with hvc drivers.

Thanks,

Hitesh

Hitesh Khandelwal | Graduate Student | Purdue University | http://www.hiteshkhandelwal.com

On Thu, Feb 17, 2011 at 9:03 PM, Wenda Ni <wonda.ni@gmail.com> wrote:

Yes. Follow Simon's advice, which is copied as follows:

I would suggest you try with something like arping and see if you get any different results. I can think of one thing that would make the VM "disappear" from the network for a while during/after migration apart from the downtime while the migration actually happens :

Every switch (bridge) on the network will have learned the port to which the node is attached and cached it in it's MAC <-> Port table. When you move a machine, these switches will continue to forward unicast packets via the port it has in it's tables UNTIL it has reason to update the table entry. Thus packets may get sent to the wrong place for a while - and so the migrated machine just doesn't get any traffic.

Most switches will update the table as soon as it sees a packet from the device arrive on a different port. Depending on what's going on, that may take a second, or minutes.
Arping uses ARP request packets to probe a device rather than unicast ICP-Ping packets. These being broadcast will be flooded to the entire network and so will reach the device even though it's now on a different port. When it replies, the intermediate switches will update their MAC table accordingly.

Also, I've found some switches can be slow to update. We've some HP 1800-24G switches at work, and they seem to have a 5 minute timeout before they'll update the MAC table. I guess it's probably configurable via CLI - but I haven't looked.
--
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

On Thu, Feb 17, 2011 at 8:24 PM, Hitesh Khandelwal <hitesh.iitk@xxxxxxxxx> wrote:

Wenda Ni <wonda.ni <at> gmail.com> writes:

>
>
> I am now testing XEN live migration on two physical hosts with XEN 4.0.1 pvops
Ubuntu 10.10 (2.6.35-22). Host A also acts as a NFS server, and Host B acts as a
NFS client.
> When I migrate a guest domain from B to A. The ssh connection experiences a
downtime of about 1 minute (the terminal does not react to the keyboard input
till about 1 minute later). However, the "sudo xm console" does not work unless
a reboot is triggered.
> When I migrate the guest domain from A to back B, the situation is even worse
-- neither ssh connection nor console works.
> In both migration cases, you can notice that the guest domain is there by
using "sudo xm list". Could anyone shed some light on the potential problems I
am facing? I have been working on this for quite some time, but have no idea
what goes wrong. The following is the log file copied from host B, when the
guest domain is migrated from A back to B.[2011-02-06 23:35:51 5208] DEBUG
(XendDomainInfo:235) XendDomainInfo.restore(['domain', ['domid', '8'],
['cpu_weight', '256'], ['cpu_cap', '0'], ['bootloader', '/usr/bin/pygrub'],
['on_crash', 'restart'], ['uuid', 'aaf5c3c7-a5a6-d306-dc77-483cea04bff5'],
['bootloader_args', '-q'], ['vcpus', '1'], ['description'], ['name', 'Elite'],
['cpus', [[]]], ['on_reboot', 'restart'], ['on_poweroff', 'destroy'], ['maxmem',
'128'], ['memory', '128'], ['shadow_memory', '0'], ['vcpu_avail', '1'],
['features'], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'],
['start_time', '1297053017.52'], ['cpu_time', '7.409878829'], ['online_vcpus',
'1'], ['image', ['linux', ['kernel'], ['args', 'root=/dev/sda2 ro '],
['superpages', '0'], ['videoram', '4'], ['pci', []], ['nomigrate', '0'],
['tsc_mode', '0'], ['notes', ['HV_START_LOW', '4118806528'], ['FEATURES',
'!writable_page_tables|pae_pgdir_above_4gb'], ['VIRT_BASE', '3221225472'],
['GUEST_VERSION', '2.6'], ['PADDR_OFFSET', '0'], ['GUEST_OS', 'linux'],
['HYPERCALL_PAGE', '3222282240'], ['LOADER', 'generic'], ['SUSPEND_CANCEL',
'1'], ['PAE_MODE', 'yes'], ['ENTRY', '3229974528'], ['XEN_VERSION', 'xen-
3.0']]]], ['status', '2'], ['state', '-b----'], ['store_mfn', '472195'],
['console_mfn', '472194'], ['device', ['vif', ['ip', '138.15.106.226'], ['mac',
'00:16:3E:E9:17:69'], ['script', '/etc/xen/scripts/vif-bridge'], ['uuid',
'b3761d83-4167-d29e-2833-98065be549c8'], ['backend', '0']]], ['device',
['console', ['protocol', 'vt100'], ['location', '2'], ['uuid', '99ef56f7-bbd9-
0a3c-277e-bbcc8f5cebce']]], ['device', ['vbd', ['protocol', 'x86_32-abi'],
['uuid', '638d020f-b8cd-4100-69a0-c3efc5334b08'], ['bootable', '1'], ['dev',
'sda2:disk'], ['uname', 'file:/home/user1/vm-storage-
export/domains/Elite/disk.img'], ['mode', 'w'], ['backend', '0'], ['VDI']]],
['device', ['vbd', ['protocol', 'x86_32-abi'], ['uuid', 'dac29d55-cda5-29c5-
b362-dee4b24e60d6'], ['bootable', '0'], ['dev', 'sda1:disk'], ['uname',
'file:/home/user1/vm-storage-export/domains/Elite/swap.img'], ['mode', 'w'],
['backend', '0'], ['VDI']]], ['change_home_server', 'False']])[2011-02-06
23:35:51 5208] DEBUG (XendDomainInfo:2508) XendDomainInfo.constructDomain[2011-
02-06 23:35:51 5208] DEBUG (balloon:220) Balloon: 263812 KiB free; need 16384;
done.[2011-02-06 23:35:51 5208] DEBUG (XendDomain:464) Adding Domain: 15[2011-
02-06 23:35:51 5208] DEBUG (XendDomainInfo:3400) Storing VM details:
{'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': 'aaf5c3c7-a5a6-d306-
dc77-483cea04bff5', 'on_reboot': 'restart', 'start_time': '1297053017.52',
'on_poweroff': 'destroy', 'bootloader_args': '-q', 'on_xend_start': 'ignore',
'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail':
'1', 'bootloader': '/usr/bin/pygrub', 'image': "(linux (kernel ) (args
'root=/dev/sda2 ro ') (superpages 0) (videoram 4) (pci ()) (nomigrate 0)
(tsc_mode 0) (notes (HV_START_LOW 4118806528) (FEATURES
'!writable_page_tables|pae_pgdir_above_4gb') (VIRT_BASE 3221225472)
(GUEST_VERSION 2.6) (PADDR_OFFSET 0) (GUEST_OS linux) (HYPERCALL_PAGE
3222282240) (LOADER generic) (SUSPEND_CANCEL 1) (PAE_MODE yes) (ENTRY
3229974528) (XEN_VERSION xen-3.0)))", 'name': 'Elite'}[2011-02-06 23:35:51 5208]
INFO (XendDomainInfo:2367) createDevice: console : {'protocol': 'vt100',
'location': '2', 'uuid': '99ef56f7-bbd9-0a3c-277e-bbcc8f5cebce'}[2011-02-06
23:35:51 5208] DEBUG (DevController:95) DevController: writing {'state': '1',
'backend-id': '0', 'backend': '/local/domain/0/backend/console/15/0'} to
/local/domain/15/device/console/0.[2011-02-06 23:35:51 5208] DEBUG
(DevController:97) DevController: writing {'domain': 'Elite', 'frontend':
'/local/domain/15/device/console/0', 'uuid': '99ef56f7-bbd9-0a3c-277e-
bbcc8f5cebce', 'frontend-id': '15', 'state': '1', 'location': '2', 'online':
'1', 'protocol': 'vt100'} to /local/domain/0/backend/console/15/0.[2011-02-06
23:35:51 5208] INFO (XendDomainInfo:2367) createDevice: vbd : {'protocol':
'x86_32-abi', 'uuid': '638d020f-b8cd-4100-69a0-c3efc5334b08', 'bootable': 1,
'driver': 'paravirtualised', 'dev': 'sda2:disk', 'uname': 'file:/home/user1/vm-
storage-export/domains/Elite/disk.img', 'mode': 'w', 'backend': '0'}[2011-02-06
23:35:51 5208] DEBUG (DevController:95) DevController: writing {'virtual-
device': '2050', 'protocol': 'x86_32-abi', 'device-type': 'disk', 'backend-id':
'0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/15/2050'} to
/local/domain/15/device/vbd/2050.[2011-02-06 23:35:51 5208] DEBUG
(DevController:97) DevController: writing {'domain': 'Elite', 'frontend':
'/local/domain/15/device/vbd/2050', 'uuid': '638d020f-b8cd-4100-69a0-
c3efc5334b08', 'bootable': '1', 'dev': 'sda2', 'state': '1', 'params':
'/home/user1/vm-storage-export/domains/Elite/disk.img', 'mode': 'w', 'online':
'1', 'frontend-id': '15', 'type': 'file'} to
/local/domain/0/backend/vbd/15/2050.[2011-02-06 23:35:51 5208] INFO
(XendDomainInfo:2367) createDevice: vbd : {'protocol': 'x86_32-abi', 'uuid':
'dac29d55-cda5-29c5-b362-dee4b24e60d6', 'bootable': 0, 'driver':
'paravirtualised', 'dev': 'sda1:disk', 'uname': 'file:/home/user1/vm-storage-
export/domains/Elite/swap.img', 'mode': 'w', 'backend': '0'}[2011-02-06 23:35:51
5208] DEBUG (DevController:95) DevController: writing {'virtual-device': '2049',
'protocol': 'x86_32-abi', 'device-type': 'disk', 'backend-id': '0', 'state':
'1', 'backend': '/local/domain/0/backend/vbd/15/2049'} to
/local/domain/15/device/vbd/2049.[2011-02-06 23:35:51 5208] DEBUG
(DevController:97) DevController: writing {'domain': 'Elite', 'frontend':
'/local/domain/15/device/vbd/2049', 'uuid': 'dac29d55-cda5-29c5-b362-
dee4b24e60d6', 'bootable': '0', 'dev': 'sda1', 'state': '1', 'params':
'/home/user1/vm-storage-export/domains/Elite/swap.img', 'mode': 'w', 'online':
'1', 'frontend-id': '15', 'type': 'file'} to
/local/domain/0/backend/vbd/15/2049.[2011-02-06 23:35:51 5208] INFO
(XendDomainInfo:2367) createDevice: vif : {'ip': '138.15.106.226', 'mac':
'00:16:3E:E9:17:69', 'script': '/etc/xen/scripts/vif-bridge', 'uuid': 'b3761d83-
4167-d29e-2833-98065be549c8', 'backend': '0'}[2011-02-06 23:35:51 5208] DEBUG
(DevController:95) DevController: writing {'backend-id': '0', 'mac':
'00:16:3E:E9:17:69', 'handle': '0', 'state': '1', 'backend':
'/local/domain/0/backend/vif/15/0'} to /local/domain/15/device/vif/0.[2011-02-06
23:35:51 5208] DEBUG (DevController:97) DevController: writing {'domain':
'Elite', 'handle': '0', 'uuid': 'b3761d83-4167-d29e-2833-98065be549c8',
'script': '/etc/xen/scripts/vif-bridge', 'ip': '138.15.106.226', 'mac':
'00:16:3E:E9:17:69', 'frontend-id': '15', 'state': '1', 'online': '1',
'frontend': '/local/domain/15/device/vif/0'} to
/local/domain/0/backend/vif/15/0.[2011-02-06 23:35:51 5208] DEBUG
(XendDomainInfo:1804) Storing domain details: {'image/entry': '3229974528',
'console/port': '2', 'image/loader': 'generic', 'vm': '/vm/aaf5c3c7-a5a6-d306-
dc77-483cea04bff5', 'control/platform-feature-multiprocessor-suspend': '1',
'image/hv-start-low': '4118806528', 'image/guest-os': 'linux', 'image/virt-
base': '3221225472', 'memory/target': '131072', 'image/guest-version': '2.6',
'image/pae-mode': 'yes', 'description': '', 'console/limit': '1048576',
'image/paddr-offset': '0', 'image/hypercall-page': '3222282240', 'image/suspend-
cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-
4gb': '1', 'image/features/writable-page-tables': '0', 'console/type':
'xenconsoled', 'name': 'Elite', 'domid': '15', 'image/xen-version': 'xen-3.0',
'store/port': '1'}[2011-02-06 23:35:52 5208] DEBUG (XendCheckpoint:286)
restore:shadow=0x0, _static_max=0x8000000, _static_min=0x0, [2011-02-06 23:35:52
5208] DEBUG (XendCheckpoint:305) [xc_restore]: /usr/lib/xen/bin/xc_restore 4 15
1 2 0 0 0 0[2011-02-06 23:35:52 5208] INFO (XendCheckpoint:423)
xc_domain_restore start: p2m_size = 8000[2011-02-06 23:35:52 5208] INFO
(XendCheckpoint:423) Reloading memory pages:   0%[2011-02-06 23:36:03 5208] INFO
(XendCheckpoint:423) ERROR Internal error: Error when reading batch size[2011-
02-06 23:36:03 5208] INFO (XendCheckpoint:423) ERROR Internal error: error when
buffering batch, finishing[2011-02-06 23:36:03 5208] INFO (XendCheckpoint:423)
[2011-02-06 23:36:03 5208] INFO (XendCheckpoint:423) 100%[2011-02-06 23:36:03
5208] INFO (XendCheckpoint:423) Memory reloaded (9067 pages)[2011-02-06 23:36:03
5208] INFO (XendCheckpoint:423) read VCPU 0[2011-02-06 23:36:03 5208] INFO
(XendCheckpoint:423) Completed checkpoint load[2011-02-06 23:36:03 5208] INFO
(XendCheckpoint:423) Domain ready to be built.[2011-02-06 23:36:03 5208] INFO
(XendCheckpoint:423) Restore exit with rc=0[2011-02-06 23:36:03 5208] DEBUG
(XendCheckpoint:394) store-mfn 163723[2011-02-06 23:36:03 5208] DEBUG
(XendCheckpoint:394) console-mfn 163722[2011-02-06 23:36:03 5208] DEBUG
(XendDomainInfo:2992) XendDomainInfo.completeRestore[2011-02-06 23:36:03 5208]
DEBUG (XendDomainInfo:1804) Storing domain details: {'console/ring-ref':
'163722', 'image/entry': '3229974528', 'console/port': '2', 'store/ring> -ref':
'163723', 'image/loader': 'generic', 'vm': '/vm/aaf5c3c7-a5a6-d306-dc77-
483cea04bff5', 'control/platform-feature-multiprocessor-suspend': '1',
'image/hv-start-low': '4118806528', 'image/guest-os': 'linux', 'image/virt-
base': '3221225472', 'memory/target': '131072', 'image/guest-version': '2.6',
'image/pae-mode': 'yes', 'description': '', 'console/limit': '1048576',
'image/paddr-offset': '0', 'image/hypercall-page': '3222282240', 'image/suspend-
cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-
4gb': '1', 'image/features/writable-page-tables': '0', 'console/type':
'xenconsoled', 'name': 'Elite', 'domid': '15', 'image/xen-version': 'xen-3.0',
'store/port': '1'}[2011-02-06 23:36:03 5208] DEBUG (XendDomainInfo:3005)
XendDomainInfo.completeRestore done[2011-02-06 23:36:03 5208] DEBUG
(DevController:139) Waiting for devices tap2.[2011-02-06 23:36:03 5208] DEBUG
(XendDomainInfo:1891) XendDomainInfo.handleShutdownWatch[2011-02-06 23:36:03
5208] DEBUG (DevController:139) Waiting for devices vif.[2011-02-06 23:36:03
5208] DEBUG (DevController:144) Waiting for 0.[2011-02-06 23:36:03 5208] DEBUG
(DevController:628) hotplugStatusCallback
/local/domain/0/backend/vif/15/0/hotplug-status.[2011-02-06 23:36:03 5208] DEBUG
(DevController:642) hotplugStatusCallback 1.[2011-02-06 23:36:03 5208] DEBUG
(DevController:139) Waiting for devices vkbd.[2011-02-06 23:36:03 5208] DEBUG
(DevController:139) Waiting for devices ioports.[2011-02-06 23:36:03 5208] DEBUG
(DevController:139) Waiting for devices tap.[2011-02-06 23:36:03 5208] DEBUG
(DevController:139) Waiting for devices vif2.[2011-02-06 23:36:03 5208] DEBUG
(DevController:139) Waiting for devices console.[2011-02-06 23:36:03 5208] DEBUG
(DevController:144) Waiting for 0.[2011-02-06 23:36:03 5208] DEBUG
(DevController:139) Waiting for devices vscsi.[2011-02-06 23:36:03 5208] DEBUG
(DevController:139) Waiting for devices vbd.[2011-02-06 23:36:03 5208] DEBUG
(DevController:144) Waiting for 2050.[2011-02-06 23:36:03 5208] DEBUG
(DevController:628) hotplugStatusCallback
/local/domain/0/backend/vbd/15/2050/hotplug-status.[2011-02-06 23:36:03 5208]
DEBUG (DevController:642) hotplugStatusCallback 1.[2011-02-06 23:36:03 5208]
DEBUG (DevController:144) Waiting for 2049.[2011-02-06 23:36:03 5208] DEBUG
(DevController:628) hotplugStatusCallback
/local/domain/0/backend/vbd/15/2049/hotplug-status.[2011-02-06 23:36:03 5208]
DEBUG (DevController:642) hotplugStatusCallback 1.[2011-02-06 23:36:03 5208]
DEBUG (DevController:139) Waiting for devices irq.[2011-02-06 23:36:03 5208]
DEBUG (DevController:139) Waiting for devices vfb.[2011-02-06 23:36:03 5208]
DEBUG (DevController:139) Waiting for devices pci.[2011-02-06 23:36:03 5208]
DEBUG (DevController:139) Waiting for devices vusb.[2011-02-06 23:36:03 5208]
DEBUG (DevController:139) Waiting for devices vtpm.-- Cheers,
>  
> Wenda Ni, Ph.D.
> Dept. of Computer Science and Engineering
> State University of New York (SUNY) at Buffalo
> T: 716 982 6746
>
>
>

> _______________________________________________
> Xen-users mailing list
> Xen-users <at> lists.xensource.com
> http://lists.xensource.com/xen-users


Were you able to resolve the problem? I am also experiencing the same problem
when using ubuntu 10.10 as DomU. Console as well as ssh connection both freezes
for 10 mins in both the scenarios (A -> B or B -> A).


Hitesh Khandelwal | Graduate Student | Purdue University |
http://www.cs.purdue.edu/~hkhande


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

--

Cheers,

 

Wenda Ni, Ph.D.

Dept. of Computer Science and Engineering

State University of New York (SUNY) at Buffalo

T: 716 982 6746

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users