Re: [Xen-users] Re: XEN live migration: cannot console or ssh to the migrated guest VM (domU)

[Paras pradhan <pradhanparas@xxxxxxxxx>]

This might not apply to other distros than Redhat 6, but I guess its
a similar bug explained in this thread. see this

https://bugzilla.redhat.com/show_bug.cgi?id=663755

Paras.

On Mon, Feb 21, 2011 at 5:23 PM, Hitesh Khandelwal
<hitesh.iitk@xxxxxxxxx> wrote:
>
> The problem is unrelated to networking. I cannot even "xm console" to the VM
> migrated, from the final dom0. I suspect the problem is with hvc drivers.
> Thanks,
> Hitesh
>
> Hitesh Khandelwal | Graduate Student | Purdue University
> | http://www.hiteshkhandelwal.com
>
>
> On Thu, Feb 17, 2011 at 9:03 PM, Wenda Ni <wonda.ni@xxxxxxxxx> wrote:
>>
>> Yes. Follow Simon's advice, which is copied as follows:
>>
>> I would suggest you try with something like arping and see if you get any
>> different results. I can think of one thing that would make the VM
>> "disappear" from the network for a while during/after migration apart from
>> the downtime while the migration actually happens :
>>
>> Every switch (bridge) on the network will have learned the port to which
>> the node is attached and cached it in it's MAC <-> Port table. When you move
>> a machine, these switches will continue to forward unicast packets via the
>> port it has in it's tables UNTIL it has reason to update the table entry.
>> Thus packets may get sent to the wrong place for a while - and so the
>> migrated machine just doesn't get any traffic.
>>
>> Most switches will update the table as soon as it sees a packet from the
>> device arrive on a different port. Depending on what's going on, that may
>> take a second, or minutes.
>> Arping uses ARP request packets to probe a device rather than unicast
>> ICP-Ping packets. These being broadcast will be flooded to the entire
>> network and so will reach the device even though it's now on a different
>> port. When it replies, the intermediate switches will update their MAC table
>> accordingly.
>>
>> Also, I've found some switches can be slow to update. We've some HP
>> 1800-24G switches at work, and they seem to have a 5 minute timeout before
>> they'll update the MAC table. I guess it's probably configurable via CLI -
>> but I haven't looked.
>> --
>> Simon Hobson
>>
>> Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
>> author Gladys Hobson. Novels - poetry - short stories - ideal as
>> Christmas stocking fillers. Some available as e-books.
>>
>>
>>
>> On Thu, Feb 17, 2011 at 8:24 PM, Hitesh Khandelwal <hitesh.iitk@xxxxxxxxx>
>> wrote:
>>>
>>> Wenda Ni <wonda.ni <at> gmail.com> writes:
>>>
>>> >
>>> >
>>> > I am now testing XEN live migration on two physical hosts with XEN
>>> > 4.0.1 pvops
>>> Ubuntu 10.10 (2.6.35-22). Host A also acts as a NFS server, and Host B
>>> acts as a
>>> NFS client.
>>> > When I migrate a guest domain from B to A. The ssh connection
>>> > experiences a
>>> downtime of about 1 minute (the terminal does not react to the keyboard
>>> input
>>> till about 1 minute later). However, the "sudo xm console" does not work
>>> unless
>>> a reboot is triggered.
>>> > When I migrate the guest domain from A to back B, the situation is even
>>> > worse
>>> -- neither ssh connection nor console works.
>>> > In both migration cases, you can notice that the guest domain is there
>>> > by
>>> using "sudo xm list". Could anyone shed some light on the potential
>>> problems I
>>> am facing? I have been working on this for quite some time, but have no
>>> idea
>>> what goes wrong. The following is the log file copied from host B, when
>>> the
>>> guest domain is migrated from A back to B.[2011-02-06 23:35:51 5208]
>>> DEBUG
>>> (XendDomainInfo:235) XendDomainInfo.restore(['domain', ['domid', '8'],
>>> ['cpu_weight', '256'], ['cpu_cap', '0'], ['bootloader',
>>> '/usr/bin/pygrub'],
>>> ['on_crash', 'restart'], ['uuid',
>>> 'aaf5c3c7-a5a6-d306-dc77-483cea04bff5'],
>>> ['bootloader_args', '-q'], ['vcpus', '1'], ['description'], ['name',
>>> 'Elite'],
>>> ['cpus', [[]]], ['on_reboot', 'restart'], ['on_poweroff', 'destroy'],
>>> ['maxmem',
>>> '128'], ['memory', '128'], ['shadow_memory', '0'], ['vcpu_avail', '1'],
>>> ['features'], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'],
>>> ['start_time', '1297053017.52'], ['cpu_time', '7.409878829'],
>>> ['online_vcpus',
>>> '1'], ['image', ['linux', ['kernel'], ['args', 'root=/dev/sda2 ro '],
>>> ['superpages', '0'], ['videoram', '4'], ['pci', []], ['nomigrate', '0'],
>>> ['tsc_mode', '0'], ['notes', ['HV_START_LOW', '4118806528'], ['FEATURES',
>>> '!writable_page_tables|pae_pgdir_above_4gb'], ['VIRT_BASE',
>>> '3221225472'],
>>> ['GUEST_VERSION', '2.6'], ['PADDR_OFFSET', '0'], ['GUEST_OS', 'linux'],
>>> ['HYPERCALL_PAGE', '3222282240'], ['LOADER', 'generic'],
>>> ['SUSPEND_CANCEL',
>>> '1'], ['PAE_MODE', 'yes'], ['ENTRY', '3229974528'], ['XEN_VERSION', 'xen-
>>> 3.0']]]], ['status', '2'], ['state', '-b----'], ['store_mfn', '472195'],
>>> ['console_mfn', '472194'], ['device', ['vif', ['ip', '138.15.106.226'],
>>> ['mac',
>>> '00:16:3E:E9:17:69'], ['script', '/etc/xen/scripts/vif-bridge'], ['uuid',
>>> 'b3761d83-4167-d29e-2833-98065be549c8'], ['backend', '0']]], ['device',
>>> ['console', ['protocol', 'vt100'], ['location', '2'], ['uuid',
>>> '99ef56f7-bbd9-
>>> 0a3c-277e-bbcc8f5cebce']]], ['device', ['vbd', ['protocol',
>>> 'x86_32-abi'],
>>> ['uuid', '638d020f-b8cd-4100-69a0-c3efc5334b08'], ['bootable', '1'],
>>> ['dev',
>>> 'sda2:disk'], ['uname', 'file:/home/user1/vm-storage-
>>> export/domains/Elite/disk.img'], ['mode', 'w'], ['backend', '0'],
>>> ['VDI']]],
>>> ['device', ['vbd', ['protocol', 'x86_32-abi'], ['uuid',
>>> 'dac29d55-cda5-29c5-
>>> b362-dee4b24e60d6'], ['bootable', '0'], ['dev', 'sda1:disk'], ['uname',
>>> 'file:/home/user1/vm-storage-export/domains/Elite/swap.img'], ['mode',
>>> 'w'],
>>> ['backend', '0'], ['VDI']]], ['change_home_server', 'False']])[2011-02-06
>>> 23:35:51 5208] DEBUG (XendDomainInfo:2508)
>>> XendDomainInfo.constructDomain[2011-
>>> 02-06 23:35:51 5208] DEBUG (balloon:220) Balloon: 263812 KiB free; need
>>> 16384;
>>> done.[2011-02-06 23:35:51 5208] DEBUG (XendDomain:464) Adding Domain:
>>> 15[2011-
>>> 02-06 23:35:51 5208] DEBUG (XendDomainInfo:3400) Storing VM details:
>>> {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid':
>>> 'aaf5c3c7-a5a6-d306-
>>> dc77-483cea04bff5', 'on_reboot': 'restart', 'start_time':
>>> '1297053017.52',
>>> 'on_poweroff': 'destroy', 'bootloader_args': '-q', 'on_xend_start':
>>> 'ignore',
>>> 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '1',
>>> 'vcpu_avail':
>>> '1', 'bootloader': '/usr/bin/pygrub', 'image': "(linux (kernel ) (args
>>> 'root=/dev/sda2 ro ') (superpages 0) (videoram 4) (pci ()) (nomigrate 0)
>>> (tsc_mode 0) (notes (HV_START_LOW 4118806528) (FEATURES
>>> '!writable_page_tables|pae_pgdir_above_4gb') (VIRT_BASE 3221225472)
>>> (GUEST_VERSION 2.6) (PADDR_OFFSET 0) (GUEST_OS linux) (HYPERCALL_PAGE
>>> 3222282240) (LOADER generic) (SUSPEND_CANCEL 1) (PAE_MODE yes) (ENTRY
>>> 3229974528) (XEN_VERSION xen-3.0)))", 'name': 'Elite'}[2011-02-06
>>> 23:35:51 5208]
>>> INFO (XendDomainInfo:2367) createDevice: console : {'protocol': 'vt100',
>>> 'location': '2', 'uuid':
>>> '99ef56f7-bbd9-0a3c-277e-bbcc8f5cebce'}[2011-02-06
>>> 23:35:51 5208] DEBUG (DevController:95) DevController: writing {'state':
>>> '1',
>>> 'backend-id': '0', 'backend': '/local/domain/0/backend/console/15/0'} to
>>> /local/domain/15/device/console/0.[2011-02-06 23:35:51 5208] DEBUG
>>> (DevController:97) DevController: writing {'domain': 'Elite', 'frontend':
>>> '/local/domain/15/device/console/0', 'uuid': '99ef56f7-bbd9-0a3c-277e-
>>> bbcc8f5cebce', 'frontend-id': '15', 'state': '1', 'location': '2',
>>> 'online':
>>> '1', 'protocol': 'vt100'} to
>>> /local/domain/0/backend/console/15/0.[2011-02-06
>>> 23:35:51 5208] INFO (XendDomainInfo:2367) createDevice: vbd :
>>> {'protocol':
>>> 'x86_32-abi', 'uuid': '638d020f-b8cd-4100-69a0-c3efc5334b08', 'bootable':
>>> 1,
>>> 'driver': 'paravirtualised', 'dev': 'sda2:disk', 'uname':
>>> 'file:/home/user1/vm-
>>> storage-export/domains/Elite/disk.img', 'mode': 'w', 'backend':
>>> '0'}[2011-02-06
>>> 23:35:51 5208] DEBUG (DevController:95) DevController: writing {'virtual-
>>> device': '2050', 'protocol': 'x86_32-abi', 'device-type': 'disk',
>>> 'backend-id':
>>> '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/15/2050'} to
>>> /local/domain/15/device/vbd/2050.[2011-02-06 23:35:51 5208] DEBUG
>>> (DevController:97) DevController: writing {'domain': 'Elite', 'frontend':
>>> '/local/domain/15/device/vbd/2050', 'uuid': '638d020f-b8cd-4100-69a0-
>>> c3efc5334b08', 'bootable': '1', 'dev': 'sda2', 'state': '1', 'params':
>>> '/home/user1/vm-storage-export/domains/Elite/disk.img', 'mode': 'w',
>>> 'online':
>>> '1', 'frontend-id': '15', 'type': 'file'} to
>>> /local/domain/0/backend/vbd/15/2050.[2011-02-06 23:35:51 5208] INFO
>>> (XendDomainInfo:2367) createDevice: vbd : {'protocol': 'x86_32-abi',
>>> 'uuid':
>>> 'dac29d55-cda5-29c5-b362-dee4b24e60d6', 'bootable': 0, 'driver':
>>> 'paravirtualised', 'dev': 'sda1:disk', 'uname':
>>> 'file:/home/user1/vm-storage-
>>> export/domains/Elite/swap.img', 'mode': 'w', 'backend': '0'}[2011-02-06
>>> 23:35:51
>>> 5208] DEBUG (DevController:95) DevController: writing {'virtual-device':
>>> '2049',
>>> 'protocol': 'x86_32-abi', 'device-type': 'disk', 'backend-id': '0',
>>> 'state':
>>> '1', 'backend': '/local/domain/0/backend/vbd/15/2049'} to
>>> /local/domain/15/device/vbd/2049.[2011-02-06 23:35:51 5208] DEBUG
>>> (DevController:97) DevController: writing {'domain': 'Elite', 'frontend':
>>> '/local/domain/15/device/vbd/2049', 'uuid': 'dac29d55-cda5-29c5-b362-
>>> dee4b24e60d6', 'bootable': '0', 'dev': 'sda1', 'state': '1', 'params':
>>> '/home/user1/vm-storage-export/domains/Elite/swap.img', 'mode': 'w',
>>> 'online':
>>> '1', 'frontend-id': '15', 'type': 'file'} to
>>> /local/domain/0/backend/vbd/15/2049.[2011-02-06 23:35:51 5208] INFO
>>> (XendDomainInfo:2367) createDevice: vif : {'ip': '138.15.106.226', 'mac':
>>> '00:16:3E:E9:17:69', 'script': '/etc/xen/scripts/vif-bridge', 'uuid':
>>> 'b3761d83-
>>> 4167-d29e-2833-98065be549c8', 'backend': '0'}[2011-02-06 23:35:51 5208]
>>> DEBUG
>>> (DevController:95) DevController: writing {'backend-id': '0', 'mac':
>>> '00:16:3E:E9:17:69', 'handle': '0', 'state': '1', 'backend':
>>> '/local/domain/0/backend/vif/15/0'} to
>>> /local/domain/15/device/vif/0.[2011-02-06
>>> 23:35:51 5208] DEBUG (DevController:97) DevController: writing {'domain':
>>> 'Elite', 'handle': '0', 'uuid': 'b3761d83-4167-d29e-2833-98065be549c8',
>>> 'script': '/etc/xen/scripts/vif-bridge', 'ip': '138.15.106.226', 'mac':
>>> '00:16:3E:E9:17:69', 'frontend-id': '15', 'state': '1', 'online': '1',
>>> 'frontend': '/local/domain/15/device/vif/0'} to
>>> /local/domain/0/backend/vif/15/0.[2011-02-06 23:35:51 5208] DEBUG
>>> (XendDomainInfo:1804) Storing domain details: {'image/entry':
>>> '3229974528',
>>> 'console/port': '2', 'image/loader': 'generic', 'vm':
>>> '/vm/aaf5c3c7-a5a6-d306-
>>> dc77-483cea04bff5', 'control/platform-feature-multiprocessor-suspend':
>>> '1',
>>> 'image/hv-start-low': '4118806528', 'image/guest-os': 'linux',
>>> 'image/virt-
>>> base': '3221225472', 'memory/target': '131072', 'image/guest-version':
>>> '2.6',
>>> 'image/pae-mode': 'yes', 'description': '', 'console/limit': '1048576',
>>> 'image/paddr-offset': '0', 'image/hypercall-page': '3222282240',
>>> 'image/suspend-
>>> cancel': '1', 'cpu/0/availability': 'online',
>>> 'image/features/pae-pgdir-above-
>>> 4gb': '1', 'image/features/writable-page-tables': '0', 'console/type':
>>> 'xenconsoled', 'name': 'Elite', 'domid': '15', 'image/xen-version':
>>> 'xen-3.0',
>>> 'store/port': '1'}[2011-02-06 23:35:52 5208] DEBUG (XendCheckpoint:286)
>>> restore:shadow=0x0, _static_max=0x8000000, _static_min=0x0, [2011-02-06
>>> 23:35:52
>>> 5208] DEBUG (XendCheckpoint:305) [xc_restore]:
>>> /usr/lib/xen/bin/xc_restore 4 15
>>> 1 2 0 0 0 0[2011-02-06 23:35:52 5208] INFO (XendCheckpoint:423)
>>> xc_domain_restore start: p2m_size = 8000[2011-02-06 23:35:52 5208] INFO
>>> (XendCheckpoint:423) Reloading memory pages:   0%[2011-02-06 23:36:03
>>> 5208] INFO
>>> (XendCheckpoint:423) ERROR Internal error: Error when reading batch
>>> size[2011-
>>> 02-06 23:36:03 5208] INFO (XendCheckpoint:423) ERROR Internal error:
>>> error when
>>> buffering batch, finishing[2011-02-06 23:36:03 5208] INFO
>>> (XendCheckpoint:423)
>>> [2011-02-06 23:36:03 5208] INFO (XendCheckpoint:423) 100%[2011-02-06
>>> 23:36:03
>>> 5208] INFO (XendCheckpoint:423) Memory reloaded (9067 pages)[2011-02-06
>>> 23:36:03
>>> 5208] INFO (XendCheckpoint:423) read VCPU 0[2011-02-06 23:36:03 5208]
>>> INFO
>>> (XendCheckpoint:423) Completed checkpoint load[2011-02-06 23:36:03 5208]
>>> INFO
>>> (XendCheckpoint:423) Domain ready to be built.[2011-02-06 23:36:03 5208]
>>> INFO
>>> (XendCheckpoint:423) Restore exit with rc=0[2011-02-06 23:36:03 5208]
>>> DEBUG
>>> (XendCheckpoint:394) store-mfn 163723[2011-02-06 23:36:03 5208] DEBUG
>>> (XendCheckpoint:394) console-mfn 163722[2011-02-06 23:36:03 5208] DEBUG
>>> (XendDomainInfo:2992) XendDomainInfo.completeRestore[2011-02-06 23:36:03
>>> 5208]
>>> DEBUG (XendDomainInfo:1804) Storing domain details: {'console/ring-ref':
>>> '163722', 'image/entry': '3229974528', 'console/port': '2', 'store/ring>
>>> -ref':
>>> '163723', 'image/loader': 'generic', 'vm': '/vm/aaf5c3c7-a5a6-d306-dc77-
>>> 483cea04bff5', 'control/platform-feature-multiprocessor-suspend': '1',
>>> 'image/hv-start-low': '4118806528', 'image/guest-os': 'linux',
>>> 'image/virt-
>>> base': '3221225472', 'memory/target': '131072', 'image/guest-version':
>>> '2.6',
>>> 'image/pae-mode': 'yes', 'description': '', 'console/limit': '1048576',
>>> 'image/paddr-offset': '0', 'image/hypercall-page': '3222282240',
>>> 'image/suspend-
>>> cancel': '1', 'cpu/0/availability': 'online',
>>> 'image/features/pae-pgdir-above-
>>> 4gb': '1', 'image/features/writable-page-tables': '0', 'console/type':
>>> 'xenconsoled', 'name': 'Elite', 'domid': '15', 'image/xen-version':
>>> 'xen-3.0',
>>> 'store/port': '1'}[2011-02-06 23:36:03 5208] DEBUG (XendDomainInfo:3005)
>>> XendDomainInfo.completeRestore done[2011-02-06 23:36:03 5208] DEBUG
>>> (DevController:139) Waiting for devices tap2.[2011-02-06 23:36:03 5208]
>>> DEBUG
>>> (XendDomainInfo:1891) XendDomainInfo.handleShutdownWatch[2011-02-06
>>> 23:36:03
>>> 5208] DEBUG (DevController:139) Waiting for devices vif.[2011-02-06
>>> 23:36:03
>>> 5208] DEBUG (DevController:144) Waiting for 0.[2011-02-06 23:36:03 5208]
>>> DEBUG
>>> (DevController:628) hotplugStatusCallback
>>> /local/domain/0/backend/vif/15/0/hotplug-status.[2011-02-06 23:36:03
>>> 5208] DEBUG
>>> (DevController:642) hotplugStatusCallback 1.[2011-02-06 23:36:03 5208]
>>> DEBUG
>>> (DevController:139) Waiting for devices vkbd.[2011-02-06 23:36:03 5208]
>>> DEBUG
>>> (DevController:139) Waiting for devices ioports.[2011-02-06 23:36:03
>>> 5208] DEBUG
>>> (DevController:139) Waiting for devices tap.[2011-02-06 23:36:03 5208]
>>> DEBUG
>>> (DevController:139) Waiting for devices vif2.[2011-02-06 23:36:03 5208]
>>> DEBUG
>>> (DevController:139) Waiting for devices console.[2011-02-06 23:36:03
>>> 5208] DEBUG
>>> (DevController:144) Waiting for 0.[2011-02-06 23:36:03 5208] DEBUG
>>> (DevController:139) Waiting for devices vscsi.[2011-02-06 23:36:03 5208]
>>> DEBUG
>>> (DevController:139) Waiting for devices vbd.[2011-02-06 23:36:03 5208]
>>> DEBUG
>>> (DevController:144) Waiting for 2050.[2011-02-06 23:36:03 5208] DEBUG
>>> (DevController:628) hotplugStatusCallback
>>> /local/domain/0/backend/vbd/15/2050/hotplug-status.[2011-02-06 23:36:03
>>> 5208]
>>> DEBUG (DevController:642) hotplugStatusCallback 1.[2011-02-06 23:36:03
>>> 5208]
>>> DEBUG (DevController:144) Waiting for 2049.[2011-02-06 23:36:03 5208]
>>> DEBUG
>>> (DevController:628) hotplugStatusCallback
>>> /local/domain/0/backend/vbd/15/2049/hotplug-status.[2011-02-06 23:36:03
>>> 5208]
>>> DEBUG (DevController:642) hotplugStatusCallback 1.[2011-02-06 23:36:03
>>> 5208]
>>> DEBUG (DevController:139) Waiting for devices irq.[2011-02-06 23:36:03
>>> 5208]
>>> DEBUG (DevController:139) Waiting for devices vfb.[2011-02-06 23:36:03
>>> 5208]
>>> DEBUG (DevController:139) Waiting for devices pci.[2011-02-06 23:36:03
>>> 5208]
>>> DEBUG (DevController:139) Waiting for devices vusb.[2011-02-06 23:36:03
>>> 5208]
>>> DEBUG (DevController:139) Waiting for devices vtpm.-- Cheers,
>>> >
>>> > Wenda Ni, Ph.D.
>>> > Dept. of Computer Science and Engineering
>>> > State University of New York (SUNY) at Buffalo
>>> > T: 716 982 6746
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > Xen-users mailing list
>>> > Xen-users <at> lists.xensource.com
>>> > http://lists.xensource.com/xen-users
>>>
>>>
>>> Were you able to resolve the problem? I am also experiencing the same
>>> problem
>>> when using ubuntu 10.10 as DomU. Console as well as ssh connection both
>>> freezes
>>> for 10 mins in both the scenarios (A -> B or B -> A).
>>>
>>>
>>> Hitesh Khandelwal | Graduate Student | Purdue University |
>>> http://www.cs.purdue.edu/~hkhande
>>>
>>>
>>> _______________________________________________
>>> Xen-users mailing list
>>> Xen-users@xxxxxxxxxxxxxxxxxxx
>>> http://lists.xensource.com/xen-users
>>
>>
>>
>> --
>> Cheers,
>>
>> Wenda Ni, Ph.D.
>> Dept. of Computer Science and Engineering
>> State University of New York (SUNY) at Buffalo
>> T: 716 982 6746
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users