[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] VM dies spontaneously, how to diagnose?



Hi there,

I have a Debian Lenny Xen server with a few VMs all running Debian Lenny.

One of these occasionally dies with no apparent reason. Theres nothing in its own logs that give any indication of what went wrong. I see nothing in the Xen logs that helps me directly.

I see entries like the one at the end of this mail (I'm going to put the log data at the end because it is a gigantic wall of text).


The config file for this domU contains:

<config>
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'
</config>

so I was expecting it would restart... however looking at the Xen logs it appears that when Xen tries to restart the crashed domU it comes to believe that its already running...

I'm wondering where I should start debugging this problem. It happens very intermittently.


<logentries>
[2009-10-05 20:34:54 3606] WARNING (XendDomainInfo:1258) Domain has crashed: name=monitoring-lenny id=36. [2009-10-05 20:34:54 3606] DEBUG (XendDomainInfo:1914) XendDomainInfo.destroyDomain(36)
[2009-10-05 20:34:54 3606] DEBUG (XendDomainInfo:1529) Destroying device model
[2009-10-05 20:34:54 3606] DEBUG (XendDomainInfo:1536) Releasing devices
[2009-10-05 20:34:54 3606] DEBUG (XendDomainInfo:1542) Removing vif/0
...
more 'destroyDevice' entries
...
[2009-10-05 20:34:54 3606] DEBUG (XendDomainInfo:1534) No device model
[2009-10-05 20:34:54 3606] DEBUG (XendDomainInfo:1536) Releasing devices
[2009-10-05 20:34:54 3606] DEBUG (XendDomainInfo:106) XendDomainInfo.create_from_dict({'vcpus_params': {'cap': 0, 'weight': 256}, 'PV_args': 'root=/dev/sda2 ro clocksource=jiffies', 'features': '', 'cpus': [], 'paused': 0, 'domid': 36, 'shutdown': 0, 'VCPUs_live': 1, 'PV_bootloader': '', 'actions_after_crash': 'restart', 'vbd_refs': ['8baad5aa-45c9-97f1-1a19-96615ef72229', '38a9f2bc-c8ee-8cfc-dac8-11c73eb130e4'], 'PV_ramdisk': '/boot/initrd.img-2.6.26-1-xen-686', 'is_control_domain': False, 'name_label': 'monitoring-lenny', 'VCPUs_at_startup': 1, 'HVM_boot_params': {}, 'platform': {}, 'PV_kernel': '/boot/vmlinuz-2.6.26-1-xen-686', 'console_refs': ['420f5516-d7bd-9d03-b840-d7c0d23e9963'], 'online_vcpus': 1, 'blocked': 0, 'on_xend_stop': 'ignore', 'memory_static_min': 0, 'HVM_boot_policy': '', 'shutdown_reason': 3, 'VCPUs_max': 3, 'start_time': 1249957930.470108, 'memory_static_max': 1073741824, 'actions_after_shutdown': 'destroy', 'on_xend_start': 'ignore', 'crashed': 0, 'memory_dynamic_max': 1073741824, 'actions_after_suspend': '', 'is_a_template': False, 'PV_bootloader_args': '', 'memory_dynamic_min': 1073741824, 'uuid': 'ac86ee9d-45b8-7359-b736-bdc74506068c', 'cpu_time': 482327.477346609, 'shadow_memory': 0, 'dying': 0, 'vcpu_avail': 7, 'notes': {'HV_START_LOW': 4118806528L, 'FEATURES': 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel', 'VIRT_BASE': 3221225472L, 'GUEST_VERSION': '2.6', 'PADDR_OFFSET': 0, 'GUEST_OS': 'linux', 'HYPERCALL_PAGE': 3222278144L, 'LOADER': 'generic', 'SUSPEND_CANCEL': 1, 'PAE_MODE': 'yes', 'ENTRY': 3222274048L, 'XEN_VERSION': 'xen-3.0'}, 'other_config': {}, 'running': 0, 'actions_after_reboot': 'restart', 'vif_refs': ['ef714b6e-164c-5d7f-2708-bdc6a251221f', '0238b375-fb4d-7990-9644-5f2ec53c83ca'], 'vtpm_refs': [], 'devices': {'38a9f2bc-c8ee-8cfc-dac8-11c73eb130e4': ('vbd', {'uuid': '38a9f2bc-c8ee-8cfc-dac8-11c73eb130e4', 'bootable': 0, 'devid': 2049, 'driver': 'paravirtualised', 'dev': 'sda1', 'uname': 'phy:volumes/monitoring-lenny-swap', 'mode': 'w'}), '0238b375-fb4d-7990-9644-5f2ec53c83ca': ('vif', {'bridge': 'xenbr1', 'rate': '10MB/s', 'devid': 1, 'mac': '00:16:3E:12:69:5E', 'uuid': '0238b375-fb4d-7990-9644-5f2ec53c83ca'}), 'ef714b6e-164c-5d7f-2708-bdc6a251221f': ('vif', {'bridge': 'eth0', 'rate': '10MB/s', 'devid': 0, 'mac': '00:16:3E:C7:8A:C8', 'uuid': 'ef714b6e-164c-5d7f-2708-bdc6a251221f'}), '8baad5aa-45c9-97f1-1a19-96615ef72229': ('vbd', {'uuid': '8baad5aa-45c9-97f1-1a19-96615ef72229', 'bootable': 1, 'devid': 2050, 'driver': 'paravirtualised', 'dev': 'sda2', 'uname': 'phy:volumes/monitoring-lenny', 'mode': 'w'}), '420f5516-d7bd-9d03-b840-d7c0d23e9963': ('console', {'location': '2', 'devid': 0, 'protocol': 'vt100', 'uuid': '420f5516-d7bd-9d03-b840-d7c0d23e9963', 'other_config': {}})}}) [2009-10-05 20:34:54 3606] ERROR (XendDomainInfo:111) Domain construction failed
Traceback (most recent call last):
File "/usr/lib/xen-3.2-1/lib/python/xen/xend/XendDomainInfo.py", line 109, in create_from_dict
    vm.start()
File "/usr/lib/xen-3.2-1/lib/python/xen/xend/XendDomainInfo.py", line 444, in start
    raise XendError('VM already running')
XendError: VM already running
[2009-10-05 20:34:54 3606] DEBUG (XendDomainInfo:1897) XendDomainInfo.destroy: domid=36 [2009-10-05 20:34:54 3606] ERROR (XendDomainInfo:1425) Failed to restart domain 36.
</logentries>


--
Please remember that an email is just like a postcard; it is not confidential nor private nor secure and can be read by many other people than the intended recipient. A postcard can be read by anyone at the mail sorting office and expecting what is written on it to be private and secret is not realistic. Please hold no higher expectation of email.

If you need to send confidential information in an email you need to use encryption. PGP is Pretty good for this.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.