[Xen-users] VM dies spontaneously, how to diagnose?

[Steve Wray <steve.wray@xxxxxxxxx>]

Hi there,

I have a Debian Lenny Xen server with a few VMs all running Debian Lenny.

One of these occasionally dies with no apparent reason. Theres nothing in 
its own logs that give any indication of what went wrong. I see nothing in 
the Xen logs that helps me directly.
I see entries like the one at the end of this mail (I'm going to put the 
log data at the end because it is a gigantic wall of text).

The config file for this domU contains:

<config>
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'
</config>

so I was expecting it would restart... however looking at the Xen logs it 
appears that when Xen tries to restart the crashed domU it comes to believe 
that its already running...
I'm wondering where I should start debugging this problem. It happens very 
intermittently.

<logentries>
[2009-10-05 20:34:54 3606] WARNING (XendDomainInfo:1258) Domain has 
crashed: name=monitoring-lenny id=36.
[2009-10-05 20:34:54 3606] DEBUG (XendDomainInfo:1914) 
XendDomainInfo.destroyDomain(36)
[2009-10-05 20:34:54 3606] DEBUG (XendDomainInfo:1529) Destroying device model
[2009-10-05 20:34:54 3606] DEBUG (XendDomainInfo:1536) Releasing devices
[2009-10-05 20:34:54 3606] DEBUG (XendDomainInfo:1542) Removing vif/0
...
more 'destroyDevice' entries
...
[2009-10-05 20:34:54 3606] DEBUG (XendDomainInfo:1534) No device model
[2009-10-05 20:34:54 3606] DEBUG (XendDomainInfo:1536) Releasing devices
[2009-10-05 20:34:54 3606] DEBUG (XendDomainInfo:106) 
XendDomainInfo.create_from_dict({'vcpus_params': {'cap': 0, 'weight': 256}, 
'PV_args': 'root=/dev/sda2 ro clocksource=jiffies', 'features': '', 'cpus': 
[], 'paused': 0, 'domid': 36, 'shutdown': 0, 'VCPUs_live': 1, 
'PV_bootloader': '', 'actions_after_crash': 'restart', 'vbd_refs': 
['8baad5aa-45c9-97f1-1a19-96615ef72229', 
'38a9f2bc-c8ee-8cfc-dac8-11c73eb130e4'], 'PV_ramdisk': 
'/boot/initrd.img-2.6.26-1-xen-686', 'is_control_domain': False, 
'name_label': 'monitoring-lenny', 'VCPUs_at_startup': 1, 'HVM_boot_params': 
{}, 'platform': {}, 'PV_kernel': '/boot/vmlinuz-2.6.26-1-xen-686', 
'console_refs': ['420f5516-d7bd-9d03-b840-d7c0d23e9963'], 'online_vcpus': 
1, 'blocked': 0, 'on_xend_stop': 'ignore', 'memory_static_min': 0, 
'HVM_boot_policy': '', 'shutdown_reason': 3, 'VCPUs_max': 3, 'start_time': 
1249957930.470108, 'memory_static_max': 1073741824, 
'actions_after_shutdown': 'destroy', 'on_xend_start': 'ignore', 'crashed': 
0, 'memory_dynamic_max': 1073741824, 'actions_after_suspend': '', 
'is_a_template': False, 'PV_bootloader_args': '', 'memory_dynamic_min': 
1073741824, 'uuid': 'ac86ee9d-45b8-7359-b736-bdc74506068c', 'cpu_time': 
482327.477346609, 'shadow_memory': 0, 'dying': 0, 'vcpu_avail': 7, 'notes': 
{'HV_START_LOW': 4118806528L, 'FEATURES': 
'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel', 
'VIRT_BASE': 3221225472L, 'GUEST_VERSION': '2.6', 'PADDR_OFFSET': 0, 
'GUEST_OS': 'linux', 'HYPERCALL_PAGE': 3222278144L, 'LOADER': 'generic', 
'SUSPEND_CANCEL': 1, 'PAE_MODE': 'yes', 'ENTRY': 3222274048L, 
'XEN_VERSION': 'xen-3.0'}, 'other_config': {}, 'running': 0, 
'actions_after_reboot': 'restart', 'vif_refs': 
['ef714b6e-164c-5d7f-2708-bdc6a251221f', 
'0238b375-fb4d-7990-9644-5f2ec53c83ca'], 'vtpm_refs': [], 'devices': 
{'38a9f2bc-c8ee-8cfc-dac8-11c73eb130e4': ('vbd', {'uuid': 
'38a9f2bc-c8ee-8cfc-dac8-11c73eb130e4', 'bootable': 0, 'devid': 2049, 
'driver': 'paravirtualised', 'dev': 'sda1', 'uname': 
'phy:volumes/monitoring-lenny-swap', 'mode': 'w'}), 
'0238b375-fb4d-7990-9644-5f2ec53c83ca': ('vif', {'bridge': 'xenbr1', 
'rate': '10MB/s', 'devid': 1, 'mac': '00:16:3E:12:69:5E', 'uuid': 
'0238b375-fb4d-7990-9644-5f2ec53c83ca'}), 
'ef714b6e-164c-5d7f-2708-bdc6a251221f': ('vif', {'bridge': 'eth0', 'rate': 
'10MB/s', 'devid': 0, 'mac': '00:16:3E:C7:8A:C8', 'uuid': 
'ef714b6e-164c-5d7f-2708-bdc6a251221f'}), 
'8baad5aa-45c9-97f1-1a19-96615ef72229': ('vbd', {'uuid': 
'8baad5aa-45c9-97f1-1a19-96615ef72229', 'bootable': 1, 'devid': 2050, 
'driver': 'paravirtualised', 'dev': 'sda2', 'uname': 
'phy:volumes/monitoring-lenny', 'mode': 'w'}), 
'420f5516-d7bd-9d03-b840-d7c0d23e9963': ('console', {'location': '2', 
'devid': 0, 'protocol': 'vt100', 'uuid': 
'420f5516-d7bd-9d03-b840-d7c0d23e9963', 'other_config': {}})}})
[2009-10-05 20:34:54 3606] ERROR (XendDomainInfo:111) Domain construction 
failed
Traceback (most recent call last):
  File "/usr/lib/xen-3.2-1/lib/python/xen/xend/XendDomainInfo.py", line 
109, in create_from_dict
    vm.start()
  File "/usr/lib/xen-3.2-1/lib/python/xen/xend/XendDomainInfo.py", line 
444, in start
    raise XendError('VM already running')
XendError: VM already running
[2009-10-05 20:34:54 3606] DEBUG (XendDomainInfo:1897) 
XendDomainInfo.destroy: domid=36
[2009-10-05 20:34:54 3606] ERROR (XendDomainInfo:1425) Failed to restart 
domain 36.
</logentries>


--
Please remember that an email is just like a postcard; it is not 
confidential nor private nor secure and can be read by many other people 
than the intended recipient. A postcard can be read by anyone at the mail 
sorting office and expecting what is written on it to be private and secret 
is not realistic. Please hold no higher expectation of email.
If you need to send confidential information in an email you need to use 
encryption. PGP is Pretty good for this.
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users