Hi,
Unfortunately there are
no errors either In logs or console output. In xen.log there is:
[2010-07-11 04:00:02
4078] WARNING (XendDomainInfo:1258) Domain has crashed: name=web id=5.
[2010-07-11 04:00:02
4078] DEBUG (XendDomainInfo:1914) XendDomainInfo.destroyDomain(5)
[2010-07-11 04:00:02
4078] DEBUG (XendDomainInfo:1529) Destroying device model
[2010-07-11 04:00:02
4078] DEBUG (XendDomainInfo:1536) Releasing devices
[2010-07-11 04:00:02
4078] DEBUG (XendDomainInfo:1542) Removing vif/0
[2010-07-11 04:00:02
4078] DEBUG (XendDomainInfo:590) XendDomainInfo.destroyDevice: deviceClass =
vif, device = vif/0
[2010-07-11 04:00:02
4078] DEBUG (XendDomainInfo:1542) Removing vbd/51713
[2010-07-11 04:00:02
4078] DEBUG (XendDomainInfo:590) XendDomainInfo.destroyDevice: deviceClass =
vbd, device = vbd/51713
[2010-07-11 04:00:02
4078] DEBUG (XendDomainInfo:1542) Removing vbd/51714
[2010-07-11 04:00:02
4078] DEBUG (XendDomainInfo:590) XendDomainInfo.destroyDevice: deviceClass =
vbd, device = vbd/51714
[2010-07-11 04:00:02
4078] DEBUG (XendDomainInfo:1542) Removing console/0
[2010-07-11 04:00:02
4078] DEBUG (XendDomainInfo:590) XendDomainInfo.destroyDevice: deviceClass =
console, device = console/0
[2010-07-11 04:00:02
4078] DEBUG (XendDomainInfo:1534) No device model
[2010-07-11 04:00:02
4078] DEBUG (XendDomainInfo:1536) Releasing devices
[2010-07-11 04:00:02
4078] DEBUG (XendDomainInfo:106)
XendDomainInfo.create_from_dict({'vcpus_params': {'cap': 0, 'weight': 256},
'PV_args': 'root=/dev/xvda2 ro clocksource
=jiffies',
'features': '', 'cpus': [], 'paused': 0, 'actions_after_reboot': 'restart',
'shutdown': 0, 'VCPUs_live': 1, 'PV_bootloader': '', 'actions_after_crash':
'restart'
, 'vbd_refs':
['4af090ff-7ded-2e40-e641-0f6a812cb1b7',
'0bd0bd32-432e-62f0-291f-e52caadef552'], 'PV_ramdisk':
'/boot/initrd.img-2.6.26-2-xen-686', 'is_control_domain': Fals
e, 'name_label': 'web',
'VCPUs_at_startup': 1, 'HVM_boot_params': {}, 'platform': {}, 'PV_kernel':
'/boot/vmlinuz-2.6.26-2-xen-686', 'console_refs': ['0de68767-8e44
-424e-bce3-4e75700a90fd'],
'online_vcpus': 4, 'blocked': 0, 'on_xend_stop': 'ignore', 'memory_static_min':
0, 'HVM_boot_policy': '', 'shutdown_reason': 3, 'VCPUs_max': 4, '
start_time':
1278275055.0122731, 'memory_static_max': 2147483648L, 'actions_after_shutdown':
'destroy', 'on_xend_start': 'ignore', 'crashed': 0, 'memory_dynamic_max': 21474
83648L,
'actions_after_suspend': '', 'is_a_template': False, 'PV_bootloader_args': '',
'memory_dynamic_min': 2147483648L, 'uuid':
'103c0d3b-7612-fd0e-9adf-4345c84e5602', 'c
pu_time':
45506.519344966, 'shadow_memory': 0, 'dying': 0, 'vcpu_avail': 15, 'notes': {'HV_START_LOW':
4118806528L, 'FEATURES': 'writable_page_tables|writable_descriptor_ta
bles|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel',
'VIRT_BASE': 3221225472L, 'GUEST_VERSION': '2.6', 'PADDR_OFFSET': 0,
'GUEST_OS': 'linux', 'HYPERCA
LL_PAGE':
3222278144L, 'LOADER': 'generic', 'SUSPEND_CANCEL': 1, 'PAE_MODE': 'yes',
'ENTRY': 3222274048L, 'XEN_VERSION': 'xen-3.0'}, 'other_config': {}, 'running':
0, 'domi
d': 5, 'vif_refs':
['583adee6-2d81-2ee4-e5cf-06bce4c1dc4a'], 'vtpm_refs': [], 'devices':
{'0bd0bd32-432e-62f0-291f-e52caadef552': ('vbd', {'uuid':
'0bd0bd32-432e-62f0-291f-
e52caadef552',
'bootable': 0, 'devid': 51714, 'driver': 'paravirtualised', 'dev': 'xvda2',
'uname': 'phy:/dev/lvg/web-disk', 'mode': 'w'}), '4af090ff-7ded-2e40-e641
-0f6a812cb1b7':
('vbd', {'uuid': '4af090ff-7ded-2e40-e641-0f6a812cb1b7', 'bootable': 1,
'devid': 51713, 'driver': 'paravirtualised', 'dev': 'xvda1', 'uname':
'phy:/dev/lvg/
web-swap', 'mode':
'w'}), '0de68767-8e44-424e-bce3-4e75700a90fd': ('console', {'location': '2',
'devid': 0, 'protocol': 'vt100', 'uuid': '0de68767-8e44-424e-bce3-4e
75700a90fd',
'other_config': {}}), '583adee6-2d81-2ee4-e5cf-06bce4c1dc4a': ('vif', {'ip':
'10.20.22.131', 'mac': '00:16:3E:16:9A:43', 'devid': 0, 'uuid':
'583adee6-2d81-2ee
4-e5cf-06bce4c1dc4a',
'bridge': 'eth1'})}})
[2010-07-11 04:00:02
4078] ERROR (XendDomainInfo:111) Domain construction failed
Traceback (most
recent call last):
File
"/usr/lib/xen-3.2-1/lib/python/xen/xend/XendDomainInfo.py", line 109,
in create_from_dict
vm.start()
File
"/usr/lib/xen-3.2-1/lib/python/xen/xend/XendDomainInfo.py", line 444,
in start
raise
XendError('VM already running')
XendError: VM already
running
[2010-07-11 04:00:02
4078] DEBUG (XendDomainInfo:1897) XendDomainInfo.destroy: domid=5
[2010-07-11 04:00:02
4078] ERROR (XendDomainInfo:1425) Failed to restart domain 5.
Same time on domU
(restart was initiated by watchdog on dom0 – script on cron checking if
every domU is up and running):
Jul 11 03:59:01 web
/USR/SBIN/CRON[32055]: (xxxx) CMD (sh /home/xxxx/generator.sh 1>/dev/null)
Jul 11 03:59:01 web
/USR/SBIN/CRON[32056]: (xyyy) CMD (sh /home/yyyy/generator.sh 1>/dev/null)
Jul 11 04:02:12 web
kernel: imklog 3.18.6, log source = /proc/kmsg started.
Jul 11 04:02:12 web
kernel: [ 0.000000] Initializing cgroup subsys cpuset
Jul 11 04:02:12 web
kernel: [ 0.000000] Initializing cgroup subsys cpu
Jul 11 04:02:12 web
kernel: [ 0.000000] Linux version 2.6.26-2-xen-686 (Debian 2.6.26-24)
(dannf@xxxxxxxxxx) (gcc version 4.1.3 20080704 (prerelease) (Debian 4.1.2-25))
#1 SMP Mon Jun 21
10:37:05 UTC 2010
Jul 11 04:02:12 web
kernel: [ 0.000000] Reserving virtual address space above 0xf5800000
Jul 11 04:02:12 web
kernel: [ 0.000000] BIOS-provided physical RAM map:
Jul 11 04:02:12 web
kernel: [ 0.000000] Xen: 0000000000000000 - 0000000080800000 (usable)
Jul 11 04:02:12 web
kernel: [ 0.000000] 1328MB HIGHMEM available.
Jul 11 04:02:12 web
kernel: [ 0.000000] 728MB LOWMEM available.
Jul 11 04:02:12 web
kernel: [ 0.000000] NX (Execute Disable) protection: active
Jul 11 04:02:12 web
kernel: [ 0.000000] Entering add_active_range(0, 0, 526336) 0 entries of 256
used
Jul 11 04:02:12 web
kernel: [ 0.000000] Zone PFN ranges:
Jul 11 04:02:12 web
kernel: [ 0.000000] DMA 0 -> 4096
Jul 11 04:02:12 web
kernel: [ 0.000000] Normal 4096 -> 186368
Jul 11 04:02:12 web
kernel: [ 0.000000] HighMem 186368 -> 526336
Jul 11 04:02:12 web
kernel: [ 0.000000] Movable zone start PFN for each node
Jul 11 04:02:12 web
kernel: [ 0.000000] early_node_map[1] active PFN ranges
Jul 11 04:02:12 web
kernel: [ 0.000000] 0: 0 -> 526336
Jul 11 04:02:12 web
kernel: [ 0.000000] On node 0 totalpages: 526336
Cron scripts have schedule
to run every minute, so those calls aren’t unusual and cannot be primary
reason for crash…
There are some segfault
from php5-cgi, but they were not present at the crash time (guess they have something
to do with php5-suhosin module)…
Jul 8 22:12:51 web
kernel: [345103.172278] php5-cgi[9975]: segfault at 120 ip 082dd95f sp bfff76d0
error 4 in php5-cgi[8048000+4cf000]
Jul 8 22:20:24 web
kernel: [345556.528923] php5-cgi[10293]: segfault at 120 ip 082dd95f sp
bfff7fb0 error 4 in php5-cgi[8048000+4cf000]
Jul 8 22:24:44 web
kernel: [345816.800299] php5-cgi[10524]: segfault at 120 ip 082dd95f sp
bfff8ba0 error 4 in php5-cgi[8048000+4cf000]
Jul 8 22:36:39 web
kernel: [346531.763830] php5-cgi[10640]: segfault at 120 ip 082dd95f sp
bfff7510 error 4 in php5-cgi[8048000+4cf000]
Jul 8 22:48:49 web
kernel: [347261.661357] php5-cgi[10949]: segfault at 120 ip 082dd95f sp
bfff7b10 error 4 in php5-cgi[8048000+4cf000]
Jul 8 22:49:10 web
kernel: [347282.927768] php5-cgi[11241]: segfault at 120 ip 082dd95f sp
bfff7690 error 4 in php5-cgi[8048000+4cf000]
Jul 8 22:51:10 web
kernel: [347402.846540] php5-cgi[11253]: segfault at 120 ip 082dd95f sp
bfff7fe0 error 4 in php5-cgi[8048000+4cf000]
Jul 9 06:25:01 web
kernel: Kernel logging (proc) stopped.
Jul 9 06:25:02 web
kernel: imklog 3.18.6, log source = /proc/kmsg started.
Jul 9 06:25:02 web
rsyslogd: [origin software="rsyslogd" swVersion="3.18.6"
x-pid="2120" x-info="http://www.rsyslog.com"] restart
Jul 10 06:25:01 web
kernel: Kernel logging (proc) stopped.
Jul 10 06:25:01 web
kernel: imklog 3.18.6, log source = /proc/kmsg started.
Jul 10 06:25:01 web
rsyslogd: [origin software="rsyslogd" swVersion="3.18.6"
x-pid="2120" x-info="http://www.rsyslog.com"] restart
Jul 11 04:02:12 web
kernel: imklog 3.18.6, log source = /proc/kmsg started.
Jul 11 04:02:12 web
kernel: [ 0.000000] Initializing cgroup subsys cpuset
Jul 11 04:02:12 web
kernel: [ 0.000000] Initializing cgroup subsys cpu
Jul 11 04:02:12 web
kernel: [ 0.000000] Linux version 2.6.26-2-xen-686 (Debian 2.6.26-24)
(dannf@xxxxxxxxxx) (gcc version 4.1.3 20080704 (prerelease) (Debian 4.1.2-25))
#1 SMP Mon Jun 21
10:37:05 UTC 2010
