Re: [Xen-users] SOLUTION FOUND FOR xend does not start "No space left..."

[Oliver Heil <o.heil@xxxxxxxxxxxxxxxxxx>]

Hi,

I finally found a solution for my problem.
I still do not know, what the exact problem was. The symptoms
are, that the xen dom0, Domain-0 does not start and that e.g.
xm asks, if xend is running, but xend is in the process list.

>> # xm list
>> Error: Unable to connect to xend: No such file or directory. Is xend
>> running?

The xend.log file reports error messages like

> RuntimeError: (28, 'No space left on device, while writing 
/vm/00000000-0000-0000-0000-
> 000000000000/xend/restart_count : 0')

but every device and memory has plenty space available.

The system is a openSUSE 10.2 on 2 processor XEON 64bit machine
with 4Gbyte RAM. XEN is build from source xen-3.0.4_1

To workaround and to get xen back again I just
removed the directories /var/lib/xen, /var/lib/xend
and /var/lib/xenstored.
In /var/lib/xen/save was 1 Gbyte file named like the only
guest virtual machine I am running on the host:
/var/lib/xen/save/vm01
This is propably a memory image of the guest vm and I believe
that the existance of this file was the cause of the trouble.

There are a few other threads unanswered in the net, which maybe
solved the same way. Here are some of the subjects I found with google:
[Fedora-xen] Xen0 not booting into Login console
[Fedora-xen] Exception when starting xend: error while reading 
/vm/00000000-0000-0000-0000-000000000000/ssidref

Sincerelly,

Oli

Oliver Heil wrote:
> Hi XEN Users and Developers,
>
> xend does not start into a clean state. The process is running...
>
> > #ps -efl | grep xen
> > 1 S root        20    19  0  70  -5 -     0 xenwat Mar06 ?        
> > 00:00:00 [xenwatch]
> > 1 S root        21    19  0  70  -5 -     0 xb_rea Mar06 ?        
> > 00:00:00 [xenbus]
> > 5 S root     12663     1  0  76   0 -  1274 -      09:56 ?        
> > 00:00:00 xenstored --pid-file /var/run/xenstore.pid
> > 5 S root     12667     1  0  84   0 -  3552 stext  09:56 ?        
> > 00:00:00 xenconsoled
> > 1 S root     12671     1  0  79   0 - 13265 wait   09:56 ?        
> > 00:00:00 python /usr/sbin/xend start
> > 5 S root     12673 12671  0  76   0 - 15412 184466 09:56 ?        
> > 00:00:00 python /usr/sbin/xend start
> ...but no virtual machine can be started and xm answers e.g with this:
> > # xm list
> > Error: Unable to connect to xend: No such file or directory. Is xend 
> > running?
> In the xend.log there is the following error:
> > [2007-03-07 09:56:13 xend 12659] WARNING (XendAuthSessions:27) 
> > python-pam is required for XenAPI support.
> > [2007-03-07 09:56:13 xend 12673] INFO (SrvDaemon:322) Xend Daemon started
> > [2007-03-07 09:56:13 xend 12673] INFO (SrvDaemon:326) Xend changeset: 
> > unavailable.
> > [2007-03-07 09:56:13 xend 12673] INFO (SrvDaemon:333) Xend version: 
> > Unknown.
> > [2007-03-07 09:56:13 xend.XendDomainInfo 12673] DEBUG 
> > (XendDomainInfo:121) XendDomainInfo.recreate({'max_vcpu_id': 3, 
> > 'cpu_time': 11
> > 74593321820L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 0, 'dying': 
> > 0, 'online_vcpus': 4, 'domid': 0, 'paused': 0, 'crashed': 0, 'r
> > unning': 1, 'maxmem_kb': 17179869180, 'shutdown': 0, 'mem_kb': 
> > 3985144, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
> > 'blocked': 0, 'name': 'Domain-0'})
> > [2007-03-07 09:56:13 xend.XendDomainInfo 12673] INFO 
> > (XendDomainInfo:135) Recreating domain 0, UUID 
> > 00000000-0000-0000-0000-00000000
> > 0000. at /local/domain/0
> > [2007-03-07 09:56:13 xend.XendDomainInfo 12673] WARNING 
> > (XendDomainInfo:149) /local/domain/0/vm is missing. recreate is 
> > confused, tr
> > ying our best to recover
> > [2007-03-07 09:56:13 xend.XendDomainInfo 12673] DEBUG 
> > (XendDomainInfo:1714) Storing VM details: {'on_xend_stop': 'ignore', 
> > 'shadow_m
> > emory': '0', 'uuid': '00000000-0000-0000-0000-000000000000', 'name': 
> > 'Domain-0', 'on_reboot': 'restart', 'on_poweroff': 'destroy', '
> > on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': 
> > '0', 'vcpus': '4', 'vcpu_avail': '15', 'memory': '3892', 'max
> > mem': '16777216'}
> > [2007-03-07 09:56:13 xend 12673] ERROR (SrvDaemon:344) Exception 
> > starting xend ((28, 'No space left on device, while writing /vm/000
> > 00000-0000-0000-0000-000000000000/xend/restart_count : 0'))
> > Traceback (most recent call last):
> >   File "//usr/lib64/python/xen/xend/server/SrvDaemon.py", line 336, in 
> > run
> >     servers = SrvServer.create()
> >   File "//usr/lib64/python/xen/xend/server/SrvServer.py", line 226, in 
> > create
> >     root.putChild('xend', SrvRoot())
> >   File "//usr/lib64/python/xen/xend/server/SrvRoot.py", line 40, in 
> > __init__
> >     self.get(name)
> >   File "//usr/lib64/python/xen/web/SrvDir.py", line 82, in get
> >     val = val.getobj()
> >   File "//usr/lib64/python/xen/web/SrvDir.py", line 52, in getobj
> >     self.obj = klassobj()
> >   File "//usr/lib64/python/xen/xend/server/SrvDomainDir.py", line 41, 
> > in __init__
> >     self.xd = XendDomain.instance()
> >   File "//usr/lib64/python/xen/xend/XendDomain.py", line 1399, in 
> > instance
> >     inst.init()
> >   File "//usr/lib64/python/xen/xend/XendDomain.py", line 116, in init
> >     dom0 = XendDomainInfo.recreate(dom0info, True)
> >   File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 174, in 
> > recreate
> >     vm._storeVmDetails()
> >   File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 1716, in 
> > _storeVmDetails
> >     self._writeVm(to_store)
> >   File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 672, in 
> > _writeVm
> >     return xstransact.Write(self.vmpath, *args)
> >   File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 302, 
> > in Write
> >     complete(path, lambda t: t.write(*args))
> >   File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 354, 
> > in complete
> >     result = f(t)
> >   File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 302, 
> > in <lambda>
> >     complete(path, lambda t: t.write(*args))
> >   File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 87, 
> > in write
> >     self._write(key, d[key])
> >   File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 75, 
> > in _write
> >     ('%s, while writing %s : %s' %
> > RuntimeError: (28, 'No space left on device, while writing 
> > /vm/00000000-0000-0000-0000-000000000000/xend/restart_count : 0')
> > [2007-03-07 09:56:13 xend 12675] WARNING (XendAuthSessions:27) 
> > python-pam is required for XenAPI support.
> Does somebody know a cure for this behaviour?
>
> With regards,
>
> Oli
--
/////////////////////////////////
Oliver Heil

DKFZ - Deutsches Krebsforschungszentrum
Im Neuenheimer Feld 515
69120 Heidelberg

Tel    (+49 6221) / 42 4708

Email  o.heil@xxxxxxx
WWW    www.dkfz.de
/////////////////////////////////



-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.0.6 (MingW32)

mQGiBDyzHxERBACFUsv9ZzMAstEUKpV7ZI+//CqMwNWteCdcet8ONYQkE+meq7qp
L2yQgqdGUdlRz/O0Ix9ERuMw+6qnRYxXCGvRjWX6t+wDSALfFjjEIUVecedsMKGv
7bINfUQEJQ+7hZAonOJ/iEL3adh8fvl1Bl0yNYxATbqp+q1YGawsHLu4TwCg10VR
AkS9zZnv1gsGBg2rDm8f2CcD/0siN0AUvjvj38Hzxr7CCG22c5QYHZLhFvM7+jSA
OF2qTNGtFJ0fVkIlqzkd1D21efmM2xKo9Jzlhb5D1Dr1GQ7epxvpcGV/n8vHtwOv
+N8UoOaNfrJIoM9lV4IXKTAnwlTcyy61Bthx4L+dKNTH0ejbPk3/SIUlYAju6V9y
uqOCA/0euPYCj0GQ4YfdULUu/ZTLOAX3X+fUXqSQThBUt8BpTv/wqNqSifURttH9
ERb+Q22E9ck5mYs2zZmd2xvmxU6swFqJJVZMBzs1/iQcc7mC1v84Riddcuj+C4YA
whph8LxHAdFPN+ZHoaQTIu8vC0KdtHJX88KNxvl7HVYL8x9yW7QcT2xpdmVyIEhl
aWwgPG8uaGVpbEBka2Z6LmRlPohXBBMRAgAXBQI8sx8RBQsHCgMEAxUDAgMWAgEC
F4AACgkQ87yOFfCjArRQUgCgqL5irUC6wdZntArL+8n19RWQWrMAn32Dee0njHm1
2qTW27TcBSPAWGwSuQINBDyzH18QCADC8Em8AKrM1QrBR1qpmb9kN+KmF5FZDwXP
4ioO4k0ee2cfcriUiZJXeNo/TpUvGpGRhINMyCKDpiXeU970Gxhy6ksU+q+hEqFZ
nAqYuq00xQQ+SxMb+TgGytgmq3Jz8q8jJFokkd3Mc0mcicPKlKNv4Sq+sGAzHXTk
in/XJRTB7RA7zl7K+iz+u4lnXyyhM+cT82EtduQ2XvPGYjQwymLMcIRhoWlbvB1w
1uGGCaOQSdlH36LEG+P41TwMjB4Eq2dhk9PL+6pUpbRXrtWtnUzfQuEFwHlMwITv
RCit6Pp+5f2USyWeUP0oc/39I0iTOqINZ7S031FD7xa61vtMK01bAAMGCAC2sX9z
vormmdWvHWyH5+d9PiYUS86JNy8ncVqgoLtin6fqpy0UlGMHH2ewfpExfVip8CWj
Cp7vzpmzhsu+wIDyVdAVx3OldKFUH7Al3PDdMe4hNkfB0om+DDAYPO33C7PhpcUL
+Muz14fc6ti5PKs8zNcqENGyL0sv/Uun28Ebb9cOHWfEcgJ8ZbfmLiW8b2vXYFPj
F6T0+tOTCSNDuWGBAWfvumOhZB6sFokPUxs+5CqibGTgGtCUYW1VXVTu+d3bugMN
NcuQBucGKdq7v6rREgfNfqvfodyk77Zy9w9wbGvPROUe77PaYubdRviiCiMYfQQc
501JeSIwwARsSt5viEYEGBECAAYFAjyzH18ACgkQ87yOFfCjArSc9ACdEY4YqlqM
lIuB4uWb6XZ3DTTUs3QAoL+a4UJp9OLLVtjTsijb/hpu3uhO
=I3OA
-----END PGP PUBLIC KEY BLOCK-----

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users