Re: [Xen-devel] xend failure, restart doesn't work

[Marius Karthaus <Marius@xxxxxxxxxxx>]

Ok heres some more info:

currently there are multiple domains up and running perfectly. i'm on dom0

wilde root 0 #  xm list
(111, 'Connection refused')
Error: Error connecting to xend, is xend running?

check if it's running:

wilde root 0 #  ps |grep xend
root     13347  0.0  0.3   1436   404 pts/0    D+   11:57   0:00 grep xend
wilde root 0 #  ps |grep xfrd
root     26122  0.0  0.3   3252   464 ?        S    Mar16   0:00 xfrd


cleared the logs:
wilde root 0 #  cat /var/log/xend.log /var/log/xend-debug.log
wilde root 0 # 

stopping xend (just to make sure):
wilde root 0 #  /etc/init.d/xend stop
wilde root 0 #  cat /var/log/xend.log /var/log/xend-debug.log
wilde root 0 # 

starting xend:

wilde root 0 #  /etc/init.d/xend start
.........
wilde root 3 # 

(exit status 3, takes about 5 seconds of time-out-time)


see what is runnning:


wilde root 0 #  ps |grep xend
root     13469  0.0  0.3   1436   472 pts/0    R+   12:03   0:00 grep xend
wilde root 0 #  ps |grep xfr
root     13420  0.0  0.6   3048   792 ?        S    12:02   0:00 xfrd
root     13471  0.0  0.3   1436   472 pts/0    R+   12:04   0:00 grep xfr
wilde root 0 #  ps |grep xcs
root     13477  0.0  0.3   1436   472 pts/0    R+   12:04   0:00 grep xcs

So only xfrd has started. lets see what is in the logs:

wilde root 1 #  cat /var/log/xend.log                        
[2005-03-25 12:02:45 xend] INFO (SrvDaemon:610) Xend Daemon started

( i wish :S )

wilde root 0 #  cat /var/log/xend-debug.log
network start bridge=xen-br0 netdev=eth0 antispoof=yes
Traceback (most recent call last):
  File "/usr/sbin/xend", line 121, in ?
    sys.exit(main())
  File "/usr/sbin/xend", line 107, in main
    return daemon.start()
  File "/usr/lib/python/xen/xend/server/SrvDaemon.py", line 525, in start
    self.run()
  File "/usr/lib/python/xen/xend/server/SrvDaemon.py", line 615, in run
    SrvServer.create(bridge=1)
  File "/usr/lib/python/xen/xend/server/SrvServer.py", line 47, in create
    xend = SrvRoot()
  File "/usr/lib/python/xen/xend/server/SrvRoot.py", line 29, in __init__
    self.get(name)
  File "/usr/lib/python/xen/xend/server/SrvDir.py", line 69, in get
    val = val.getobj()
  File "/usr/lib/python/xen/xend/server/SrvDir.py", line 39, in getobj
    self.obj = klassobj()
  File "/usr/lib/python/xen/xend/server/SrvDomainDir.py", line 25, in __init__
    self.xd = XendDomain.instance()
  File "/usr/lib/python/xen/xend/XendDomain.py", line 798, in instance
    inst = XendDomain()
  File "/usr/lib/python/xen/xend/XendDomain.py", line 65, in __init__
    self.initial_refresh()
  File "/usr/lib/python/xen/xend/XendDomain.py", line 153, in initial_refresh
    d_dom = self._new_domain(config, doms[domid])
  File "/usr/lib/python/xen/xend/XendDomain.py", line 188, in _new_domain
    deferred = XendDomainInfo.vm_recreate(savedinfo, info)
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 218, in vm_recreate
    d = vm.construct(config)
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 453, in construct
    self.construct_image()
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 480, in construct_image
    image_handler(self, image)
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 1065, in vm_image_linux
    vm.create_domain("linux", kernel, ramdisk, cmdline)
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 757, in create_domain
    self.create_channel()
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 782, in create_channel
    remote_port=remote)
  File "/usr/lib/python/xen/xend/server/SrvDaemon.py", line 660, in createDomChannel
    remote_port=remote_port)
  File "/usr/lib/python/xen/xend/server/channel.py", line 59, in domChannel
    remote_port=remote_port)
  File "/usr/lib/python/xen/xend/server/channel.py", line 229, in __init__
    remote_port=remote_port)
  File "/usr/lib/python/xen/xend/server/channel.py", line 113, in createPort
    remote_port=int(remote_port))
xen.lowlevel.xu.PortError: Failed to map domain control interface

So there is the problem i guess. But i don't know what it means or how i should fix it. Any ideas ?

Ian Pratt wrote:

Hi,
Every now and then xend seems to fail, the domains keep running but 
control is lost completely.
A restart seems not possible. The logfile says xend restarts but the 
only thing that restarts is xfrd binding on port 8002.
The only way to get control back seems to be to reboot the machine, 
rebooting all of the client domains too wich is a *bad thing* for our 
clients.
Is there any way to get control back?
Is anyone working on the problem is there a patch or even a 
know cause??
Are there any steps i can take (or NOT take) to prevent this 
from happening?
    

You could try 'xend stop' and then kill 'xcs' manually, then 
'xend start'.

You'll need to give us more help to debug the actual problem your
experiencing.

Ian