[Xen-users] DomU hang in run state (Debian Lenny)

[Matt Baker <m@xxxxxxxxxxxx>]

Hi all,

 We have a number of Xen nodes used in a bunch of Ganeti clusters
running on Debian Lenny. Most are 64bit kernels with a mix of 32/64bit
user land VMs. Where we have a paravirtualised Lenny DomU we are
experiencing a hang at seemingly random occasions. When inspecting the
hypervisor it states that the DomU is in a run state (with xm list) and
(with xm top) the CPUs are all maxed out. I am not able to get into the
DomU either over the network or via a console. Sometimes I get output to
the console but there is no information since the standard boot messages
which were usually printed there from a week or so ago so not relevant.

I do not have any information in the Hypervisors xen logs or kernel logs
and similarly in the DomU kernel logs. I have ran a script in the DomU 
capturing the output of ps every 10 seconds and alerting to processes 
which are using more than 30% memory or cpu. Neither of these show any 
output at the time of the hang. I am also monitoring all DomUs via munin 
which is also not recording a gradual creep in resource usage.
I have had a problem with the "time went backwards" issue and have
attempted to fix the problem as shown on the Xen FAQ by setting the 
clock source to "jiffies". This was the most successful as it stopped 
time messages, but still exhibited the hang problem above. Before, I was 
experiencing kernel panics with the default clocksource of "xen" and 
independant_wallclock=0. I have also tried setting "disable kernel" in 
ntp.conf (with clocksource=xen and independent_wallclock=0) which has 
appeared recently as an option, but unfortunately I am back to the 
original problem of the physical host hanging needing a hard reset.
I am considering an attempt to move these hosts to a newer version of 
Xen if there's a possibility it will be more stable. Current version is 
standard for Lenny, xen = 3.2, kernel 2.6.26.
Any assistance or advice on this would be greatly appreciated.

Many thanks,

Matt

--
 Matthew Baker, UNIX Systems Administrator
 -----------------------------------------------------
 Institute for Learning and Research Technology (ILRT)
 A: University of Bristol,
    8-10 Berkeley Square,
    Bristol.
    BS8 1HH
 W: http://www.ilrt.bris.ac.uk/
 E: matt.baker@xxxxxxxxxx
 T: Berkeley Square
    +44 (0)117 33 14325
 T: Computer Centre
    +44 (0)117 33 17467
 F: 35BB AD51 9892 D694 7664  8BFD 2EF9 BBA4 1FDA 89C3
 -----------------------------------------------------

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users