[Xen-users] Server hanging - Please help!

[James Pifer <jep@xxxxxxxxxxxxxxxx>]

I'm having a very weird problem that started about a week ago. 

I have a couple sles11 xen 3.4 servers. The first server has a local
400gb disk provided by our NAS. My VMs are file based. For some reason
that server has started to hang randomly. 

I thought something was wrong with the server, possibly the hardware. I
moved the 400gb disk and assigned it to a second sles11 xen 3.4 server
and brought the VMs up on that server. 

Now it's hanging! I don't see anything in /var/log/messages.
When this happens the server is still pingable.
I can get on the server's console, and hit enter and the cursor moves
down, but it's not usable. 

I've forced an fcsk on the 400gb drive and it comes back clean. 

Below is the last thing in the log before I have to restart the server. 

Can anyone suggest troubleshooting for something like this? I'm stuck. 

Help is appreciated. 

James

May 21 12:30:00 xen06 logger: /etc/xen/scripts/xen-hotplug-cleanup: 
XENBUS_PATH=backend/console/3/0
May 21 12:30:00 xen06 logger: /etc/xen/scripts/xen-hotplug-cleanup: 
XENBUS_PATH=backend/vfb/3/0
May 21 12:30:00 xen06 logger: /etc/xen/scripts/block: remove 
XENBUS_PATH=backend/vbd/3/768
May 21 12:30:00 xen06 logger: /etc/xen/scripts/vif-bridge: offline 
XENBUS_PATH=backend/vif/3/0
May 21 12:30:00 xen06 kernel: br0: port 3(vif3.0) entering disabled state
May 21 12:30:00 xen06 logger: /etc/xen/scripts/xen-hotplug-cleanup: 
XENBUS_PATH=backend/vbd/3/768
May 21 12:30:00 xen06 kernel: br0: port 3(vif3.0) entering disabled state
May 21 12:30:00 xen06 logger: /etc/xen/scripts/vif-bridge: brctl delif br0 
vif3.0 failed
May 21 12:30:00 xen06 ifdown:     vif3.0
May 21 12:30:00 xen06 logger: /etc/xen/scripts/vif-bridge: ifconfig vif3.0 down 
failed
May 21 12:30:00 xen06 ifdown: Interface not available and no configuration 
found.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 logger: /etc/xen/scripts/vif-bridge: Successful 
vif-bridge offline for vif3.0, bridge br0.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:00 xen06 logger: /etc/xen/scripts/xen-hotplug-cleanup: 
XENBUS_PATH=backend/vif/3/0
May 21 12:30:00 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:01 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:30:01 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:31:29 xen06 libvirtd: 12:31:29.391: error : 
xenInotifyXendDomainsDirLookup:163 : internal error finding dom on config list
May 21 12:31:29 xen06 libvirtd: 12:31:29.464: error : 
xenInotifyXendDomainsDirLookup:163 : internal error finding dom on config list
May 21 12:32:42 xen06 libvirtd: 12:32:42.463: error : 
xenInotifyXendDomainsDirLookup:163 : internal error finding dom on config list
May 21 12:32:42 xen06 libvirtd: 12:32:42.577: error : 
xenInotifyXendDomainsDirLookup:163 : internal error finding dom on config list
May 21 12:33:39 xen06 logger: /etc/xen/scripts/block: add 
XENBUS_PATH=backend/vbd/19/768
May 21 12:33:39 xen06 logger: /etc/xen/scripts/vif-bridge: online 
XENBUS_PATH=backend/vif/19/0
May 21 12:33:39 xen06 kernel: device tap19.0 entered promiscuous mode
May 21 12:33:39 xen06 kernel: br0: topology change detected, propagating
May 21 12:33:39 xen06 kernel: br0: port 2(tap19.0) entering forwarding state
May 21 12:33:39 xen06 kernel: device vif19.0 entered promiscuous mode
May 21 12:33:39 xen06 kernel: br0: topology change detected, propagating
May 21 12:33:39 xen06 kernel: br0: port 3(vif19.0) entering forwarding state
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:39 xen06 logger: /etc/xen/scripts/vif-bridge: iptables setup 
failed. This may affect guest networking.
May 21 12:33:39 xen06 ifup:     vif19.0
May 21 12:33:40 xen06 logger: /etc/xen/scripts/block: Writing 
backend/vbd/19/768/physical-device 8:10 to xenstore.
May 21 12:33:39 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:40 xen06 kernel: physdev match: using --physdev-out in the OUTPUT, 
FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore.
May 21 12:33:40 xen06 ifup:               No configuration found for vif19.0
May 21 12:33:40 xen06 logger: /etc/xen/scripts/vif-bridge: Successful 
vif-bridge online for vif19.0, bridge br0.
May 21 12:33:40 xen06 logger: /etc/xen/scripts/vif-bridge: Writing 
backend/vif/19/0/hotplug-status connected to xenstore.
May 21 12:33:40 xen06 logger: /etc/xen/scripts/block: Writing 
backend/vbd/19/768/hotplug-status connected to xenstore.
May 21 12:33:40 xen06 kernel: (cdrom_add_media_watch() 
file=drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/19/768
May 21 12:33:40 xen06 kernel: (cdrom_is_type() 
file=drivers/xen/blkback/cdrom.c, line=95) type:0
May 21 12:34:01 xen06 kernel: br0: port 2(tap19.0) entering disabled state
May 21 12:34:01 xen06 kernel: device tap19.0 left promiscuous mode
May 21 12:34:01 xen06 kernel: br0: port 2(tap19.0) entering disabled state
May 21 12:34:01 xen06 kernel: blkback: ring-ref 8, event-channel 5, protocol 2 
(x86_32-abi)



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users