[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] libvirtd hang on CentOS6 after latest updates



On 07/22/2018 04:03 PM, Karel Hendrych wrote:
Hi, I am seeing frequent libvirtd hangs (clients not responding) after last CentOS6-Xen update :

xen-devel is not the best place to seek help with downstream issues, particularly libvirt ones :-). You would have better luck contacting the CentOS6 maintainers.

Regards,
Jim


libvirt-libs-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-driver-network-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-driver-nwfilter-4.1.0-2.xen46.el6.x86_64
libgcc-4.4.7-18.el6_9.2.x86_64
2:qemu-img-0.12.1.2-2.503.el6_9.5.x86_64
libvirt-daemon-driver-storage-core-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-driver-secret-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-driver-interface-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-driver-nodedev-4.1.0-2.xen46.el6.x86_64
10:centos-release-xen-common-8-4.el6.x86_64
xen-licenses-4.6.6-12.el6.x86_64
xen-libs-4.6.6-12.el6.x86_64
libvirt-daemon-driver-libxl-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-driver-xen-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-driver-qemu-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-driver-storage-gluster-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-driver-storage-logical-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-driver-storage-mpath-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-driver-storage-disk-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-driver-storage-scsi-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-driver-storage-iscsi-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-driver-storage-4.1.0-2.xen46.el6.x86_64
libstdc++-4.4.7-18.el6_9.2.x86_64
libvirt-daemon-config-nwfilter-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-config-network-4.1.0-2.xen46.el6.x86_64
libvirt-daemon-driver-lxc-4.1.0-2.xen46.el6.x86_64
libvirt-client-4.1.0-2.xen46.el6.x86_64
linux-firmware-20171215-82.git2451bb22.el6.noarch
12:dhcp-common-4.1.1-53.P1.el6.centos.4.x86_64
12:dhclient-4.1.1-53.P1.el6.centos.4.x86_64
libvirt-4.1.0-2.xen46.el6.x86_64
10:centos-release-xen-46-8-4.el6.x86_64
10:centos-release-xen-44-8-4.el6.x86_64
tzdata-2018e-3.el6.noarch
libgomp-4.4.7-18.el6_9.2.x86_64
kernel-4.9.86-30.el6.x86_64
xen-hypervisor-4.6.6-12.el6.x86_64
xen-runtime-4.6.6-12.el6.x86_64
xen-4.6.6-12.el6.x86_64
libvirt-daemon-xen-4.1.0-2.xen46.el6.x86_64

Remedy is to kill -9 libvirtd and start again. Issue can be replicated within few domU starts. Usually libvirtd hangs when domU is bringing up xen drivers or something around udev, like:

xen_netfront: Initialising Xen virtual ethernet driver

I've been looking into libvirtd strace and debug logs, so far most suspicious in libvirtd debug log is this:

libvirtd.log:2018-05-22 08:32:44.760+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-7' libvirtd.log:2018-05-22 08:32:44.761+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-6' libvirtd.log:2018-05-22 08:32:44.761+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-4' libvirtd.log:2018-05-22 08:32:44.762+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-5' libvirtd.log:2018-05-22 08:32:44.763+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-2' libvirtd.log:2018-05-22 08:32:44.764+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-3' libvirtd.log:2018-05-22 08:32:44.765+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-6' libvirtd.log:2018-05-22 08:32:44.766+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-5' libvirtd.log:2018-05-22 08:32:44.767+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-4' libvirtd.log:2018-05-22 08:32:44.767+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-7' libvirtd.log:2018-05-22 08:32:44.768+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-2' libvirtd.log:2018-05-22 08:32:44.769+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-3'

I could not get rid of this by reducing amount of driver queues (not sure if that applies to PV)

Is someone out there seeing similar issues? Anyone perhaps interested in reviewing full debug log / strace ?

Cheers
Karel


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.