[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] Dom0 crashes without logging lately on Debian Stretch with Xen 4.8


  • To: Roalt Zijlstra | webpower <roalt.zijlstra@xxxxxxxxxxx>, xen-users@xxxxxxxxxxxxxxxxxxxx
  • From: Jean-Louis Dupond <jean-louis@xxxxxxxxx>
  • Date: Tue, 30 Oct 2018 12:12:39 +0100
  • Arc-authentication-results: i=1; ORIGINATING; auth=pass smtp.auth=jean-louis@xxxxxxxxx smtp.mailfrom=jean-louis@xxxxxxxxx
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=dupond.be; s=dkim; t=1540897960; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BoDA/wPayEcACP598kDXvtExobhwLhX7p/+EfCCs74U=; b=kj3pL8wpZnEkSontMtdD9ijW+qUrtwBNDsu54AVfifgbN/4O2QWOW0ZB+DFP1r2rKxnyIp abx1HgfEnxygix1udM856oxo8c9ee2T4a0rP6rG85p53HnWWlPoe1RD2SbdSe3Y1kuZUuW znm6abG9eL3ZB8v7UKliZsSaJ8UUHIKCpM8guR9bCadHsxgSfWnrrL+4+Mp6HIHII/izhM 7FYAF9wH39tphXmaH9v4NU9H4hGRh0ZjubjafAsABD0aGJN/8CBffEwErbF80YVMmNK8Zs f7FS06jZ6y7yy4jfnsArOIhDYUjvBO/y9V/Y3vi7V99dtHC8fCDBb/thsh+Jjg==
  • Arc-seal: i=1; s=dkim; d=dupond.be; t=1540897960; a=rsa-sha256; cv=none; b=f8ow5sb3DBsRS81+bPC91nSKNGxNrW7pl+gBRDrjRNnWyY7yXe/SZqMTnrwqZh+tK+84sBdmHfQF7U7tlc5OTH69F+pETgAy39qRmIhi8orD8Z/i7oxMCQ9g9sFB+jFqXTe5nM1XQ8pjrESUu3rwhuqRgTA5r596UTVUkLS1ZtGYXrLxT8i2aHRFBsGHgCQfNm6xzqGkx5b0KtX0+wJzXtSoOowU2JyKri0BcUUTAMQ5RT8/pmZaMti0713l9v04GGM04VPfk2cEPCF4K/9q5hcFbl2FDairy6/q+Q91zogsT0BqV0wBNL0vtUf20i/u1+PzRewvpJ5Qqx9q/TO3Lg==
  • Delivery-date: Tue, 30 Oct 2018 11:13:49 +0000
  • List-id: Xen user discussion <xen-users.lists.xenproject.org>

Hi Roalt,

We are running Xen 4.6 on 4.9.x kernel and CentOS 6, and are having the same issues.
But not that frequent as you state. Only like once every month.

The systems (Dell R630) also crashes/resets without any message. So nothing is logged unfortunately :(

The crashes were not observed on Xen 4.4.

We configured the servers to print kernel logs to SOL (Serial Over LAN via iDRAC), and we log those.
But since then no crashed servers anymore, so we don't know yet if this will give us some more details.


Thanks
Jean-Louis

On 29/10/18 12:57, Roalt Zijlstra | webpower wrote:
Hi there,

Ever since all the Meltdown and Spectre kernel updates and possibly also Xen 4.8 updates, we experience crashes of the Dom0 just out of the blue. Sometimes after 1 day, sometimes after a few days or even 14 days, completely random.

We have two Dell P730 servers and two Dell P720 servers with this behaviour. One thing is that we updated these machine to the latest available firmware, because that is the most secure way. Then we installed Debian Stretch with Xen 4.8 support

We have done serveral installs and 4 servers seem to crash pretty fast and other don't. In the end we think that we can lead it back to the xen-4.8.4-pre version being stable and the xen-4.8.5-pre being unstable. This was kinda independent of the kernel that we were using 4.14 or 4.9.0-8-amd64. This is off course all Debian package numbering.

As last resort  we updated on one server all DomU kernels of our Jessie servers on this Dom0 to 4.9.0 from backports instead of the 3.16 kernel. For now that seems to work, but the crashes are random so it could happen any time again. The idea is that these kernels are completely spectre& meltdown unaware and might cause trouble in Xen kernel support. I am not sure if this is true at all, but we are pretty lost what the actual cause is.

We also tested with CentOS and we also had these crashes there with certain combinations of kernel/Xen. The most recent updates seem to be more stable tough. The most frustrating part is the there is absolutely no logs to be found. No kernel oops or what.. the server just resets and boots again.

Are there others experiencing problems like this? Do you see more frequent server/kernel crashes on production servers?  

Best regards,
 
Roalt Zijlstra


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.