Re: [Xen-users] 2 recently updated Centos6 Xen 4.6.3-5 server crashes

[George Dunlap <dunlapg@xxxxxxxxx>]

On Wed, Feb 8, 2017 at 4:12 PM, Brandon Shoemaker
<brandon.shoemaker@xxxxxxxxx> wrote:
> Hello list,
>
> I recently upgraded our Centos6 Xen servers to 4.6.3-5 about three weeks ago 
> and strangely this week we have had 2 servers crash with similar issues.  
> Nothing is able to be logged in /var/log/messages because the problem appears 
> to be disk related as the problem develops.  I took screenshots of console 
> for each server.
>
> https://postimg.org/image/3xmuun3gv/
> https://postimg.org/image/mw6ukvd93/
>
> It appears there are I/O errors, then the journal aborts on the root logical 
> volume, and then there is EXT4-fs errors, and then swap errors reported.
>
> Our monitoring manages to capture this information remotely before the crash. 
>  It was the same for both servers.
>
> CRIT - filesystem has switched to read-only and is probably corrupted CRIT, 
> missing: rw, exceeding: ro
>
> Centos 6.8 3.18.44-20.el6.x86_64
> xen version 4.6.3-5.el6
> 48gb RAM
> Samsung 850 pro 1 TB SSD (yes I know the consumer class of this drive)
> X10SRi-F motherboard 1.0b and 2.0a bios (two different servers running two 
> different BIOS versions)
>
> One server was about half full and the other server near full as for VPS disk 
> allocation but had plenty of CPU and RAM resources available.  Neither server 
> has heavy I/O characteristics.
>
> The servers have been in use without issue for months prior to this.
>
> I find it unlikely that both servers would have a SSD disk fail in the same 
> exact manner within a few days of each other so I'm suspicious that this is a 
> software bug.
>
> I'm wondering if this could have been caused by the recent Xen and or 
> operating system updates since it is coincidental that I am now having this 
> issue after recently updating when before the servers were stable.
>
> Has anyone else had any similar crashes recently like this?

When did you update your kernel?

We've had a lot of complaints about the 3.18.44 kernel.  That came out
in November, but if you hadn't updated since then that may be the
issue.  We're in the process of updating it to a 4.x kernel, but
Johnny Hughes has hit some snags.


 -George

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
https://lists.xen.org/xen-users