RE: [Xen-devel] disk io errors possibly caused by high network load?

[Moritz Möller <m.moeller@xxxxxxxxxxxx>]

Okay - wrong key. Message continued

-----Original Message-----
From: Moritz Möller 
Sent: Friday, September 19, 2008 3:00 PM
To: 'Ian Pratt'; xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: RE: [Xen-devel] disk io errors possibly caused by high network
load?

We rebooted the machines really quickly because it was a productive
system, so I didn't have the time to copy the logs, and on the disks I
see nothing about this in the logfiles, propably because the IO was
already down.

The machines are Supermicro, Intel Xeon Quad or Dual-Quadcore, 8 to 32
GB RAM, and some have a mdraid setup with two SATA drives with the on
board sata controller (intel ICH), other have a dedicated 3ware / AMCC
9660 or similar.

The machines that crashed were on different power lines and connected to
different switches, although on the same network segment. Also there
were no physical interferences.

The error was reported by domU and dom0 - both saying the local disk
(either sda or sdb on mdraid systems, and sda on raid systems) reports a
I/O error, but no specific information.

Network card is intel e1000.

Lsmod:

nfs                   257112  1
w83792d                39320  0
w83781d                44840  0
i2c_isa                14720  1 w83781d
w83793                 46360  0
hwmon_vid              11264  2 w83781d,w83793
hwmon                  12040  3 w83792d,w83781d,w83793
ipmi_devintf           20112  0
ipmi_si                52812  0
ipmi_msghandler        47096  2 ipmi_devintf,ipmi_si
nls_utf8               10624  3
cifs                  228112  3
xt_physdev             11152  4
iptable_filter         11392  1
ip_tables              28648  1 iptable_filter
x_tables               29064  2 xt_physdev,ip_tables
ipv6                  339072  22
bridge                 64936  0
8021q                  29584  0
nfsd                  263848  1
exportfs               14336  1 nfsd
lockd                  74800  2 nfs,nfsd
nfs_acl                12160  2 nfs,nfsd
sunrpc                186344  5 nfs,nfsd,lockd,nfs_acl
blkbk                  30776  0 [permanent]
netbk                 105184  0 [permanent]
loop                   26768  0
8250_pnp               19968  0
sg                     45224  0
sr_mod                 26148  0
cdrom                  44072  1 sr_mod
i2c_i801               17052  0
iTCO_wdt               20432  0
iTCO_vendor_support    12548  1 iTCO_wdt
8250                   50120  1 8250_pnp
serial_core            31616  1 8250
i2c_core               32256  5 w83792d,w83781d,i2c_isa,w83793,i2c_i801
serio_raw              16004  0
pcspkr                 11776  0
joydev                 19584  0
ext3                  141200  2
jbd                    72432  1 ext3
mbcache                18184  1 ext3
dm_mirror              30528  0
dm_snapshot            25416  0
dm_mod                 69520  21 dm_mirror,dm_snapshot
raid1                  32768  3
sd_mod                 35200  8
usb_storage            90304  0
ata_piix               25092  6
ata_generic            17412  0
floppy                 68904  0
ehci_hcd               41100  0
uhci_hcd               32544  0
libata                126896  2 ata_piix,ata_generic
scsi_mod              166968  5 sg,sr_mod,sd_mod,usb_storage,libata
e1000                 130880  0
xenbus_be              12800  2 blkbk,netbk
xennet                 37512  0
xenblk                 26720  0




-----Original Message-----
From: Ian Pratt [mailto:Ian.Pratt@xxxxxxxxxxxxx] 
Sent: Friday, September 19, 2008 2:44 PM
To: Moritz Möller; xen-devel@xxxxxxxxxxxxxxxxxxx
Cc: Ian Pratt
Subject: RE: [Xen-devel] disk io errors possibly caused by high network
load?

> we had a very strange situation yesterday. In one second, 13 of 25 xen
> boxes died with disk errors (domU and dom0, something like
end_request:
> I/O error dev hda sector ...), but worked well again after a reboot.
> 
> Some minutes before a technician plugged in a wrong cable, creating a
> network loop - so the error could be caused by a high network io load.
> The disks are okay, and the error occurred with both scsi raid
> controllers and plain sata disks.

This is quite remarkable -- I don't think anyone has reported anything
similar before, despite there being many large xen deployments.

Are you saying that IO errors were reported from both dom0 and the
domU's? 

Did you actually track down the specific device major/minor that was
reporting the error?

Is there any network storage (e.g. iSCSI, AOE) in your setup?

Ian 

> Here is some info of a host that crashed:
> 
> root/mmoeller@srv002050:/root$ xm info
> host                   : srv002050
> release                : 2.6.21-2950.fc8xen
> version                : #1 SMP Tue Oct 23 12:23:33 EDT 2007
> machine                : x86_64
> nr_cpus                : 8
> nr_nodes               : 1
> cores_per_socket       : 4
> threads_per_core       : 1
> cpu_mhz                : 1866
> hw_caps                :
> bfebfbff:20100800:00000000:00000140:0004e3bd:00000000:00000001
> total_memory           : 8190
> free_memory            : 12
> node_to_cpu            : node0:0-7
> xen_major              : 3
> xen_minor              : 2
> xen_extra              : .0
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          : unavailable
> cc_compiler            : gcc version 4.1.2 20061115 (prerelease)
> (Debian
> 4.1.1-21)
> cc_compile_by          : root
> cc_compile_domain      : office.bigpoint.net
> cc_compile_date        : Tue Mar 11 13:57:28 CET 2008
> xend_config_format     : 4
> root/mmoeller@srv002050:/root$ uname -r
> 2.6.21-2950.fc8xen
> 
> And here of a host that did not crash:
> 
> root/mmoeller@srv006215:/root$ xm info
> host                   : srv006215
> release                : 2.6.21-2950.fc8xen
> version                : #1 SMP Tue Oct 23 12:23:33 EDT 2007
> machine                : x86_64
> nr_cpus                : 4
> nr_nodes               : 1
> cores_per_socket       : 4
> threads_per_core       : 1
> cpu_mhz                : 2394
> hw_caps                :
> bfebfbff:20100800:00000000:00000140:0000e3bd:00000000:00000001
> total_memory           : 8190
> free_memory            : 10
> node_to_cpu            : node0:0-3
> xen_major              : 3
> xen_minor              : 2
> xen_extra              : .0
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          : unavailable
> cc_compiler            : gcc version 4.1.2 20061115 (prerelease)
> (Debian
> 4.1.1-21)
> cc_compile_by          : root
> cc_compile_domain      : office.bigpoint.net
> cc_compile_date        : Tue Mar 11 13:57:28 CET 2008
> xend_config_format     : 4
> root/mmoeller@srv006215:/root$ uname -r
> 2.6.21-2950.fc8xen
> 
> Does someone have an idea how this could happen?
> 
> 
> Thanks,
> 
> 
> Moritz
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel