May 3 12:00:56 node2-1 kernel: [823831.223139] drbd resource5: PingAck did not arrive in time. May 3 12:00:56 node2-1 kernel: [823831.223170] drbd resource5: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) May 3 12:00:56 node2-1 kernel: [823831.223213] block drbd5: new current UUID 3E7DEAF33A27E26B:A0BD25A8FD641CC9:B4B6F34A07F38E51:B4B5F34A07F38E51 May 3 12:00:56 node2-1 kernel: [823831.274520] drbd resource5: asender terminated May 3 12:00:56 node2-1 kernel: [823831.274526] drbd resource5: Terminating drbd_a_resource May 3 12:00:56 node2-1 kernel: [823831.282885] drbd resource5: Connection closed May 3 12:00:56 node2-1 kernel: [823831.282964] drbd resource5: conn( NetworkFailure -> Unconnected ) May 3 12:00:56 node2-1 kernel: [823831.282966] drbd resource5: receiver terminated May 3 12:00:56 node2-1 kernel: [823831.282967] drbd resource5: Restarting receiver thread May 3 12:00:56 node2-1 kernel: [823831.282968] drbd resource5: receiver (re)started May 3 12:00:56 node2-1 kernel: [823831.282973] drbd resource5: conn( Unconnected -> WFConnection ) May 3 12:00:57 node2-1 kernel: [823832.343138] drbd resource4: PingAck did not arrive in time. May 3 12:00:57 node2-1 kernel: [823832.343184] drbd resource4: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) May 3 12:00:57 node2-1 kernel: [823832.343238] block drbd4: new current UUID 34773250B235E1C9:9AF08F3F70299491:EC0A7F59E439C05B:EC097F59E439C05B May 3 12:00:57 node2-1 kernel: [823832.359941] drbd resource4: asender terminated May 3 12:00:57 node2-1 kernel: [823832.359945] drbd resource4: Terminating drbd_a_resource May 3 12:00:57 node2-1 kernel: [823832.368264] drbd resource4: Connection closed May 3 12:00:57 node2-1 kernel: [823832.368351] drbd resource4: conn( NetworkFailure -> Unconnected ) May 3 12:00:57 node2-1 kernel: [823832.368353] drbd resource4: receiver terminated May 3 12:00:57 node2-1 kernel: [823832.368354] drbd resource4: Restarting receiver thread May 3 12:00:57 node2-1 kernel: [823832.368355] drbd resource4: receiver (re)started May 3 12:00:57 node2-1 kernel: [823832.368361] drbd resource4: conn( Unconnected -> WFConnection ) May 3 12:00:59 node2-1 kernel: [823834.207075] drbd resource2: PingAck did not arrive in time. May 3 12:00:59 node2-1 kernel: [823834.207107] drbd resource2: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) May 3 12:00:59 node2-1 kernel: [823834.284603] drbd resource2: asender terminated May 3 12:00:59 node2-1 kernel: [823834.284607] drbd resource2: Terminating drbd_a_resource May 3 12:00:59 node2-1 kernel: [823834.284672] drbd resource2: Connection closed May 3 12:00:59 node2-1 kernel: [823834.284680] drbd resource2: conn( NetworkFailure -> Unconnected ) May 3 12:00:59 node2-1 kernel: [823834.284682] drbd resource2: receiver terminated May 3 12:00:59 node2-1 kernel: [823834.284682] drbd resource2: Restarting receiver thread May 3 12:00:59 node2-1 kernel: [823834.284683] drbd resource2: receiver (re)started May 3 12:00:59 node2-1 kernel: [823834.284687] drbd resource2: conn( Unconnected -> WFConnection ) May 3 12:01:00 node2-1 kernel: [823834.443224] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:00 node2-1 kernel: [823834.443224] Tx Queue <1> May 3 12:01:00 node2-1 kernel: [823834.443224] TDH, TDT <165>, <16d> May 3 12:01:00 node2-1 kernel: [823834.443224] next_to_use <16d> May 3 12:01:00 node2-1 kernel: [823834.443224] next_to_clean <165> May 3 12:01:00 node2-1 kernel: [823834.443224] tx_buffer_info[next_to_clean] May 3 12:01:00 node2-1 kernel: [823834.443224] time_stamp <10c458e08> May 3 12:01:00 node2-1 kernel: [823834.443224] jiffies <10c459092> May 3 12:01:00 node2-1 kernel: [823834.443234] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:00 node2-1 kernel: [823834.443234] Tx Queue <3> May 3 12:01:00 node2-1 kernel: [823834.443234] TDH, TDT , May 3 12:01:00 node2-1 kernel: [823834.443234] next_to_use May 3 12:01:00 node2-1 kernel: [823834.443234] next_to_clean May 3 12:01:00 node2-1 kernel: [823834.443234] tx_buffer_info[next_to_clean] May 3 12:01:00 node2-1 kernel: [823834.443234] time_stamp <10c458e30> May 3 12:01:00 node2-1 kernel: [823834.443234] jiffies <10c459092> May 3 12:01:00 node2-1 kernel: [823834.443235] ixgbe 0000:02:00.0 eth0: tx hang 1 detected on queue 3, resetting adapter May 3 12:01:00 node2-1 kernel: [823834.443236] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout May 3 12:01:00 node2-1 kernel: [823834.443245] ixgbe 0000:02:00.0 eth0: Reset adapter May 3 12:01:00 node2-1 kernel: [823834.443366] ixgbe 0000:02:00.0 eth0: tx hang 2 detected on queue 1, resetting adapter May 3 12:01:00 node2-1 kernel: [823834.633614] ixgbe 0000:02:00.0: master disable timed out May 3 12:01:00 node2-1 kernel: [823834.835059] ixgbe 0000:02:00.0 eth0: detected SFP+: 3 May 3 12:01:01 node2-1 kernel: [823835.851075] drbd resource1: PingAck did not arrive in time. May 3 12:01:01 node2-1 kernel: [823835.851137] drbd resource1: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) May 3 12:01:01 node2-1 kernel: [823835.881036] drbd resource1: asender terminated May 3 12:01:01 node2-1 kernel: [823835.881045] drbd resource1: Terminating drbd_a_resource May 3 12:01:01 node2-1 kernel: [823835.881129] drbd resource1: Connection closed May 3 12:01:01 node2-1 kernel: [823835.881136] drbd resource1: conn( NetworkFailure -> Unconnected ) May 3 12:01:01 node2-1 kernel: [823835.881137] drbd resource1: receiver terminated May 3 12:01:01 node2-1 kernel: [823835.881138] drbd resource1: Restarting receiver thread May 3 12:01:01 node2-1 kernel: [823835.881139] drbd resource1: receiver (re)started May 3 12:01:01 node2-1 kernel: [823835.881143] drbd resource1: conn( Unconnected -> WFConnection ) May 3 12:01:02 node2-1 kernel: [823836.679074] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 3 12:01:02 node2-1 kernel: [823836.879198] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:02 node2-1 kernel: [823836.879198] Tx Queue <4> May 3 12:01:02 node2-1 kernel: [823836.879198] TDH, TDT <0>, <4> May 3 12:01:02 node2-1 kernel: [823836.879198] next_to_use <4> May 3 12:01:02 node2-1 kernel: [823836.879198] next_to_clean <0> May 3 12:01:02 node2-1 kernel: [823836.879198] tx_buffer_info[next_to_clean] May 3 12:01:02 node2-1 kernel: [823836.879198] time_stamp <10c4592da> May 3 12:01:02 node2-1 kernel: [823836.879198] jiffies <10c4592f3> May 3 12:01:02 node2-1 kernel: [823836.879404] ixgbe 0000:02:00.0 eth0: tx hang 2 detected on queue 4, resetting adapter May 3 12:01:02 node2-1 kernel: [823836.879407] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout May 3 12:01:02 node2-1 kernel: [823836.879463] ixgbe 0000:02:00.0 eth0: Reset adapter May 3 12:01:02 node2-1 kernel: [823836.879601] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:02 node2-1 kernel: [823836.879759] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:02 node2-1 kernel: [823836.879915] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:02 node2-1 kernel: [823836.880071] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:02 node2-1 kernel: [823836.880227] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:02 node2-1 kernel: [823836.880383] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:02 node2-1 kernel: [823836.880539] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:02 node2-1 kernel: [823836.880696] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:02 node2-1 kernel: [823837.070993] ixgbe 0000:02:00.0: master disable timed out May 3 12:01:02 node2-1 kernel: [823837.091387] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:02 node2-1 kernel: [823837.093673] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:02 node2-1 kernel: [823837.095263] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:02 node2-1 kernel: [823837.097531] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:02 node2-1 kernel: [823837.099236] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:02 node2-1 kernel: [823837.101456] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:02 node2-1 kernel: [823837.103188] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:02 node2-1 kernel: [823837.105454] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:02 node2-1 kernel: [823837.271018] ixgbe 0000:02:00.0 eth0: detected SFP+: 3 May 3 12:01:03 node2-1 kernel: [823838.311064] drbd resource0: PingAck did not arrive in time. May 3 12:01:03 node2-1 kernel: [823838.311114] drbd resource0: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) May 3 12:01:03 node2-1 kernel: [823838.311158] block drbd0: new current UUID E0DD45FF99952B91:ECEAA5796E85E1C9:3D70B072BA877A8F:3D6FB072BA877A8F May 3 12:01:03 node2-1 kernel: [823838.388495] drbd resource0: asender terminated May 3 12:01:03 node2-1 kernel: [823838.388503] drbd resource0: Terminating drbd_a_resource May 3 12:01:03 node2-1 kernel: [823838.396892] drbd resource0: Connection closed May 3 12:01:03 node2-1 kernel: [823838.397024] drbd resource0: conn( NetworkFailure -> Unconnected ) May 3 12:01:03 node2-1 kernel: [823838.397026] drbd resource0: receiver terminated May 3 12:01:03 node2-1 kernel: [823838.397027] drbd resource0: Restarting receiver thread May 3 12:01:03 node2-1 kernel: [823838.397028] drbd resource0: receiver (re)started May 3 12:01:03 node2-1 kernel: [823838.397033] drbd resource0: conn( Unconnected -> WFConnection ) May 3 12:01:04 node2-1 kernel: [823838.595008] drbd resource3: PingAck did not arrive in time. May 3 12:01:04 node2-1 kernel: [823838.595060] drbd resource3: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) May 3 12:01:04 node2-1 kernel: [823838.608577] drbd resource3: asender terminated May 3 12:01:04 node2-1 kernel: [823838.608581] drbd resource3: Terminating drbd_a_resource May 3 12:01:04 node2-1 kernel: [823838.608649] drbd resource3: Connection closed May 3 12:01:04 node2-1 kernel: [823838.608656] drbd resource3: conn( NetworkFailure -> Unconnected ) May 3 12:01:04 node2-1 kernel: [823838.608657] drbd resource3: receiver terminated May 3 12:01:04 node2-1 kernel: [823838.608658] drbd resource3: Restarting receiver thread May 3 12:01:04 node2-1 kernel: [823838.608659] drbd resource3: receiver (re)started May 3 12:01:04 node2-1 kernel: [823838.608663] drbd resource3: conn( Unconnected -> WFConnection ) May 3 12:01:04 node2-1 kernel: [823839.123017] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 3 12:01:06 node2-1 kernel: [823841.327099] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:06 node2-1 kernel: [823841.327099] Tx Queue <3> May 3 12:01:06 node2-1 kernel: [823841.327099] TDH, TDT <0>, <2> May 3 12:01:06 node2-1 kernel: [823841.327099] next_to_use <2> May 3 12:01:06 node2-1 kernel: [823841.327099] next_to_clean <0> May 3 12:01:06 node2-1 kernel: [823841.327099] tx_buffer_info[next_to_clean] May 3 12:01:06 node2-1 kernel: [823841.327099] time_stamp <10c459542> May 3 12:01:06 node2-1 kernel: [823841.327099] jiffies <10c45974b> May 3 12:01:06 node2-1 kernel: [823841.327106] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:06 node2-1 kernel: [823841.327106] Tx Queue <0> May 3 12:01:06 node2-1 kernel: [823841.327106] TDH, TDT <0>, <2> May 3 12:01:06 node2-1 kernel: [823841.327106] next_to_use <2> May 3 12:01:06 node2-1 kernel: [823841.327106] next_to_clean <0> May 3 12:01:06 node2-1 kernel: [823841.327106] tx_buffer_info[next_to_clean] May 3 12:01:06 node2-1 kernel: [823841.327106] time_stamp <10c459556> May 3 12:01:06 node2-1 kernel: [823841.327106] jiffies <10c45974b> May 3 12:01:06 node2-1 kernel: [823841.327107] ixgbe 0000:02:00.0 eth0: tx hang 3 detected on queue 0, resetting adapter May 3 12:01:06 node2-1 kernel: [823841.327109] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout May 3 12:01:06 node2-1 kernel: [823841.327120] ixgbe 0000:02:00.0 eth0: Reset adapter May 3 12:01:06 node2-1 kernel: [823841.327229] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:06 node2-1 kernel: [823841.327342] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:06 node2-1 kernel: [823841.327454] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:06 node2-1 kernel: [823841.327564] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:06 node2-1 kernel: [823841.327671] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:06 node2-1 kernel: [823841.327779] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:06 node2-1 kernel: [823841.327851] ixgbe 0000:02:00.0 eth0: tx hang 4 detected on queue 3, resetting adapter May 3 12:01:06 node2-1 kernel: [823841.327886] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:06 node2-1 kernel: [823841.328046] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:07 node2-1 kernel: [823841.518278] ixgbe 0000:02:00.0: master disable timed out May 3 12:01:07 node2-1 kernel: [823841.538765] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:07 node2-1 kernel: [823841.541012] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:07 node2-1 kernel: [823841.543200] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:07 node2-1 kernel: [823841.545505] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:07 node2-1 kernel: [823841.547126] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:07 node2-1 kernel: [823841.549361] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:07 node2-1 kernel: [823841.551185] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:07 node2-1 kernel: [823841.553457] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:07 node2-1 kernel: [823841.723057] ixgbe 0000:02:00.0 eth0: detected SFP+: 3 May 3 12:01:12 node2-1 kernel: [823846.737022] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 3 12:01:15 node2-1 kernel: [823849.870988] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:15 node2-1 kernel: [823849.870988] Tx Queue <5> May 3 12:01:15 node2-1 kernel: [823849.870988] TDH, TDT <0>, <2> May 3 12:01:15 node2-1 kernel: [823849.870988] next_to_use <2> May 3 12:01:15 node2-1 kernel: [823849.870988] next_to_clean <0> May 3 12:01:15 node2-1 kernel: [823849.870988] tx_buffer_info[next_to_clean] May 3 12:01:15 node2-1 kernel: [823849.870988] time_stamp <10c459d0c> May 3 12:01:15 node2-1 kernel: [823849.870988] jiffies <10c459fa3> May 3 12:01:15 node2-1 kernel: [823849.871195] ixgbe 0000:02:00.0 eth0: tx hang 4 detected on queue 5, resetting adapter May 3 12:01:15 node2-1 kernel: [823849.871198] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout May 3 12:01:15 node2-1 kernel: [823849.871257] ixgbe 0000:02:00.0 eth0: Reset adapter May 3 12:01:15 node2-1 kernel: [823849.871394] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:15 node2-1 kernel: [823849.871551] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:15 node2-1 kernel: [823849.871707] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:15 node2-1 kernel: [823849.871863] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:15 node2-1 kernel: [823849.872019] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:15 node2-1 kernel: [823849.872176] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:15 node2-1 kernel: [823849.872332] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:15 node2-1 kernel: [823849.872488] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:15 node2-1 kernel: [823850.062858] ixgbe 0000:02:00.0: master disable timed out May 3 12:01:15 node2-1 kernel: [823850.083156] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:15 node2-1 kernel: [823850.085389] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:15 node2-1 kernel: [823850.087015] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:15 node2-1 kernel: [823850.089257] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:15 node2-1 kernel: [823850.091004] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:15 node2-1 kernel: [823850.093278] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:15 node2-1 kernel: [823850.095060] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:15 node2-1 kernel: [823850.097314] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:15 node2-1 kernel: [823850.262798] ixgbe 0000:02:00.0 eth0: detected SFP+: 3 May 3 12:01:17 node2-1 kernel: [823852.106788] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 3 12:01:21 node2-1 kernel: [823856.314846] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:21 node2-1 kernel: [823856.314846] Tx Queue <5> May 3 12:01:21 node2-1 kernel: [823856.314846] TDH, TDT <0>, <2> May 3 12:01:21 node2-1 kernel: [823856.314846] next_to_use <2> May 3 12:01:21 node2-1 kernel: [823856.314846] next_to_clean <0> May 3 12:01:21 node2-1 kernel: [823856.314846] tx_buffer_info[next_to_clean] May 3 12:01:21 node2-1 kernel: [823856.314846] time_stamp <10c45a20b> May 3 12:01:21 node2-1 kernel: [823856.314846] jiffies <10c45a5ee> May 3 12:01:21 node2-1 kernel: [823856.314848] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:21 node2-1 kernel: [823856.314848] Tx Queue <1> May 3 12:01:21 node2-1 kernel: [823856.314848] TDH, TDT <0>, <2> May 3 12:01:21 node2-1 kernel: [823856.314848] next_to_use <2> May 3 12:01:21 node2-1 kernel: [823856.314848] next_to_clean <0> May 3 12:01:21 node2-1 kernel: [823856.314848] tx_buffer_info[next_to_clean] May 3 12:01:21 node2-1 kernel: [823856.314848] time_stamp <10c45a39d> May 3 12:01:21 node2-1 kernel: [823856.314848] jiffies <10c45a5ee> May 3 12:01:21 node2-1 kernel: [823856.314854] ixgbe 0000:02:00.0 eth0: tx hang 5 detected on queue 1, resetting adapter May 3 12:01:21 node2-1 kernel: [823856.314856] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:21 node2-1 kernel: [823856.314856] Tx Queue <2> May 3 12:01:21 node2-1 kernel: [823856.314856] TDH, TDT <0>, <2> May 3 12:01:21 node2-1 kernel: [823856.314856] next_to_use <2> May 3 12:01:21 node2-1 kernel: [823856.314856] next_to_clean <0> May 3 12:01:21 node2-1 kernel: [823856.314856] tx_buffer_info[next_to_clean] May 3 12:01:21 node2-1 kernel: [823856.314856] time_stamp <10c45a28d> May 3 12:01:21 node2-1 kernel: [823856.314856] jiffies <10c45a5ee> May 3 12:01:21 node2-1 kernel: [823856.314856] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout May 3 12:01:21 node2-1 kernel: [823856.314857] ixgbe 0000:02:00.0 eth0: tx hang 5 detected on queue 2, resetting adapter May 3 12:01:21 node2-1 kernel: [823856.314858] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout May 3 12:01:21 node2-1 kernel: [823856.314866] ixgbe 0000:02:00.0 eth0: Reset adapter May 3 12:01:21 node2-1 kernel: [823856.314977] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:21 node2-1 kernel: [823856.315085] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:21 node2-1 kernel: [823856.315196] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:21 node2-1 kernel: [823856.315306] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:21 node2-1 kernel: [823856.315416] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:21 node2-1 kernel: [823856.315523] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:21 node2-1 kernel: [823856.315634] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:21 node2-1 kernel: [823856.315744] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:21 node2-1 kernel: [823856.315865] ixgbe 0000:02:00.0 eth0: tx hang 6 detected on queue 5, resetting adapter May 3 12:01:22 node2-1 kernel: [823856.506084] ixgbe 0000:02:00.0: master disable timed out May 3 12:01:22 node2-1 kernel: [823856.526772] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:22 node2-1 kernel: [823856.535860] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:22 node2-1 kernel: [823856.538188] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:22 node2-1 kernel: [823856.540470] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:22 node2-1 kernel: [823856.542729] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:22 node2-1 kernel: [823856.544972] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:22 node2-1 kernel: [823856.546910] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:22 node2-1 kernel: [823856.549172] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:22 node2-1 kernel: [823856.714608] ixgbe 0000:02:00.0 eth0: detected SFP+: 3 May 3 12:01:24 node2-1 kernel: [823858.558670] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 3 12:01:28 node2-1 kernel: [823862.766738] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:28 node2-1 kernel: [823862.766738] Tx Queue <3> May 3 12:01:28 node2-1 kernel: [823862.766738] TDH, TDT <0>, <2> May 3 12:01:28 node2-1 kernel: [823862.766738] next_to_use <2> May 3 12:01:28 node2-1 kernel: [823862.766738] next_to_clean <0> May 3 12:01:28 node2-1 kernel: [823862.766738] tx_buffer_info[next_to_clean] May 3 12:01:28 node2-1 kernel: [823862.766738] time_stamp <10c45a990> May 3 12:01:28 node2-1 kernel: [823862.766738] jiffies <10c45ac3b> May 3 12:01:28 node2-1 kernel: [823862.766745] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:28 node2-1 kernel: [823862.766745] Tx Queue <1> May 3 12:01:28 node2-1 kernel: [823862.766745] TDH, TDT <0>, <4> May 3 12:01:28 node2-1 kernel: [823862.766745] next_to_use <4> May 3 12:01:28 node2-1 kernel: [823862.766745] next_to_clean <0> May 3 12:01:28 node2-1 kernel: [823862.766745] tx_buffer_info[next_to_clean] May 3 12:01:28 node2-1 kernel: [823862.766745] time_stamp <10c45a882> May 3 12:01:28 node2-1 kernel: [823862.766745] jiffies <10c45ac3b> May 3 12:01:28 node2-1 kernel: [823862.766747] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:28 node2-1 kernel: [823862.766747] Tx Queue <5> May 3 12:01:28 node2-1 kernel: [823862.766747] TDH, TDT <0>, <1> May 3 12:01:28 node2-1 kernel: [823862.766747] next_to_use <1> May 3 12:01:28 node2-1 kernel: [823862.766747] next_to_clean <0> May 3 12:01:28 node2-1 kernel: [823862.766747] tx_buffer_info[next_to_clean] May 3 12:01:28 node2-1 kernel: [823862.766747] time_stamp <10c45a8e4> May 3 12:01:28 node2-1 kernel: [823862.766747] jiffies <10c45ac3b> May 3 12:01:28 node2-1 kernel: [823862.766748] ixgbe 0000:02:00.0 eth0: tx hang 6 detected on queue 1, resetting adapter May 3 12:01:28 node2-1 kernel: [823862.766748] ixgbe 0000:02:00.0 eth0: tx hang 6 detected on queue 5, resetting adapter May 3 12:01:28 node2-1 kernel: [823862.766749] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout May 3 12:01:28 node2-1 kernel: [823862.766749] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout May 3 12:01:28 node2-1 kernel: [823862.766757] ixgbe 0000:02:00.0 eth0: Reset adapter May 3 12:01:28 node2-1 kernel: [823862.766867] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:28 node2-1 kernel: [823862.766981] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:28 node2-1 kernel: [823862.767093] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:28 node2-1 kernel: [823862.767208] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:28 node2-1 kernel: [823862.767323] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:28 node2-1 kernel: [823862.767431] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:28 node2-1 kernel: [823862.767538] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:28 node2-1 kernel: [823862.767652] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:28 node2-1 kernel: [823862.767785] ixgbe 0000:02:00.0 eth0: tx hang 7 detected on queue 3, resetting adapter May 3 12:01:28 node2-1 kernel: [823862.957938] ixgbe 0000:02:00.0: master disable timed out May 3 12:01:28 node2-1 kernel: [823862.978416] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:28 node2-1 kernel: [823862.980705] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:28 node2-1 kernel: [823862.982794] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:28 node2-1 kernel: [823862.985058] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:28 node2-1 kernel: [823862.986747] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:28 node2-1 kernel: [823862.988973] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:28 node2-1 kernel: [823862.990744] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:28 node2-1 kernel: [823862.993039] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:28 node2-1 kernel: [823863.158617] ixgbe 0000:02:00.0 eth0: detected SFP+: 3 May 3 12:01:30 node2-1 kernel: [823865.002605] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 3 12:01:34 node2-1 kernel: [823869.214591] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:34 node2-1 kernel: [823869.214591] Tx Queue <0> May 3 12:01:34 node2-1 kernel: [823869.214591] TDH, TDT <0>, <1> May 3 12:01:34 node2-1 kernel: [823869.214591] next_to_use <1> May 3 12:01:34 node2-1 kernel: [823869.214591] next_to_clean <0> May 3 12:01:34 node2-1 kernel: [823869.214591] tx_buffer_info[next_to_clean] May 3 12:01:34 node2-1 kernel: [823869.214591] time_stamp <10c45aeb0> May 3 12:01:34 node2-1 kernel: [823869.214591] jiffies <10c45b287> May 3 12:01:34 node2-1 kernel: [823869.214593] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:34 node2-1 kernel: [823869.214593] Tx Queue <3> May 3 12:01:34 node2-1 kernel: [823869.214593] TDH, TDT <0>, <1> May 3 12:01:34 node2-1 kernel: [823869.214593] next_to_use <1> May 3 12:01:34 node2-1 kernel: [823869.214593] next_to_clean <0> May 3 12:01:34 node2-1 kernel: [823869.214593] tx_buffer_info[next_to_clean] May 3 12:01:34 node2-1 kernel: [823869.214593] time_stamp <10c45af00> May 3 12:01:34 node2-1 kernel: [823869.214593] jiffies <10c45b287> May 3 12:01:34 node2-1 kernel: [823869.214598] ixgbe 0000:02:00.0 eth0: tx hang 7 detected on queue 3, resetting adapter May 3 12:01:34 node2-1 kernel: [823869.214599] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout May 3 12:01:34 node2-1 kernel: [823869.214611] ixgbe 0000:02:00.0 eth0: Reset adapter May 3 12:01:34 node2-1 kernel: [823869.214719] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:34 node2-1 kernel: [823869.214828] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:34 node2-1 kernel: [823869.214937] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:34 node2-1 kernel: [823869.215044] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:34 node2-1 kernel: [823869.215153] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:34 node2-1 kernel: [823869.215261] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:34 node2-1 kernel: [823869.215267] ixgbe 0000:02:00.0 eth0: tx hang 8 detected on queue 0, resetting adapter May 3 12:01:34 node2-1 kernel: [823869.215423] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:34 node2-1 kernel: [823869.215584] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:34 node2-1 kernel: [823869.405902] ixgbe 0000:02:00.0: master disable timed out May 3 12:01:35 node2-1 kernel: [823869.426460] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:35 node2-1 kernel: [823869.428701] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:35 node2-1 kernel: [823869.430666] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:35 node2-1 kernel: [823869.432953] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:35 node2-1 kernel: [823869.434657] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:35 node2-1 kernel: [823869.436892] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:35 node2-1 kernel: [823869.438639] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:35 node2-1 kernel: [823869.440864] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:35 node2-1 kernel: [823869.606474] ixgbe 0000:02:00.0 eth0: detected SFP+: 3 May 3 12:01:37 node2-1 kernel: [823871.450509] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 3 12:01:41 node2-1 kernel: [823875.658519] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:41 node2-1 kernel: [823875.658519] Tx Queue <5> May 3 12:01:41 node2-1 kernel: [823875.658519] TDH, TDT <0>, <1> May 3 12:01:41 node2-1 kernel: [823875.658519] next_to_use <1> May 3 12:01:41 node2-1 kernel: [823875.658519] next_to_clean <0> May 3 12:01:41 node2-1 kernel: [823875.658519] tx_buffer_info[next_to_clean] May 3 12:01:41 node2-1 kernel: [823875.658519] time_stamp <10c45b5aa> May 3 12:01:41 node2-1 kernel: [823875.658519] jiffies <10c45b8d2> May 3 12:01:41 node2-1 kernel: [823875.658530] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:41 node2-1 kernel: [823875.658530] Tx Queue <2> May 3 12:01:41 node2-1 kernel: [823875.658530] TDH, TDT <0>, <1> May 3 12:01:41 node2-1 kernel: [823875.658530] next_to_use <1> May 3 12:01:41 node2-1 kernel: [823875.658530] next_to_clean <0> May 3 12:01:41 node2-1 kernel: [823875.658530] tx_buffer_info[next_to_clean] May 3 12:01:41 node2-1 kernel: [823875.658530] time_stamp <10c45b5aa> May 3 12:01:41 node2-1 kernel: [823875.658530] jiffies <10c45b8d2> May 3 12:01:41 node2-1 kernel: [823875.658532] ixgbe 0000:02:00.0 eth0: tx hang 8 detected on queue 2, resetting adapter May 3 12:01:41 node2-1 kernel: [823875.658533] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout May 3 12:01:41 node2-1 kernel: [823875.658546] ixgbe 0000:02:00.0 eth0: Reset adapter May 3 12:01:41 node2-1 kernel: [823875.658656] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:41 node2-1 kernel: [823875.658763] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:41 node2-1 kernel: [823875.658874] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:41 node2-1 kernel: [823875.658982] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:41 node2-1 kernel: [823875.659091] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:41 node2-1 kernel: [823875.659193] ixgbe 0000:02:00.0 eth0: tx hang 9 detected on queue 5, resetting adapter May 3 12:01:41 node2-1 kernel: [823875.659197] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:41 node2-1 kernel: [823875.659358] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:41 node2-1 kernel: [823875.659514] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:41 node2-1 kernel: [823875.849777] ixgbe 0000:02:00.0: master disable timed out May 3 12:01:41 node2-1 kernel: [823875.870245] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:41 node2-1 kernel: [823875.872526] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:41 node2-1 kernel: [823875.874588] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:41 node2-1 kernel: [823875.876877] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:41 node2-1 kernel: [823875.878596] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:41 node2-1 kernel: [823875.880832] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:41 node2-1 kernel: [823875.882533] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:41 node2-1 kernel: [823875.884804] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:41 node2-1 kernel: [823876.050329] ixgbe 0000:02:00.0 eth0: detected SFP+: 3 May 3 12:01:43 node2-1 kernel: [823877.894326] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 3 12:01:47 node2-1 kernel: [823882.102372] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:47 node2-1 kernel: [823882.102372] Tx Queue <0> May 3 12:01:47 node2-1 kernel: [823882.102372] TDH, TDT <0>, <1> May 3 12:01:47 node2-1 kernel: [823882.102372] next_to_use <1> May 3 12:01:47 node2-1 kernel: [823882.102372] next_to_clean <0> May 3 12:01:47 node2-1 kernel: [823882.102372] tx_buffer_info[next_to_clean] May 3 12:01:47 node2-1 kernel: [823882.102372] time_stamp <10c45bc38> May 3 12:01:47 node2-1 kernel: [823882.102372] jiffies <10c45bf1d> May 3 12:01:47 node2-1 kernel: [823882.102391] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:47 node2-1 kernel: [823882.102391] Tx Queue <5> May 3 12:01:47 node2-1 kernel: [823882.102391] TDH, TDT <0>, <1> May 3 12:01:47 node2-1 kernel: [823882.102391] next_to_use <1> May 3 12:01:47 node2-1 kernel: [823882.102391] next_to_clean <0> May 3 12:01:47 node2-1 kernel: [823882.102391] tx_buffer_info[next_to_clean] May 3 12:01:47 node2-1 kernel: [823882.102391] time_stamp <10c45bb86> May 3 12:01:47 node2-1 kernel: [823882.102391] jiffies <10c45bf1d> May 3 12:01:47 node2-1 kernel: [823882.102393] ixgbe 0000:02:00.0 eth0: tx hang 9 detected on queue 5, resetting adapter May 3 12:01:47 node2-1 kernel: [823882.102393] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout May 3 12:01:47 node2-1 kernel: [823882.102405] ixgbe 0000:02:00.0 eth0: Reset adapter May 3 12:01:47 node2-1 kernel: [823882.102520] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:47 node2-1 kernel: [823882.102635] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:47 node2-1 kernel: [823882.102749] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:47 node2-1 kernel: [823882.102861] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:47 node2-1 kernel: [823882.102973] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:47 node2-1 kernel: [823882.103087] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:47 node2-1 kernel: [823882.103201] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:47 node2-1 kernel: [823882.103313] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:47 node2-1 kernel: [823882.109908] ixgbe 0000:02:00.0 eth0: tx hang 10 detected on queue 0, resetting adapter May 3 12:01:47 node2-1 kernel: [823882.293594] ixgbe 0000:02:00.0: master disable timed out May 3 12:01:47 node2-1 kernel: [823882.314106] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:47 node2-1 kernel: [823882.316342] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:47 node2-1 kernel: [823882.318424] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:47 node2-1 kernel: [823882.320714] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:47 node2-1 kernel: [823882.322449] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:47 node2-1 kernel: [823882.324730] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:47 node2-1 kernel: [823882.326478] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:47 node2-1 kernel: [823882.328710] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:48 node2-1 kernel: [823882.498260] ixgbe 0000:02:00.0 eth0: detected SFP+: 3 May 3 12:01:49 node2-1 kernel: [823884.342218] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 3 12:01:54 node2-1 kernel: [823888.550284] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:54 node2-1 kernel: [823888.550284] Tx Queue <2> May 3 12:01:54 node2-1 kernel: [823888.550284] TDH, TDT <0>, <1> May 3 12:01:54 node2-1 kernel: [823888.550284] next_to_use <1> May 3 12:01:54 node2-1 kernel: [823888.550284] next_to_clean <0> May 3 12:01:54 node2-1 kernel: [823888.550284] tx_buffer_info[next_to_clean] May 3 12:01:54 node2-1 kernel: [823888.550284] time_stamp <10c45c1ce> May 3 12:01:54 node2-1 kernel: [823888.550284] jiffies <10c45c569> May 3 12:01:54 node2-1 kernel: [823888.550287] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang May 3 12:01:54 node2-1 kernel: [823888.550287] Tx Queue <5> May 3 12:01:54 node2-1 kernel: [823888.550287] TDH, TDT <0>, <1> May 3 12:01:54 node2-1 kernel: [823888.550287] next_to_use <1> May 3 12:01:54 node2-1 kernel: [823888.550287] next_to_clean <0> May 3 12:01:54 node2-1 kernel: [823888.550287] tx_buffer_info[next_to_clean] May 3 12:01:54 node2-1 kernel: [823888.550287] time_stamp <10c45c1f1> May 3 12:01:54 node2-1 kernel: [823888.550287] jiffies <10c45c569> May 3 12:01:54 node2-1 kernel: [823888.550292] ixgbe 0000:02:00.0 eth0: tx hang 10 detected on queue 5, resetting adapter May 3 12:01:54 node2-1 kernel: [823888.550293] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout May 3 12:01:54 node2-1 kernel: [823888.550297] ixgbe 0000:02:00.0 eth0: Reset adapter May 3 12:01:54 node2-1 kernel: [823888.550411] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:54 node2-1 kernel: [823888.550521] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:54 node2-1 kernel: [823888.550630] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:54 node2-1 kernel: [823888.550739] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:54 node2-1 kernel: [823888.550848] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:54 node2-1 kernel: [823888.550955] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:54 node2-1 kernel: [823888.550966] ixgbe 0000:02:00.0 eth0: tx hang 11 detected on queue 2, resetting adapter May 3 12:01:54 node2-1 kernel: [823888.551116] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:54 node2-1 kernel: [823888.551272] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:54 node2-1 kernel: [823888.740382] ixgbe 0000:02:00.0: master disable timed out May 3 12:01:54 node2-1 kernel: [823888.760364] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 0 not cleared within the polling period May 3 12:01:54 node2-1 kernel: [823888.762290] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 1 not cleared within the polling period May 3 12:01:54 node2-1 kernel: [823888.764540] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 2 not cleared within the polling period May 3 12:01:54 node2-1 kernel: [823888.766357] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 3 not cleared within the polling period May 3 12:01:54 node2-1 kernel: [823888.768643] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 4 not cleared within the polling period May 3 12:01:54 node2-1 kernel: [823888.770342] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 5 not cleared within the polling period May 3 12:01:54 node2-1 kernel: [823888.772629] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 6 not cleared within the polling period May 3 12:01:54 node2-1 kernel: [823888.774351] ixgbe 0000:02:00.0 eth0: RXDCTL.ENABLE on Rx queue 7 not cleared within the polling period May 3 12:01:54 node2-1 kernel: [823888.942267] ixgbe 0000:02:00.0 eth0: detected SFP+: 3 May 3 12:01:56 node2-1 kernel: [823890.782118] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX