Re: [Xen-devel] Network dies and kernel errors

[John McMonagle <johnm@xxxxxxxxxxx>]

On Friday, July 29, 2011 10:03:48 am Konrad Rzeszutek Wilk wrote:
> On Mon, Jul 25, 2011 at 02:18:21PM -0500, John McMonagle wrote:
> > Have a new amd 6100 based server.
> > http://www.supermicro.com/Aplus/system/2U/2022/AS-2022G-URF.cfm
> > Running debian squeeze with debian 2.6.32 xen kernel
> > Running xen 4.1.1 built from source from xen.org
> > 
> > I'm seeing 2 errors.
> > during boot get this:
> > 
> > [    0.004823] ------------[ cut here ]------------
> > [    0.004833] WARNING:
> > at
> > /build/buildd-linux-2.6_2.6.32-35-amd64-aZSlKL/linux-2.6-2.6.32/debian/b
> > uild/source_amd64_xen/arch/x86/xen/enlighten.c:726
> > init_hw_perf_events+0x32d/0x3cd()
> > [    0.004838] Hardware name: H8DGU
> > [    0.004841] Modules linked in:
> > [    0.004847] Pid: 0, comm: swapper Not tainted 2.6.32-5-xen-amd64 #1
> > [    0.004850] Call Trace:
> > [    0.004857]  [<ffffffff81510efc>] ? init_hw_perf_events+0x32d/0x3cd
> > [    0.004862]  [<ffffffff81510efc>] ? init_hw_perf_events+0x32d/0x3cd
> > [    0.004870]  [<ffffffff8104ef00>] ? warn_slowpath_common+0x77/0xa3
> > [    0.004875]  [<ffffffff81510efc>] ? init_hw_perf_events+0x32d/0x3cd
> > [    0.004881]  [<ffffffff813044dc>] ? identify_cpu+0x2f7/0x300
> > [    0.004888]  [<ffffffff8100eccf>] ? xen_restore_fl_direct_end+0x0/0x1
> > [    0.004895]  [<ffffffff810e81d5>] ? kmem_cache_alloc+0x8c/0xf0
> > [    0.004900]  [<ffffffff81510a16>] ? identify_boot_cpu+0x15/0x3e
> > [    0.004904]  [<ffffffff81510baa>] ? check_bugs+0x9/0x2e
> > [    0.004910]  [<ffffffff81509cce>] ? start_kernel+0x3cd/0x3e8
> > [    0.004915]  [<ffffffff8150bc93>] ? xen_start_kernel+0x586/0x58a
> 
> You can ignore that one. It just means that you can't do profiling which we
> haven't yet up-ported.
> 
> ..
> 
> > Then next one may not be xen but I only had the problem after running a
> > domu. After a while I get kernel error and networking stops.
> 
> And some other user with a bnx2 driver seems to see a similar problem. Let
> me CC them here.
> 
> > This is the error:
> > [ 1411.813376] ------------[ cut here ]------------
> > [ 1411.813398] WARNING:
> > at
> > /build/buildd-linux-2.6_2.6.32-35-amd64-aZSlKL/linux-2.6-2.6.32/debian/b
> > uild/source_amd64_xen/net/sched/s ch_generic.c:261
> > dev_watchdog+0xe2/0x194()
> 
> OK, this is one is more worrysome.
> 
> > [ 1411.813410] Hardware name: H8DGU
> > [ 1411.813417] NETDEV WATCHDOG: peth0 (igb): transmit queue 1 timed out
> > [ 1411.813424] Modules linked in: xt_physdev iptable_filter tun ip_tables
> > x_tables bridge stp sg sr_mod cdrom xfs exportfs ipmi_si i
> > pmi_devintf ipmi_watchdog ipmi_msghandler xen_evtchn blktap xenfs loop
> > snd_pcm snd_timer snd soundcore snd_page_alloc pcspkr psmouse joydev
> > evdev serio_raw i2c_piix
> > 4 edac_core k10temp edac_mce_amd i2c_core processor button acpi_processor
> > ext4 mbcache jbd2 crc16 usbhid hid dm_mod raid1 md_mod sd_mod crc_t10dif
> > ata_generic usb_s
> > torage pata_atiixp ahci ohci_hcd libata ehci_hcd usbcore nls_base
> > scsi_mod igb dca thermal thermal_sys [last unloaded: scsi_wait_scan]
> > 
> >  [ 1411.813656] Pid: 4, comm: ksoftirqd/0 Tainted: G        W
> > 
> > 2.6.32-5-xen-amd64 #1
> > [ 1411.813664] Call Trace:
> > [ 1411.813671]  <IRQ>  [<ffffffff81272e42>] ? dev_watchdog+0xe2/0x194
> > [ 1411.813697]  [<ffffffff81272e42>] ? dev_watchdog+0xe2/0x194
> > [ 1411.813711]  [<ffffffff8104ef00>] ? warn_slowpath_common+0x77/0xa3
> > [ 1411.813724]  [<ffffffff81272d60>] ? dev_watchdog+0x0/0x194
> > [ 1411.813736]  [<ffffffff8104ef88>] ? warn_slowpath_fmt+0x51/0x59
> > [ 1411.813751]  [<ffffffff8130d42a>] ? _spin_unlock_irqrestore+0xd/0xe
> > [ 1411.813762]  [<ffffffff8104b41e>] ? try_to_wake_up+0x289/0x29b
> > [ 1411.813778]  [<ffffffff81272d34>] ? netif_tx_lock+0x3d/0x69
> > [ 1411.813791]  [<ffffffff8125d7da>] ? netdev_drivername+0x3b/0x40
> > [ 1411.813803]  [<ffffffff81272e42>] ? dev_watchdog+0xe2/0x194
> > [ 1411.813816]  [<ffffffff8100ece2>] ? check_events+0x12/0x20
> > [ 1411.813827]  [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
> > [ 1411.813841]  [<ffffffff8105b5ef>] ? run_timer_softirq+0x1c9/0x268
> > [ 1411.813855]  [<ffffffff81054c9b>] ? __do_softirq+0xdd/0x1a6
> > [ 1411.813867]  [<ffffffff81012cac>] ? call_softirq+0x1c/0x30
> > [ 1411.813873]  <EOI>  [<ffffffff8101422b>] ? do_softirq+0x3f/0x7c
> > [ 1411.813893]  [<ffffffff810548c2>] ? ksoftirqd+0x5f/0xd3
> > [ 1411.813905]  [<ffffffff81054863>] ? ksoftirqd+0x0/0xd3
> > [ 1411.813915]  [<ffffffff81065c39>] ? kthread+0x79/0x81
> > [ 1411.813926]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
> > [ 1411.813937]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
> > [ 1411.813948]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
> > [ 1411.813958]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
> > [ 1411.813966] ---[ end trace a7919e7f17c0a727 ]---
> > [ 1412.052253] eth0: port 1(peth0) entering disabled state
> > [ 1635.796207] frontend_changed: backend/vbd/3/768: prepare for reconnect
> > [ 1647.137513] eth0: port 3(vif3.0) entering disabled state
> > [ 1647.157527] eth0: port 3(vif3.0) entering disabled state
> > 
> >  Kernel logging (proc) stopped.
> > 
> > In this case dom0 locked up. Some times just networking stops and some
> > times networking recovers.
> > 
> > Looks like it uses msi-x interrupts.
> > 
> > Concerning igb error I have tried the following  one at a time:
> > New igb driver from Intel site.
> > kernel parameter  pcie_aspm=off
> > ethtool -K eth0 tx off  on dom0
> > ethtool -K eth0 gro off  on dom0
> 
> OK.
> 
> > It has never died doing iperf from dom0 or domu  <> external.
> > Never died during network backup.
> > 
> > Usually takes a least a few hours and has never made it a day running a
> > domu. Wish I could get it to die faster :-)
> > Any ideas?
> > I'm pretty much down to trying different network cards
> 
> Did you try that? Did that make any difference?

Not tested I did install one.

I think I found a way to keep it running.
On the new igb driver I built from new intel source added module parameter 
IntMode=1.

This puts it in msi mode. It was in msi-x mode.
It's never died with that setting.
It's up now over a day.
No real experience with msi-x. I think it's the first time I have seen a 
driver use msi-x interrupts.
Maybe that gives you more ideas?


> 
> > Any ideas?
> 
> There is a Xen parameter called 'noirqbalance' . Try that. Also see if you
> can limit the CPUs in the dom0 using these two arguments on Xen
> hypervisor:
> 
Should I turn off the irqbalence daemon also?
Just in case you wonder it does with out it.

> dom0_vcpus=2 dom0_vcpus_pin=1
> 
> 
> It would be interesting to narrow down _when_ you trigger this failure. B/c
> we can pull Xen to see what the MSI's are 'xl debug-keys M' _before_ and
> _after_ your failure to see if something is amiss.
> 
> Mainly to figure out if the vectors are moving around the CPUs (or not)
> 
> (XEN)  MSI    29 vec=21 lowest  edge   assert  log lowest dest=00000001
> mask=0/0/-1
> 
> and also 'xl debug-keys i' to see if the domain has ACK-ed the interrupt:
> (XEN)    IRQ:  29 affinity:00000000,00000000,00000000,00000001 vec:21
> type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:275(----),
> 
> (the last '----' might have something else in in them - if so that is a
> sign that dom0 hasn't picked up the event/vector).

Much of my frustration is that I have not found a way to get it to fail other 
than waiting a long time :-(

Thanks

John



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel