RE: [Xen-users] domU network interface half-dies regularly

[James Dingwall <james.dingwall@xxxxxxxxxx>]

> I'm trying to figure out how to debug this. Any suggestions would be
> appreciated.
> 
> Every once in a while a random domU on a random xen server of ours has
> its
> network interface die. I've recently figured out what the exact
> symptoms are:
> TX count on that interface (as seen from inside the domU) stops
> increasing.
> There's no way of actually sending anything from within the domU. Even
> arp
> packets aren't sent. Everything works fine with receiving packets
> however.
> 
> Of the things I did check:
> - Doing an ip set link down/up on both dom0/domU doesn't do anything.
> - Removing/reattaching the dom0 interface from/to its bridge doesn't
> help.
> - It's interface-specific. I'm currently logged onto a domU that has
> one of
> its net interfaces half-dead as described, but the other perfectly
> functional.
> - Interestingly, the problem prevents "xm save" from working. It
> timeouts
> without anything getting written to disk (except a kilobyte or so of,
> I'm
> guessing, some headers).
> - I'm seeing this problem across:
>   - 2.6.18 xen.org dom0 3.3.X and 3.4.X
>   - xen.org hypervisor 3.3.X and 3.4.X
>   - domU xen.org 2.6.18.8_xen3.3.0U
>   - kernel.org 2.6.29.6 (pvops)
>   - A few different machines from different vendors.
> - Nothing in dom0/domU kernel logs.
> 
> Whatever the cause is, I seriously doubt it's domU's fault, considering
> I'm
> seeing the problem on both xen.org and kernel.org domU kernels. I also
> don't
> know what the trigger is (plus, those are production systems), so
> enabling a
> bunch of DEBUG prints in xen isn't much of an option.
> 
> Any suggestions/hints on where to look next? I'm guessing there are
> ways of
> inspecting various network code structures.
> 

I was experiencing some network issues in the past but typically they affected 
all domU at once and not individually.  I managed to solve my problems with 
these config options in the kernel:

# CONFIG_XEN_COMPAT_030002_AND_LATER is not set
# CONFIG_XEN_COMPAT_030004_AND_LATER is not set
# CONFIG_XEN_COMPAT_030100_AND_LATER is not set
# CONFIG_XEN_COMPAT_030200_AND_LATER is not set
# CONFIG_XEN_COMPAT_030300_AND_LATER is not set
CONFIG_XEN_COMPAT_LATEST_ONLY=y

(Was CONFIG_XEN_COMPAT_030002_AND_LATER=y)

CONFIG_XEN_NETDEV_TX_SHIFT=10

(Was CONFIG_XEN_NETDEV_TX_SHIFT=8)

I think it is actually the second that really made the difference as it looks 
like it controls the size of some buffer and it seems that heavy net traffic 
caused it to fill and break things.

James
This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,
you may review at http://www.amdocs.com/email_disclaimer.asp


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users