Re: [Xen-devel] new netfront and occasional receive path lockup

[Jeremy Fitzhardinge <jeremy@xxxxxxxx>]

 On 08/22/2010 09:43 AM, Christophe Saout wrote:
> Hi,
>
> I've been playing with some of the new pvops code, namely DomU guest
> code.  What I've been observing on one of the virtual machines is that
> the network (vif) is dying after about ten to sixty minutes of uptime.
> The unfortunate thing here is that I can only repoduce it on a
> production VM and have been unlucky so far to trigger the bug on a test
> machine.  While this has not been tragic - rebooting fixed the issue,
> unfortunately I can't spend very much time on debugging after the issue
> pops up.

Ah, OK.  I've seen this a couple of times as well.  And it just happened
to me then...


> Now, what is happening is that the receive path goes dead.  The DomU can
> send packets to Dom0 and those are visible using tcpdump on the Dom0 on
> the virtual interface, but not the other way around.

I hadn't got to that level of diagnosis, but I can confirm that that's
what seems to be happening here too.

> Now, I have done more than one change at a time (I'd like to avoid going
> into pinning it down since I can only reproduce it on a production
> machine, as I said, so suggestions are welcome), but my suspicion is
> that it might have to do with the new "smart polling" feature in
> xen/netfront.  Note that I have also updated Dom0 to pull in the latest
> dom0/backend and netback changes, just to make sure it's not due to an
> issue that has been fixed there, but I'm still seeing the same.

I agree.  I think I started seeing this once I merged smartpoll into
netfront.

    J

> The production machine is a machine that doesn't have much network load,
> but deals with a lot of small network requests (DNS and smtp mostly).  A
> workload which is hard to reproduce on the test machine.  Heavy network
> load (NFS, FTP and so on) for days hasn't triggered the problem.  Also,
> segmentation offloading and similar settings don't have any effect.
>
> The machine has 2 physical and the VM 2 virtual CPUs, DomU has PREEMPT
> enabled.
>
> I've been looking at the code, if there might be a race condition
> somewhere, something like where one could run into a situation where the
> hrtimer doesn't run and Dom0 believes the DomU should be polling and
> doesn't emit an interrupt or something, but I'm afraid I don't know
> enough to judge this (I mean, there are spinlocks which look safe to
> me).
>
> Do you have any suggestions what to try?  I can trigger the issue on the
> production VM again, but debugging should not take more than a few
> minutes if it happens.  Access is only possible via the console.
> Neither Dom0 nor the guest show anything unusual in the kernel message
> and continue to behave normally after the network goes dead (also able
> to shut down the guest normally).
>
> Thanks,
>       Christophe
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
>


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel