[Xen-users] RFH: Windows2003+GPLPV packet-receive breaks after some time (Xen 3.4.3 amd64)

[Philipp Hahn <hahn@xxxxxxxxxxxxx>]

Hello,

one of our domU Windows system with GPL-PV driver regularly has problems
with its network connection: After some time the VM does not receive any
packets anymore. It's seems to be only a problem with receiving, since sending
ARP packets still works:

        tcpdump -i vif147.0 -n arp | grep -FA1 --color XXX.X.71.77

If I try to ping the domU from the dom0, I only see the request going to the
domU, but no answer:
        13:49:17.106405 arp who-has XXX.X.71.77 tell XXX.X.12.47

If I try to ping some host from the domU, I see the request leaving the domU
and the answer arriving for the domU, but no following ICMP messaged:
        13:48:37.569618 arp who-has XXX.X.22.12 tell XXX.X.71.77
        13:48:37.570002 arp reply XXX.X.22.12 is-at 00:16:3e:aa:ed:fa

We have saved the state of the VM to a file, which when restored puts the domU
back in the broken state.

We collected some information, but now are stuck on how to best proceed, since
we don't know enough of Xens and GPLPVs internal working.
Can we (or someone els) diagnose, why received packages are not properly 
handled?
Should we install the debug driver and what should we do when the problem next 
occurs. (I'm not afraid of debuggers and assembler, but only on Linux and not 
much with Windows)

Arch: amd64
Xen: 3.4.3
dom0: 2.6.32-17 (Debian)
domU: Windows 2003 Service Pack 2
GPLPV: 0.11.0.238
        Xen network device settings:
                Check checksum on RX packets: Enabled
                Checksum Offload: Enabled
                Large Send Offload: 61440
                Locally Administrated Address: Not set
                MTU: 1500
                Rx Interrupt Moderation (beta): Disabled
                Scatter/Gather: Enabled

# xm network-list 147
Idx BE     MAC Addr.     handle state evt-ch tx-/rx-ring-ref BE-path
0   0  00:16:3e:af:fa:a5    0     4      9     
15732/15741   /local/domain/0/backend/vif/147/0

# xenstore-ls /local/domain/0/backend/vif/147
0 = ""
 bridge = "XXXXXX0"
 domain = "XXXX010"
 handle = "0"
 uuid = "c550619d-3a4f-edfd-a22c-4b11a84b5728"
 script = "/etc/xen/scripts/vif-bridge"
 state = "4"
 frontend = "/local/domain/147/device/vif/0"
 mac = "00:16:3e:af:fa:a5"
 online = "1"
 frontend-id = "147"
 feature-sg = "1"
 feature-gso-tcpv4 = "1"
 feature-rx-copy = "1"
 feature-rx-flip = "0"
 feature-smart-poll = "1"
 hotplug-status = "connected"

# netstat -s
Ip:
    2982289885 total packets received
    2708867 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    2931918645 incoming packets delivered
    1504949163 requests sent out
    1 outgoing packets dropped
    1 dropped because of missing route
    2683 reassemblies required
    1137 packets reassembled ok
    2630 fragments received ok
    5669 fragments created
Icmp:
    811639 ICMP messages received
    400 input ICMP message failed.
    ICMP-Eingabehistogramm:
        destination unreachable: 4632
        redirects: 619
        echo requests: 806046
        echo replies: 185
        timestamp request: 44
        address mask request: 67
    876674 ICMP messages sent
    0 ICMP messages failed
    ICMP-Ausgabehistogramm:
        destination unreachable: 70142
        echo request: 452
        echo replies: 806036
        timestamp replies: 44
IcmpMsg:
        InType0: 185
        InType3: 4632
        InType5: 619
        InType8: 806046
        InType10: 2
        InType13: 44
        InType17: 67
        InType37: 44
        OutType0: 806036
        OutType3: 70142
        OutType8: 452
        OutType14: 44
Tcp:
    537972 active connections openings
    141489 passive connection openings
    3940 failed connection attempts
    90207 connection resets received
    9 connections established
    2877624333 segments received
    1502888618 segments send out
    378040 segments retransmited
    0 bad segments received.
    703715 resets sent
Udp:
    132980 packets received
    69241 packets to unknown port received.
    0 packet receive errors
    804134 packets sent
UdpLite:
TcpExt:
    56 resets received for embryonic SYN_RECV sockets
    514936 TCP sockets finished time wait in fast timer
    35 time wait sockets recycled by time stamp
    8541829 delayed acks sent
    7168 delayed acks further delayed because of locked socket
    Quick ack mode was activated 52 times
    1019468 packets directly queued to recvmsg prequeue.
    2562618 bytes directly in process context from backlog
    623450772 bytes directly received in process context from prequeue
    1470140290 packet headers predicted
    431554 packets header predicted and directly queued to user
    6735803 acknowledgments not containing data payload received
    1939166757 predicted acknowledgments
    61400 times recovered from packet loss by selective acknowledgements
    Detected reordering 1 times using FACK
    1 congestion windows fully recovered without slow start   
    5 congestion windows partially recovered using Hoe heuristic
    691 congestion windows recovered without slow start after partial ack
    302881 TCP data loss events
    TCPLostRetransmit: 6025
    719 timeouts after SACK recovery
    2 timeouts in loss state
    347805 fast retransmits
    19845 forward retransmits
    5795 retransmits in slow start
    3332 other TCP timeouts
    94 SACK retransmits failed
    77 DSACKs sent for old packets
    42 DSACKs received
    65852 connections reset due to unexpected data
    87707 connections reset due to early user close
    3 connections aborted due to timeout
    TCP ran low on memory 1 times
    TCPDSACKIgnoredOld: 40
    TCPDSACKIgnoredNoUndo: 2
    TCPSpuriousRTOs: 1
    TCPSackShifted: 1746475
    TCPSackMerged: 379798
    TCPSackShiftFallback: 115430
IpExt:
    InMcastPkts: 880
    InBcastPkts: 53279804
    InOctets: -1233700460
    OutOctets: 753104249
    InMcastOctets: 26566
    InBcastOctets: 1189547847

Sincerely
Philipp Hahn
-- 
Philipp Hahn           Open Source Software Engineer      hahn@xxxxxxxxxxxxx
Univention GmbH        Linux for Your Business        fon: +49 421 22 232- 0
Mary-Somerville-Str.1  28359 Bremen                   fax: +49 421 22 232-99
                                                   http://www.univention.de/
** Besuchen Sie uns auf der CeBIT in Hannover **
** Auf dem Univention Stand D36 in Halle 2    **
** Vom 01. bis 05. März 2011                  **

Attachment: signature.asc
Description: This is a digitally signed message part.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users