Re: mirage-www

[Anil Madhavapeddy <anil@xxxxxxxxxx>]

This corruption is *really* bad, but I can't reproduce it myself for
some reason.  We definitely need a little regression test that checks
the body of a packet (our iperf tests don't do this at the moment).

Corrupting the header on an outgoing packet implies that a Cstruct page
is being free'd too early, and being reused being before flushed from
the ring by the dom0 backend.  It definitely wouldn't be fixed by
Dave's most recent fix (which just calculates the checksum correctly
for an odd number of bytes).

-anil

On 11 Sep 2012, at 14:56, Balraj Singh <balraj.singh@xxxxxxxxxxxx> wrote:

> Sorry, I was offline for a few days.  This was the exact bug that I
> was working on before vacation/August/...
> 
>> From the outside the main symptom was that packets were getting lost.
> This was compounded because of a bug that got introduced when I added
> the last bits of window scaling - the bug prevented the sender from
> recognising dup-acks and so fast rexmit was never triggered.  That
> meant that all rexmits happened on timer which really slowed things
> down.  But after that was fixed there was still a problem because
> sometimes multiple packets in a window would get lost and since we
> don't yet do SACK the second+ loss would still only be sent on timer.
> 
> So the real problem was pkt loss, which there should have been none of
> in my test rig.  It turned out that the packets weren't being lost,
> they were just arriving with invalid tcp checksums.  Unfortunately
> this was not because the checksum was computed incorrectly, but
> because the payload was damaged after the checksum computation.  More
> unfortunately, my retransmission code saves the payload after the
> first transmission and if the pkt has to be retransmitted, it would
> recompute the checksum.  So if, as it happens now, the payload is
> corrupted after the checksum is computed, the packet would correctly
> be dropped by the receiver, but when the pkt is rexmited it would
> recompute the checksum of the corrupted payload, which would result in
> the packet being accepted and the whole stream getting corrupted!
> This is the worst kind of TCP bug - badness factor 100 to me.
> 
> What the tcp code should do is compute the checksum as soon as it
> receives the payload from the application and then only adjust the
> chksum for any changes it needs to make on first or subsequent
> transmissions.  That would prevent the stream corruption which is at
> the top of tcp's guarantees - it would just stop all transmission on
> the stream but not let it get corrupted.
> 
> So, I was/am trying to figure out what was causing the corruption.  It
> has an interesting signature - most of the time the IP+TCP header of
> that particular packet was being written over the first 40 bytes of
> the payload.  Dave, would the problem that you fixed in the xen tcp
> checksum code cause this kind of corruption?  If yes that would be a
> relief.  I'll try to get your fix and see if the corruption goes away
> (prob do this tomorrow).
> 
> Thx, sorry about the saga.
> 
> Balraj
> 
> 
> 
> On Tue, Sep 11, 2012 at 9:36 AM, David Scott <scott.dj@xxxxxxxxx> wrote:
>> On Tue, Sep 11, 2012 at 9:24 AM, Richard Mortier
>> <Richard.Mortier@xxxxxxxxxxxxxxxx> wrote:
>>> 
>>> On 11 Sep 2012, at 07:52, David Scott wrote:
>>> 
>>> Heh, no problem. While I was debugging this I did a bit of "yak shaving" and
>>> now have a nice bit of code which can capture packets to a block device--
>>> this may be useful in future.
>>> 
>>> 
>>> oh- nice. can you point me to it please? (I can sense some student project
>>> proposals coming on :)
>> 
>> This patch causes all incoming/outgoing ethernet frames to be
>> 'captured' by which I mean added to a non-blocking Lwt_stream:
>> 
>> https://github.com/djs55/mirage-net/commit/6bbb77f7af9f6d210b7af2f6b3b239f0272861d2
>> 
>> Initially the stream size is set to 0, so all packets are dropped i.e.
>> no recording is actually done.
>> 
>> This small repo extracts the pcap example from ocaml-cstruct and adds
>> some scaffolding around it:
>> 
>> https://github.com/djs55/ocaml-pcap
>> 
>> It's a bit minimal but could be easily expanded. This module:
>> 
>> https://github.com/djs55/ocaml-pcap/blob/master/mirage/pcap_mirage.ml
>> 
>> contains code to pull packets from the Lwt_stream and append them --
>> with pcap headers -- to the block device. The top function is
>> basically an experiment in making a 'file-like thing'.
>> 
>> I started the capturing in the mirage-www main loop:
>> 
>> https://github.com/djs55/mirage-www/blob/djs-blog/src/server.ml
>> 
>> HTH,
>> Dave
>> 
>>> 
>>> 
>>> On Tuesday, September 11, 2012, Anil Madhavapeddy wrote:
>>>> 
>>>> That was my fault, sorry! This sort of thing should be fixed soon when
>>>> we can share C bindings more easily among the cross-compilation targets.
>>>> 
>>>> -anil
>>>> 
>>>> On 10 Sep 2012, at 22:31, David Scott <scott.dj@xxxxxxxxx> wrote:
>>>> 
>>>>> Right, problem solved: TCP checksum on xen was slightly broken. There
>>>>> was a fix made to Unix to cope with odd-length packets, this needed to
>>>>> be applied to xen as well. Many of the packets had valid checksums,
>>>>> and linux would ACK up to the sequence number in the last one of
>>>>> those.
>>>>> 
>>>>> Now that the TCP bug is squashed, I can get back to writing my blog
>>>>> post -- all just a day in the life of a mirage hacker :-)
>>>>> 
>>>>> On Sat, Sep 8, 2012 at 9:06 AM, Dave Scott <Dave.Scott@xxxxxxxxxxxxx>
>>>>> wrote:
>>>>>> 
>>>>>> 
>>>>>> On Sep 7, 2012, at 5:59 PM, "Anil Madhavapeddy" <anil@xxxxxxxxxx>
>>>>>> wrote:
>>>>>> 
>>>>>>> On Fri, Sep 07, 2012 at 02:02:45PM +0100, David Scott wrote:
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> I've built mirage-www for xen and modified the static IP to fit my
>>>>>>>> environment. (I tried DHCP first but this didn't work -- if I get the
>>>>>>>> time I'll try to debug).
>>>>>>>> 
>>>>>>>> I can ping the server fine, and it's certainly receiving a lot of
>>>>>>>> traffic on my (probably fairly busy) local network. When I try to
>>>>>>>> fetch a URL the TCP connection hangs. On the console I get:
>>>>>>>> 
>>>>>>>> Dispatch: dynamic URL /
>>>>>>>> ... irrelevant spam
>>>>>>>> TCP retransmission on timer seq = -889321980
>>>>>>>> 
>>>>>>>> I've attached a small tcpdump of the conversation. I started with
>>>>>>>> ping
>>>>>>>> and then tried HTTP. According to tcpdump/wireshark it went like
>>>>>>>> this:
>>>>>>>> 
>>>>>>>> $ tcpdump -r mirage.pcap -n
>>>>>>>> reading from file mirage.pcap, link-type EN10MB (Ethernet)
>>>>>>>> 13:47:07.543119 IP 10.80.2.32 > 10.80.239.140: ICMP echo request, id
>>>>>>>> 9300, seq 1, length 64
>>>>>>>> 13:47:07.543756 IP 10.80.239.140 > 10.80.2.32: ICMP echo reply, id
>>>>>>>> 9300, seq 1, length 64
>>>>>>>> 13:47:08.542112 IP 10.80.2.32 > 10.80.239.140: ICMP echo request, id
>>>>>>>> 9300, seq 2, length 64
>>>>>>>> 13:47:08.542422 IP 10.80.239.140 > 10.80.2.32: ICMP echo reply, id
>>>>>>>> 9300, seq 2, length 64
>>>>>>>> 13:47:09.541288 IP 10.80.2.32 > 10.80.239.140: ICMP echo request, id
>>>>>>>> 9300, seq 3, length 64
>>>>>>>> 13:47:09.541609 IP 10.80.239.140 > 10.80.2.32: ICMP echo reply, id
>>>>>>>> 9300, seq 3, length 64
>>>>>>>> 13:47:10.541286 IP 10.80.2.32 > 10.80.239.140: ICMP echo request, id
>>>>>>>> 9300, seq 4, length 64
>>>>>>>> 13:47:10.541580 IP 10.80.239.140 > 10.80.2.32: ICMP echo reply, id
>>>>>>>> 9300, seq 4, length 64
>>>>>>>> 13:47:11.541286 IP 10.80.2.32 > 10.80.239.140: ICMP echo request, id
>>>>>>>> 9300, seq 5, length 64
>>>>>>>> 13:47:11.541932 IP 10.80.239.140 > 10.80.2.32: ICMP echo reply, id
>>>>>>>> 9300, seq 5, length 64
>>>>>>>> 
>>>>>>>> -- so far so good, this is just my initial pings. Switching to 'wget
>>>>>>>> http://10.80.239.140/'
>>>>>>>> 
>>>>>>>> 13:47:14.241216 IP 10.80.2.32.37158 > 10.80.239.140.80: Flags [S],
>>>>>>>> seq
>>>>>>>> 2284582709, win 5840, options [mss 1460,sackOK,TS val 909789846 ecr
>>>>>>>> 0,nop,wscale 6], length 0
>>>>>>>> 13:47:14.242365 IP 10.80.239.140.80 > 10.80.2.32.37158: Flags [S.],
>>>>>>>> seq 3536243828, ack 2284582710, win 65535, options [mss 1380,wscale
>>>>>>>> 2,eol], length 0
>>>>>>>> 13:47:14.242387 IP 10.80.2.32.37158 > 10.80.239.140.80: Flags [.],
>>>>>>>> ack
>>>>>>>> 1, win 92, length 0
>>>>>>>> 
>>>>>>>> -- TCP connection established
>>>>>>>> 
>>>>>>>> 13:
>>> 
>>> 
>>> 
>>> --
>>> Dave Scott
>>> 
>>> 
>>> 
>>> --
>>> Cheers,
>>> 
>>> R.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> This message and any attachment are intended solely for the addressee and
>>> may contain confidential information. If you have received this message in
>>> error, please send it back to me, and immediately delete it. Please do not
>>> use, copy or disclose the information contained in this message or in any
>>> attachment. Any views or opinions expressed by the author of this email do
>>> not necessarily reflect the views of the University of Nottingham.
>>> 
>>> This message has been checked for viruses but the contents of an attachment
>>> may still contain software viruses which could damage your computer system:
>>> you are advised to perform your own checks. Email communications with the
>>> University of Nottingham may be monitored as permitted by UK legislation.
>> 
>> 
>> 
>> --
>> Dave Scott
>