RE: [Xen-users] GPLPV drivers block when copying huge files

["James Harper" <james.harper@xxxxxxxxxxxxxxxx>]

> 
> Good news: Disabling "Large Send Offload" fixes the problem.
> 
> How does enabled LSO in the GPLPV driver interact with the LSO implemented
> by
> the NIC driver in Linux? The mentioned problem could be reproduced on
> three
> machines with Broadcom NICs (BCM5708 and BCM715), but wasn't triggered on
> a
> system with an onboard Nvidia (MCP51).
> 

It's all a bit of a mystery... but I think it goes something like this:

LSO allows the operating system to send a large (>MTU) TCP packet to the 
hardware. The hardware then breaks the packet up into header+MSS sized chunks, 
recalculates the sequence numbers and checksums, and sends them. Efficiency 
gains are to be had in doing this.

In a DomU, if a packet is sent to the virtual adapter, where it flows onto the 
Dom0 Bridge interface, and then onto a real hardware interface, then it is also 
more efficient to keep the packet 'large' all the way to the physical nic where 
the hardware then finally breaks it up and sends it.

If, instead of going to a real hardware interface, the packet goes to another 
DomU, then it is also more efficient to keep the packet 'large' all the way to 
the DomU, and never bother breaking it up. Also, because the originating DomU 
marked the flag as 'checksum correct', the destination DomU never bothers 
checking the checksum, so you save doing the original checksum calculation, and 
all the inbetween checksum validation.

There's a lot that could go wrong there isn't there? Particularly if you start 
doing NAT or something funny on the bridge!

Windows support Large Send Offload in a mostly compatible way, at least under 
Windows 2003 server. XP has a known problem with interaction with the firewall 
service, but still seems to work okay once that is disabled. Unfortunately 
Windows likes to break the packet up into more pieces than the Linux backend 
can handle, so in the GPLPV drivers I allocate some memory pages and copy the 
packet data into them. This reduces the number of scatter-gather segments to a 
maximum of 16, which also reduces the amount of space each packet takes up on 
the ring.

What Windows doesn't support though is the Large Receive Offload, which the xen 
network backend driver assumes is supported of LSO is advertised as supported. 
So I have to fudge this - if Dom0 sends me a 'large' packet, I break it up 
before giving it to Windows.

Also, although Windows says that it supports RX checksum offload, where the 
network adapter (or xennet in this case) calculates the checksum and reports on 
it's correctness, it turns out that it doesn't really. If the checksum is never 
calculated because the packet originated on a Linux DomU, and I just tell 
Windows 'the checksum is correct, just trust me', it goes and checks anyway and 
then drops the packet when it turns out to not be correct.

The above two paragraphs are why the TX speed from the GPLPV drivers is much 
faster (~2x) than RX.

So there is plenty of room for me to have made errors there...

Is anything useful reported if you run debugview from sysinternals?

James

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users