Re: [Xen-users] GPLPV drivers block when copying huge files

["Fábio M. Catunda" <fcatunda@xxxxxxxxxxxxxxxxx>]

Hi all,

I have the same problem here, I tried to find some help with google with 
no luck, finnally I received this message on the list with the solution! :-D
Well, what I could trace is that the problem occurs only when 
communicating between two DomUs and when using samba/CIFS. I made a test 
transfer with HTTP and if my memory don't fail, it worked.
I was facing this problem and I was needing a solution, the only that I 
could find was to put samba on Dom0 (ok, this is not good), and finally 
everything worked for me.
Some tcpdump showed me that I was having a lot of checksum errors when 
communicating from domU to domU, but I could not find a reason, but 
James explanation sounds great for me.
An additional information is that the problem also happen when 
transferring lot of files in a roll, not just with big files.
What is still missing for me is why the thing works just fine between 
dom0 and domU!?
Oh, one more thing, what is the best way to disable Large Send Offload? 
Is it possible to disable it just for a domU with some xen config 
parameter or do I have to do something like ethtook -k <bridge??or 
hwinterface> gso off, I'm a little confused here.
Thanks in advance.

Catunda.


James Harper escreveu:
> > Good news: Disabling "Large Send Offload" fixes the problem.
> >
> > How does enabled LSO in the GPLPV driver interact with the LSO implemented
> > by
> > the NIC driver in Linux? The mentioned problem could be reproduced on
> > three
> > machines with Broadcom NICs (BCM5708 and BCM715), but wasn't triggered on
> > a
> > system with an onboard Nvidia (MCP51).
> >
> >     
> It's all a bit of a mystery... but I think it goes something like this:
>
> LSO allows the operating system to send a large (>MTU) TCP packet to the 
> hardware. The hardware then breaks the packet up into header+MSS sized chunks, 
> recalculates the sequence numbers and checksums, and sends them. Efficiency gains 
> are to be had in doing this.
>
> In a DomU, if a packet is sent to the virtual adapter, where it flows onto the 
> Dom0 Bridge interface, and then onto a real hardware interface, then it is also 
> more efficient to keep the packet 'large' all the way to the physical nic where 
> the hardware then finally breaks it up and sends it.
>
> If, instead of going to a real hardware interface, the packet goes to another 
> DomU, then it is also more efficient to keep the packet 'large' all the way to 
> the DomU, and never bother breaking it up. Also, because the originating DomU 
> marked the flag as 'checksum correct', the destination DomU never bothers 
> checking the checksum, so you save doing the original checksum calculation, and 
> all the inbetween checksum validation.
>
> There's a lot that could go wrong there isn't there? Particularly if you start 
> doing NAT or something funny on the bridge!
>
> Windows support Large Send Offload in a mostly compatible way, at least under 
> Windows 2003 server. XP has a known problem with interaction with the firewall 
> service, but still seems to work okay once that is disabled. Unfortunately 
> Windows likes to break the packet up into more pieces than the Linux backend 
> can handle, so in the GPLPV drivers I allocate some memory pages and copy the 
> packet data into them. This reduces the number of scatter-gather segments to a 
> maximum of 16, which also reduces the amount of space each packet takes up on 
> the ring.
>
> What Windows doesn't support though is the Large Receive Offload, which the xen 
> network backend driver assumes is supported of LSO is advertised as supported. 
> So I have to fudge this - if Dom0 sends me a 'large' packet, I break it up 
> before giving it to Windows.
>
> Also, although Windows says that it supports RX checksum offload, where the 
> network adapter (or xennet in this case) calculates the checksum and reports on 
> it's correctness, it turns out that it doesn't really. If the checksum is never 
> calculated because the packet originated on a Linux DomU, and I just tell 
> Windows 'the checksum is correct, just trust me', it goes and checks anyway and 
> then drops the packet when it turns out to not be correct.
>
> The above two paragraphs are why the TX speed from the GPLPV drivers is much 
> faster (~2x) than RX.
>
> So there is plenty of room for me to have made errors there...
>
> Is anything useful reported if you run debugview from sysinternals?
>
> James
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users
>   
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users