Re: [Xen-devel] Question about TCP checksum offload in Xen

[Balraj Singh <balrajsingh@xxxxxxxx>]

Thanks very much.  This is fine and exactly what is observed (that the checksum is only calculated going to/from a real wire).  The engineering choice makes sense.  But this does mean that I need to know when the stack receives a pkt if the checksum should be verified for that packet.  I can't do a redundant verification because a bad chksum may be because the field was clobbered (not set) during earlier verification.  So, what I wanted to know was where to look for this flag - a snippet of code or a ptr to a file :).

Thanks,

Balraj

On Thu, Dec 5, 2013 at 12:37 PM, John Haxby <john.haxby@xxxxxxxxxx> wrote:

On 05/12/13 11:39, Ian Campbell wrote:
> On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote:
>> > On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote:
>>> > > Hi,
>>> > >
>>> > > I'm working on verifying TCP checksums on incoming packets in Mirage, but
>>> > > I've run into a bit of a problem.
>>> > >
>>> > > If TCP checksum offload is turned on on a virtual interface (this is the
>>> > > default), and if the TCP connection is local to the machine, it looks like
>>> > > Xen does not calculate the checksum at all.  This may be valid because Xen
>>> > > may be providing a stronger guarantee, but it means that incoming packets
>>> > > don't have a valid checksum in the header.  This then means that in Mirage
>>> > > we can't just have checksum verification turned on all the time.  This
>>> > > would have been the safe fall back option and detecting that checksum
>>> > > offload is on, and then not duplicating the verification in Mirage would
>>> > > have been an optimisation.  But it looks like this is not an option.  Now I
>>> > > need to know for every incoming packet whether checksum verification should
>>> > > be done or not.  It should ideally be for every packet since chksum offload
>>> > > can be turned off and on on the VIF and existing tcp connections should
>>> > > continue.  If not every packet, I need to get a notification or efficiently
>>> > > detect right away that the setting is changed on the VIF.
>> >
>> > This is a question that seems to keep coming up even for Linux and
>> > Windows, as the combination of local<->local VMs vs local<->off-host and
>> > the checksum offload is quite confusing.
>> >
>> > CCing xen-devel: is the appropriate behaviour for a guest VM that wants to
>> > use checksum offloading in all situations documented anywhere?
> I don't understand the question/concern. If you have enabled checksum
> offload then of course you don't recalculate the checksum, that's the
> whole point of offloading it.

I get this a lot.

There are a few different cases:

  * domain to domain traffic
  * domain to external traffic with egress from a NIC that does offload
  * domain to external through a non-offloading NIC

With xen checksum offloading, domain to domain traffic appears to be
received with a bad checksum.  This is OK, there is no point in
calculating a checksum if the packets are only going through memory.  If
your memory is going to randomly corrupt packets you have more bigger
problems to worry about.   However, this does upset at least Solaris: if
you're using a Solaris guest for NAT then the NAT module on Solaris gets
all upset if the checksum is wrong and drops the packets.  (This is
Solaris's NAT module being overly picky, it may need to recalculate or
at least invalidate the existing checksum, but it doesn't need to check
it as well.)

The second two cases are of interest from the domain perspective.  A
domain has no way of knowing how any given packet is going to leave the
host (or even if it is) so it can't know ahead of time whether to
calculate any checksums: the skb's are just marked with "checksum
needed" as usual and either the egress NIC will do the job or dom0 will
do it.

There is absolutely nothing wrong in any of this (Solaris
notwithstanding).   The difficulty is getting people to realise that
checksums are only calculated when a packet hits the cat-5.  It doesn't
need documenting, it just needs a little thought.   I got tired of
hammering the point home :)

jch