|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] netif.h clarifications
On Fri, May 20, 2016 at 10:09:43AM +0100, Paul Durrant wrote: > > -----Original Message----- > > From: Roger Pau Monné [mailto:roger.pau@xxxxxxxxxx] > > Sent: 19 May 2016 17:28 > > To: xen-devel@xxxxxxxxxxxxxxxxxxxx > > Cc: Wei Liu; David Vrabel; Paul Durrant > > Subject: netif.h clarifications > > > > Hello, > > > > While trying to solve a FreeBSD netfront bug [0] I came across a couple > > of netif.h dark spots that I think should be documented in the netif.h > > header. I'm willing to make those changes, but I want to make sure my > > understanding is right. > > > > Regarding checksum offloading, I had a hard time figuring out what the > > different flags actually mean: > > > > /* Packet data has been validated against protocol checksum. */ > > #define _NETRXF_data_validated (0) > > #define NETRXF_data_validated (1U<<_NETRXF_data_validated) > > > > /* Protocol checksum field is blank in the packet (hardware offload)? */ > > #define _NETRXF_csum_blank (1) > > #define NETRXF_csum_blank (1U<<_NETRXF_csum_blank) > > > > (Same applies to the TX flags, I'm not copying them there because they are > > the same) > > > > First of all, I assume "protocol" here refers to Layer 3 and Layer 4 > > protocol, so that would be IP and TCP/UDP/SCTP checksum offloading? In > > any > > case this needs clarification and proper wording. > > > > Then, I have some questions regarding the meaning of the flags themselves > > and the content of the checksum field in all the possible scenarios. > > > > On RX path: > > > > - NETRXF_data_validated only: data has been validated, but what's the state > > of the checksum field itself? If the data is validated again, would it > > match against the checksum? > > - NETRXF_csum_blank only: I don't think this makes much sense, data is in > > unknown state and checksum is not present, so there's no way to validate > > it. Packet should be dropped? > > Yes, in practice it's not used on its own. As you say, I don't think it makes > any sense. > > > - NETRXF_data_validated | NETRXF_csum_blank: this combination seems to > > be > > the one that makes more sense to me, data is valid, but checksum is not > > there. This matches what some real NICs already do, that is to provide > > the result of the checksum check _without_ actually providing the > > checksum itself on the RX path. > > > > In Linux netback this is set if the checksum info is partial, which I take to > mean that the packet has a valid pseudo-header checksum. I think packets > coming from NICs are more likely to be 'checksum unnecessary' which results > in NETRXF_data_validated only, which I take to mean that the checksum has > been verified but may have been trashed in the process. Hm, if the checksum might have been trashed, I think NETRXF_csum_blank should be there, in the absence of NETRXF_csum_blank I would like to assume that the checksum is there and hasn't been trashed. > > > On TX path: > > > > - NETTXF_data_validated only: I don't think this makes any sense, data is > > always valid from the senders point of view. > > - NETTXF_csum_blank only: checksum calculation offload, it should be > > performed by the other end. > > - NETTXF_data_validated | NETTXF_csum_blank: again, I don't think it makes > > much sense, data is always valid from the senders point of view, or else > > why bother sending it? > > > > In Linux netback, the code goes: > > if (txp->flags & XEN_NETTXF_csum_blank) > skb->ip_summed = CHECKSUM_PARTIAL; > else if (txp->flags & XEN_NETTXF_data_validated) > skb->ip_summed = CHECKSUM_UNNECESSARY; > > So, csum_blank with or without data_validated means that it's assumed that > the packet contains a valid pseudo-header checksum, but if csum_blank is not > set then data_validated means that the data is good but the checksum is in an > unknown state which is ok if the packet is then forwarded to another vif, and > I assume 'unnecessary' is ignored by NIC drivers on their TX side (I guess > they would only be interesting in 'partial'). OK, csum_blank means there's a pseudo-header, and data_validated means there's no checksum at all? Sorry, this is all very confusing. > > So it looks to me like we could get away with just two flags, one on the RX > > side that signals that the packet doesn't have a checksum but that the > > checksum validation has already been performed, and another one on the TX > > side to signal that the packet doesn't have a calculated checksum > > (typical checksum offload). > > > > On the TX side it would be useful to have flags which indicate: > > - Full checksum present > - Pseudo-header checksum present > - State of checksum is unknown > > On the RX side it would be useful to have > > - Data validated, checksum state unknown > - Data validated, checksum correct > - Data not validated +1 clearly. > > And then I've also seen some issues with TSO/LRO (GSO in Linux > > terminology) > > when using packet forwarding inside of a FreeBSD DomU. For example in the > > following scenario: > > > > + > > | > > +---------+ +--------------------+ +----------+ > > | |A B| router |C D| | > > | Guest 1 +-----------+ + +-----------+ Guest 2 | > > | | bridge0 | | | bridge1 | | > > +---------+ +--------------------+ +----------+ > > 172.16.1.67 172.16.1.66| 10.0.1.1 10.0.1.2 > > | > > +---------------------------------------------> > > ssh 10.0.1.2 | > > | > > | > > | > > + > > > > All those VMs are inside of the same host, and one of them acts as a gateway > > between them because they are on two different subnets. In this case I'm > > seeing issues because even though I disable TSO/LRO on the "router" at > > runtime, the backend doesn't watch the xenstore feature flag, and never > > disables it from the vif on the Dom0 bridge. This causes LRO packets > > (non-fragmented) to be received at point 'C', and then when the gateway > > tries to inject them into the other NIC it fails because the size is greater > > than the MTU, and the "no fragment" bit is set. > > > > Yes, GSO cannot be disabled/enabled dynamically on the netback tx side (i.e. > guest rx side) so you can't turn it off. The Windows PV driver leave sit on > all the time and does the fragmentation itself if the stack doesn't want GRO. > Doing the fragmentation in the frontend makes more sense anyway since the cpu > cycles are burned by the VM rather than dom0 and so it scales better. The weird thing is that GSO can usually be dinamically enabled/disabled on all network cards, so it would make sense to allow netfront to do the same. I guess the only way is to reset the netfront/netback connection when changing this property. > > How does Linux deal with this situation? Does it simply ignore the no > > fragment flag and fragments the packet? Does it simply inject the packet to > > the other end ignoring the MTU and propagating the GSO flag? > > > > I've not looked at the netfront rx code but I assume that the large packet > that is passed from netback is just marked as GSO and makes its way to > wherever it's going (being fragmented by the stack if it's forwarded to an > interface that doesn't have the TSO flag set). But it cannot be fragmented if it has the IP "don't fragment" flag set. What I'm seeing here is that at point C netback passes GSO packets to the "router" VM, this packets have not been fragmented, and then when the router VM tries to forward them to point B it has to issue a "need fragmentation" icmp message because the MTU of the interface is 1500 and the IP header has the "don't fragment" set (and of course the GSO chain is bigger than 1500). Is Linux ignoring the "don't fragment" IP flag here and simply fragmenting it? Because AFAICT I don't see any option in Linux netfront to prevent advertising "feature-gso-tcpv4", so netback will always inject GSO packets on the netfront RX path if possible. Roger. _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |