Re: [Xen-devel] large packet support in netfront driver and guest network throughput

[Anirban Chakraborty <abchak@xxxxxxxxxxx>]

On Sep 18, 2013, at 8:48 AM, Wei Liu <wei.liu2@xxxxxxxxxx> wrote:

> On Tue, Sep 17, 2013 at 05:53:43PM +0000, Anirban Chakraborty wrote:
>> 
>> 
>> On 9/17/13 1:25 AM, "Wei Liu" <wei.liu2@xxxxxxxxxx> wrote:
>> 
>>> On Tue, Sep 17, 2013 at 10:09:21AM +0800, annie li wrote:
>>>> <snip>
>>>>>> I tried dom0 to dom0 and I got 9.4 Gbps, which is what I expected
>>>>>> (with GRO turned on in the physical interface). However, when I run
>>>>>> guest to guest, things fall off. Is large packet not supported in
>>>>>> netfront? I thought otherwise. I looked at the code and I do not see
>>>>>> any call to napi_gro_receive(), rather it is using
>>>>>> netif_receive_skb(). netback seems to be sending GSO packets to the
>>>>>> netfront, but it is being segmented to 1500 byte (as it appears from
>>>>>> the tcpdump).
>>>>>> 
>>>>> OK, I get your problem.
>>>>> 
>>>>> Indeed netfront doesn't make use of GRO API at the moment.
>>>> 
>>>> This is true.
>>>> But I am wondering why large packet is not segmented into mtu size
>>>> with upstream kernel? I did see large packets with upsteam kernel on
>>>> receive guest(test between 2 domus on same host).
>>>> 
>>> 
>>> I think Anirban's setup is different. The traffic is from a DomU on
>>> another host.
>>> 
>>> I will need to setup testing environment with 10G link to test this.
>>> 
>>> Anirban, can you share your setup, especially DomU kernel version, are
>>> you using upstream kernel in DomU?
>> 
>> Sure..
>> I have two hosts, say h1 and h2 running XenServer 6.1.
>> h1 running Centos 6.4, 64bit kernel, say guest1 and h2 running identical
>> guest, guest2.
>> 
> 
> Do you have exact version of your DomU' kernel? Is it available
> somewhere online?

Yes, it is 2.6.32-358.el6.x86_64. Sorry, I missed it out last time.

> 
>> iperf server is running on guest1 with iperf client connecting from guest2.
>> 
>> I haven't tried with upstream kernel yet. However, what I found out is
>> that the netback on the receiving host is transmitting GSO segments to the
>> guest (guest1), but the packets are segmented at the netfront interface.
>> 
> 
> I just tried, with vanilla upstream kernel I can see large packet size
> on DomU's side.
> 
> I also tried to convert netfront to use GRO API (hopefully I didn't get
> it wrong), I didn't see much improvement -- it's quite obvious because I
> already saw large packet even without GRO.
> 
> If you fancy trying GRO API, see attached patch. Note that you might
> need to do some contextual adjustment as this patch is for upstream
> kernel.
> 
> Wei.
> 
> ---8<---
> From ca532dd11d7b8f5f8ce9d2b8043dd974d9587cb0 Mon Sep 17 00:00:00 2001
> From: Wei Liu <wei.liu2@xxxxxxxxxx>
> Date: Wed, 18 Sep 2013 16:46:23 +0100
> Subject: [PATCH] xen-netfront: convert to GRO API
> 
> Signed-off-by: Wei Liu <wei.liu2@xxxxxxxxxx>
> ---
> drivers/net/xen-netfront.c |    4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
> index 36808bf..dd1011e 100644
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -952,7 +952,7 @@ static int handle_incoming_queue(struct net_device *dev,
>               u64_stats_update_end(&stats->syncp);
> 
>               /* Pass it up. */
> -             netif_receive_skb(skb);
> +             napi_gro_receive(&np->napi, skb);
>       }
> 
>       return packets_dropped;
> @@ -1051,6 +1051,8 @@ err:
>       if (work_done < budget) {
>               int more_to_do = 0;
> 
> +             napi_gro_flush(napi, false);
> +
>               local_irq_save(flags);
> 
>               RING_FINAL_CHECK_FOR_RESPONSES(&np->rx, more_to_do);
> -- 
> 1.7.10.4


I was able to see a bit of improvement (from 2.65 to 3.6 Gbps) with the 
following patch (your patch plus the advertisement of NETIF_F_GRO) :
---------
diff --git a/xen-netfront.c.orig b/xen-netfront.c
index 23e467d..bc673d3 100644
--- a/xen-netfront.c.orig
+++ b/xen-netfront.c
@@ -818,6 +818,7 @@ static int handle_incoming_queue(struct net_device *dev,
 {
        int packets_dropped = 0;
        struct sk_buff *skb;
+       struct netfront_info *np = netdev_priv(dev);
 
        while ((skb = __skb_dequeue(rxq)) != NULL) {
                struct page *page = NETFRONT_SKB_CB(skb)->page;
@@ -846,7 +847,7 @@ static int handle_incoming_queue(struct net_device *dev,
                dev->stats.rx_bytes += skb->len;
 
                /* Pass it up. */
-               netif_receive_skb(skb);
+               napi_gro_receive(&np->napi, skb);
        }
 
        return packets_dropped;
@@ -981,6 +982,7 @@ err:
        if (work_done < budget) {
                int more_to_do = 0;
 
+               napi_gro_flush(napi);
                local_irq_save(flags);
 
                RING_FINAL_CHECK_FOR_RESPONSES(&np->rx, more_to_do);
@@ -1182,7 +1184,8 @@ static struct net_device * __devinit 
xennet_create_dev(struct xenbus_device *dev
        netif_napi_add(netdev, &np->napi, xennet_poll, 64);
 
        /* Assume all features and let xennet_set_features fix up.  */
-       netdev->features        = NETIF_F_IP_CSUM | NETIF_F_SG | NETIF_F_TSO;
+       netdev->features        = NETIF_F_IP_CSUM | NETIF_F_SG | NETIF_F_TSO |
+                                       NETIF_F_GRO;
 
        SET_ETHTOOL_OPS(netdev, &xennet_ethtool_ops);
        SET_NETDEV_DEV(netdev, &dev->dev);
-----------
tcpdump showed that the guest interface received large packets. I haven't 
checked upstream kernel as guest though.

Anirban



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel