[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel][PATCH][RFC] Using data polling mechanism in netfront to replace event notification between netback and netfront


  • To: "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>
  • From: "Xu, Dongxiao" <dongxiao.xu@xxxxxxxxx>
  • Date: Thu, 10 Sep 2009 15:42:05 +0800
  • Accept-language: en-US
  • Acceptlanguage: en-US
  • Delivery-date: Thu, 10 Sep 2009 00:43:38 -0700
  • List-id: Xen developer discussion <xen-devel.lists.xensource.com>
  • Thread-index: Acox6jFOUMf61+LiTw60WvyWTJXRuA==
  • Thread-topic: [Xen-devel][PATCH][RFC] Using data polling mechanism in netfront to replace event notification between netback and netfront

Hi, 
        This is a VNIF optimization patch, need for your comments. Thanks!

[Background]:
        One of the VNIF driver's scalability issues is the high event channel 
frequency. It's highly related to physical NIC's interrupt frequency in dom0, 
which could be 20K HZ in some situation. The high frequency event channel 
notification makes the guest and dom0 CPU utilization at a high value. 
Especially for HVM PV driver, it brings high rate of interrupts, which could 
cost a lot of CPU cycle. 
        The attached patches have two parts: one part is for netback, and the 
other is for netfront. The netback part is based on the latest PV-Ops Dom0, and 
the netfront part is based on the 2.6.18 HVM unmodified driver.
        This patch uses a timer in netfront to poll the ring instead of event 
channel notification. If guest is transferring data, the timer will start 
working and periodicaly send/receive data from ring. If guest is idle and no 
data is transferring, the timer will stop working automatically. It will 
restart again once there is new data transferring. 
        We set a feature flag in xenstore to indicate whether the 
netfront/netback support this feature. If there is only one side supporting it, 
the communication mechanism will fall back to default, and the new feature will 
not be used. The feature is enabled only when both sides have the flag set in 
xenstore.
        One problem is the timer polling frequency. This netfront part patch is 
based on 2.6.18 HVM unmodified driver, and in that kernel version, guest 
hrtimer is not accuracy, so I use the classical timer. The polling frequency is 
1KHz. If rebase the netfront part patch to latest pv-ops, we could use hrtimer 
instead. 

[Testing Result]:
        We used a 4-core Intel Q9550 to do the test. From below we can see 
that, the test cases include the combination of 1/3/6/9 VMs (all the VMs are UP 
guest), 50/1472/1500 packets size, we mesured the throughput, Dom0 CPU 
utilization, and guest total CPU utiliztion. We pinned each guest vcpu with one 
pcpu, and pinned dom0's vcpu with one pcpu. Take 9 guest VMs case as an 
example, we pinned Dom0's vcpu with pcpu0, and pinned guest 1~3 vcpu with 
pcpu1, guest 4~6 vcpu with pcpu2, and guest 7~9 vcpu with pcpu3. We use netperf 
to do the test, and these results are based on HVM VNIF driver. The packets 
size means: 50 (small packets size), 1472 (big packets size, near MTU), 1500 
(packets mixed with big size and small size). From below chart we could see 
that, host/guest CPU utilization decreases after applying the patch, especially 
when multiple VM is launched, while the performance is not impacted. 

VM RECEIVE CASES:

Guest UDP Receive (Single Guest VM)
Packet Size (bytes)     Test Case       Throughput (Mbps)       Dom0 CPU Util   
Guest CPU Total Util 
50                      w/o patch       83.25                   100.00%         
26.10% 
50                      w/   patch      79.56                   100.00%         
23.80% 
1472                    w/o patch       950.30                  44.80%          
        22.40% 
1472                    w/   patch      949.32                  46.00%          
        17.90% 
1500                    w/o patch       915.84                  84.70%          
        42.40% 
1500                    w/   patch      908.94                  88.30%          
        28.70% 

Guest TCP Receive (Single Guest VM)
Packet Size (bytes)     Test Case       Throughput (Mbps)       Dom0 CPU Util   
Guest CPU Total Util 
50                      w/o patch       506.57                  43.30%          
        70.30% 
50                      w/   patch      521.52                  34.50%          
        57.70% 
1472                    w/o patch       926.19                  69.00%          
        32.90% 
1472                    w/   patch      928.23                  63.00%          
        24.40% 
1500                    w/o patch       935.12                  68.60%          
        33.70% 
1500                    w/   patch      926.11                  63.80%          
        24.80% 

Guest UDP Receive (Three Guest VMs)
Packet Size (bytes)     Test Case       Throughput (Mbps)       Dom0 CPU Util   
Guest CPU Total Util 
1472                    w/o patch       963.43                  50.70%          
        41.10% 
1472                    w/   patch      964.47                  51.00%          
        25.00% 
1500                    w/o patch       859.96                  99.50%          
        73.40% 
1500                    w/   patch      861.19                  97.40%          
        39.90% 

Guest TCP Receive (Three Guest VMs)
Packet Size (bytes)     Test Case       Throughput (Mbps)       Dom0 CPU Util   
Guest CPU Total Util 
1472                    w/o patch       939.68                  78.40%          
        64.00% 
1472                    w/   patch      926.04                  65.90%          
        31.80% 
1500                    w/o patch       933.00                  78.10%          
        63.30% 
1500                    w/   patch      927.14                  66.90%          
        31.90% 

Guest UDP Receive (Six Guest VMs)
Packet Size (bytes)     Test Case       Throughput (Mbps)       Dom0 CPU Util   
Guest CPU Total Util 
1472                    w/o patch       978.85                  56.90%          
        59.20% 
1472                    w/   patch      975.05                  53.80%          
        33.50% 
1500                    w/o patch       886.92                  100.00%         
87.20% 
1500                    w/   patch      902.02                  96.90%          
        46.00% 

Guest TCP Receive (Six Guest VMs)
Packet Size (bytes)     Test Case       Throughput (Mbps)       Dom0 CPU Util   
Guest CPU Total Util 
1472                    w/o patch       962.04                  90.30%          
        104.00% 
1472                    w/   patch      958.94                  69.40%          
        43.70% 
1500                    w/o patch       960.35                  90.10%          
        103.70% 
1500                    w/   patch      957.75                  68.70%          
        42.80% 

Guest UDP Receive (Nine Guest VMs)
Packet Size (bytes)     Test Case       Throughput (Mbps)       Dom0 CPU Util   
Guest CPU Total Util 
1472                    w/o patch       987.91                  60.50%          
        70.00% 
1472                    w/ FE patch     988.30                  56.60%          
        42.70% 
1500                    w/o patch       953.48                  100.00%         
93.80% 
1500                    w/ FE patch     904.17                  98.60%          
        53.50% 

Guest TCP Receive (Nine Guest VMs)
Packet Size (bytes)     Test Case       Throughput (Mbps)       Dom0 CPU Util   
Guest CPU Total Util 
1472                    w/o patch       974.89                  90.00%          
        110.60% 
1472                    w/   patch      980.03                  73.70%          
        55.40% 
1500                    w/o patch       971.34                  89.80%          
        109.60% 
1500                    w/   patch      973.63                  73.90%          
        54.70% 


VM SEND CASES:

Guest UDP Send (Single Guest VM)
Packet Size (bytes)     Test Case       Throughput (Mbps)       Dom0 CPU Util   
Guest CPU Total Util 
1472                    w/o patch       949.84                  56.50%          
        21.70% 
1472                    w/   patch      946.25                  51.20%          
        20.10% 
1500                    w/o patch       912.46                  87.00%          
        26.60% 
1500                    w/   patch      899.29                  86.70%          
        26.20% 

Guest TCP Send (Single Guest VM)
Packet Size (bytes)     Test Case       Throughput (Mbps)       Dom0 CPU Util   
Guest CPU Total Util 
1472                    w/o patch       932.16                  71.50%          
        35.60% 
1472                    w/   patch      932.09                  66.90%          
        29.50% 
1500                    w/o patch       929.91                  72.60%          
        35.90% 
1500                    w/   patch      931.63                  66.70%          
        29.50% 

Guest UDP Send (Three Guest VMs)
Packet Size (bytes)     Test Case       Throughput (Mbps)       Dom0 CPU Util   
Guest CPU Total Util 
1472                    w/o patch       972.66                  57.60%          
        24.00% 
1472                    w/   patch      970.07                  56.30%          
        23.30% 
1500                    w/o patch       943.87                  93.50%          
        32.50% 
1500                    w/   patch      933.61                  93.90%          
        30.00% 

Guest TCP Send (Three Guest VMs)
Packet Size (bytes)     Test Case       Throughput (Mbps)       Dom0 CPU Util   
Guest CPU Total Util 
1472                    w/o patch       955.92                  70.40%          
        36.10% 
1472                    w/   patch      946.39                  72.90%          
        32.90% 
1500                    w/o patch       966.06                  73.00%          
        38.00% 
1500                    w/   patch      947.23                  72.50%          
        33.60% 

Best Regards, 
-- Dongxiao

Attachment: netback.patch
Description: netback.patch

Attachment: netfront.patch
Description: netfront.patch

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.