[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Interesting observation with network event notification and batching




On 2013-7-1 16:54, Wei Liu wrote:
On Mon, Jul 01, 2013 at 03:48:38PM +0800, annie li wrote:
On 2013-6-29 0:15, Wei Liu wrote:
Hi all,

After collecting more stats and comparing copying / mapping cases, I now
have some more interesting finds, which might contradict what I said
before.

I tuned the runes I used for benchmark to make sure iperf and netperf
generate large packets (~64K). Here are the runes I use:

   iperf -c 10.80.237.127 -t 5 -l 131072 -w 128k (see note)
   netperf -H 10.80.237.127 -l10 -f m -- -s 131072 -S 131072

                           COPY                    MAP
iperf    Tput:             6.5Gb/s             14Gb/s (was 2.5Gb/s)
So with default iperf setting, copy is about 7.9G, and map is about
2.5G? How about the result of netperf without large packets?

First question, yes.

Second question, 5.8Gb/s. And I believe for the copying scheme without
large packet the throuput is more or less the same.

          PPI               2.90                  1.07
          SPI               37.75                 13.69
          PPN               2.90                  1.07
          SPN               37.75                 13.69
          tx_count           31808                174769
Seems interrupt count does not affect the performance at all with -l
131072 -w 128k.

Right.

          nr_napi_schedule   31805                174697
          total_packets      92354                187408
          total_reqs         1200793              2392614

netperf  Tput:            5.8Gb/s             10.5Gb/s
          PPI               2.13                   1.00
          SPI               36.70                  16.73
          PPN               2.13                   1.31
          SPN               36.70                  16.75
          tx_count           57635                205599
          nr_napi_schedule   57633                205311
          total_packets      122800               270254
          total_reqs         2115068              3439751

   PPI: packets processed per interrupt
   SPI: slots processed per interrupt
   PPN: packets processed per napi schedule
   SPN: slots processed per napi schedule
   tx_count: interrupt count
   total_reqs: total slots used during test

* Notification and batching

Is notification and batching really a problem? I'm not so sure now. My
first thought when I didn't measure PPI / PPN / SPI / SPN in copying
case was that "in that case netback *must* have better batching" which
turned out not very true -- copying mode makes netback slower, however
the batching gained is not hugh.

Ideally we still want to batch as much as possible. Possible way
includes playing with the 'weight' parameter in NAPI. But as the figures
show batching seems not to be very important for throughput, at least
for now. If the NAPI framework and netfront / netback are doing their
jobs as designed we might not need to worry about this now.

Andrew, do you have any thought on this? You found out that NAPI didn't
scale well with multi-threaded iperf in DomU, do you have any handle how
that can happen?

* Thoughts on zero-copy TX

With this hack we are able to achieve 10Gb/s single stream, which is
good. But, with classic XenoLinux kernel which has zero copy TX we
didn't able to achieve this.  I also developed another zero copy netback
prototype one year ago with Ian's out-of-tree skb frag destructor patch
series. That prototype couldn't achieve 10Gb/s either (IIRC the
performance was more or less the same as copying mode, about 6~7Gb/s).

My hack maps all necessary pages permantently, there is no unmap, we
skip lots of page table manipulation and TLB flushes. So my basic
conclusion is that page table manipulation and TLB flushes do incur
heavy performance penalty.

This hack can be upstreamed in no way. If we're to re-introduce
zero-copy TX, we would need to implement some sort of lazy flushing
mechanism. I haven't thought this through. Presumably this mechanism
would also benefit blk somehow? I'm not sure yet.

Could persistent mapping (with the to-be-developed reclaim / MRU list
mechanism) be useful here? So that we can unify blk and net drivers?

* Changes required to introduce zero-copy TX

1. SKB frag destructor series: to track life cycle of SKB frags. This is
not yet upstreamed.
Are you mentioning this one 
http://old-list-archives.xen.org/archives/html/xen-devel/2011-06/msg01711.html?

<http://old-list-archives.xen.org/archives/html/xen-devel/2011-06/msg01711.html>

Yes. But I believe there's been several versions posted. The link you
have is not the latest version.

2. Mechanism to negotiate max slots frontend can use: mapping requires
backend's MAX_SKB_FRAGS >= frontend's MAX_SKB_FRAGS.

3. Lazy flushing mechanism or persistent grants: ???
I did some test with persistent grants before, it did not show
better performance than grant copy. But I was using the default
params of netperf, and not tried large packet size. Your results
reminds me that maybe persistent grants would get similar results
with larger packet size too.

"No better performance" -- that's because both mechanisms are copying?
However I presume persistent grant can scale better? From an earlier
email last week, I read that copying is done by the guest so that this
mechanism scales much better than hypervisor copying in blk's case.

The original persistent patch does memcpy in both netback and netfront side. I am thinking maybe the performance can become better if removing the memcpy from netfront. Moreover, I also have a feeling that we got persistent grant performance based on default netperf params test, just like wei's hack which does not get better performance without large packets. So let me try some test with large packets though.

Thanks
Annie

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.