[Xen-users] Freenas domU network performance issue

["G.R." <firemeteor@xxxxxxxxxxxxxxxxxxxxx>]

Hi guys,

I'm running a freenas domU (FreeBSD 8.3 based, ZFS v28, 2 vcpus mapped to the same HT capable core) to serve storage for all purpose including other domUs running on the same host. I did some study to understand how well it works and the result is kind of confusing.

In summary, the network performance between domains on the same host is worse than expected. And NFS service to other domains on the same host is not reliable. The last issue make it impossible to serve VM disk image from the nas. Anybody has met the same issue?

Benchmark details below:

1. iperf bench against external host (through physical network).

The bench can fully utilize the Giga-bit network.

ext  => nas ~930Mbps

nas => ext  ~ 900Mbps

TSO / LRO config has no impact on the bench result.

2. iperf bench against dom0 (through virtual bridge, handled fully by driver stack)

The bench result highly depends on TSO / LRO config.

dom0 => nas (LRO=1) 60Mbps ~ 300Mbps (vary a lot time to time, xentop reports low cpu usage in both domain (varies corresponding to BPS, ~ 20%)).

dom0 => nas (LRO=0) 10900Mbps (stable, high cpu usage ~190%)

nas => dom0 (TSO=1) 470Mbps (stable result, low cpu usage ~40%)

nas => dom0 (TSO=0)  860Mbps (stable result, high cpu usage ~170%. some times can stable at ~2000Mbps, not sure how to reproduce)

Similar result can be observed when talking to another linux domU. However, linux domU can generally achieve better performance than dom0.

3. As a comparasion, here is a iperf test between linux domU and dom0.

 dom0 => lin domU ~10300Mbps

lin domU => dom0  ~13500Mbps

4. file transfer from external host to nas

PS: Here the bottleneck is the network rather than the disk.

NFS: 50MB/s

FTP: 102MB/s

I'm not quite sure about how much perf to expect from NFS. But it appears to be a bottleneck.

5. file transfer from dom0 to nas

copy to the NFS mount served by the nas domU can cause hang. Almost 100% reproduce.

Typically the hang is limited to the nas and dom0 processes that accesses the mount.

"umount -f" can resolve the hang in such case. But in some cases, all ssh sessions to dom0 hang too. Not sure if this means a dom0 network hang or a full freeze.

Tcpdump seems to show that some traffic is lost. But there is no package drop statistics shown in both end from the netstat / ifconfig command.

Thanks,

Timothy

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users