[Xen-devel] Network performance issues on Qubes OS Server prototype

[Frédéric Pierret <frederic.pierret@xxxxxxxxxxxx>]

Hi all,

I'm currently working on a Qubes OS server version (example architecture can 
been seen at 
https://raw.githubusercontent.com/fepitre/qubes-mgmt-salt-qubes-server/devel-140320-extra/qubes-server.png).
 I'm using this configuration since several months on Qubes R4.0 (xen-4.8) and 
recently on Qubes R4.1 (xen-4.13). I'm writing to you because since the 
beginning I'm having network performance issues that I never succeeded to solve.

This setup is done on a HP Gen8 DL360p with 2*CPUs, 160GB memory, 1TB RAID6 SAS.

On the picture I linked you, all the colored rectangles {zone}-* for zone in 
(wan, dmz, lan, admin) are PVH VMs (Debian 10). There exist a VM not drawn in 
the picture, called 'sys-net-interfaces' which holds four 1Gbits Ethernet 
controllers of the server using PCI passthrough. It is a HVM with Linux-based 
stubdomain.

All the inner links between VMs are NAT interfaces. All the outer links on 
*-sys-net VMs are BRIDGE interfaces with backend 'sys-net-interfaces'. In VM 
'sys-net-interfaces' a LACP bond0 is done with two Ethernet controllers, which 
is a trunk, then several vlan interfaces are generated with parent device this 
bond, and finally, bridges are created and associated to those vlans.

Here are my issues. Consider one computer named 'PC-LAN' in LAN network and 
another 'PC-DMZ' in DMZ network. The considered network path is the following:

        PC-LAN (1) <-- B --> lan-sys-net (2) <-- N --> lan-sys-firewall (3) <-- 
N --> dmz-sys-firewall (4) <-- N --> dmz-sys-net (5) <-- B --> PC-DMZ (6)

where B denotes bridge interface, N denotes NAT interface and numbers for 
numbering machines. Up to 'wget', 'scp' (limited normally by ciphers), etc., I 
ran multiple iperf3 tests over 20 seconds for having a clearer view of network 
issues.

Example 1: Full path

        From (1) to (6): 165 Mbits/s
        From (2) to (6): 196 Mbits/s
        From (3) to (6): 205 Mbits/s
        From (4) to (6): 203 Mbits/s
        From (5) to (6): 714 Mbits/s


Example 2: 'dmz-sys-net' as end node

        From (1) to (5): 194 Mbits/s
        From (2) to (5): 189 Mbits/s
        From (3) to (5): 258 Mbits/s
        From (4) to (5): 500 Mbits/s

Example 3: 'lan-sys-net' as end node

        From (1) to (2): 830 Mbits/s


I've another HP Gen8 with almost the same physical configuration and network 
configuration (LACP+vlan+bridges) running under Debian 10 as bare metal KVM, 
and I obtain 1Gbits/s network workflows over bridges. The almost physical 
configuration is due to the related mail I sent you in july 2019 '[Xen-devel] 
Ethernet PCI passthrough problem'. The provided Ethernet card with 4 ports (HP 
Ethernet 1Gb 4-port 331FLR Adapter) makes the driver tg3 crashing when 
attaching those into a VM. So the Debian KVM has those HP Ethernet controllers 
whereas on the Qubes server, it has a cheap PCI express 4 Ethernet Realtek 8169 
card.

Of course physical connections on the switches have been changed, 'switched' 
between the two servers for eliminating any hardware problem.

I had a look to 
https://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide. 
Unfortunately, trying some change of options with 'ethtool' in 
'sys-net-interfaces', changing amount of RAM/VCPUs of it and other *-sys-net, 
does not do that much.

I'm writing to you for having some clues into where I can dig and what I can 
look in order to put in evidence the bottleneck. If it's purely dom0 side or 
backend network VM side (sys-net-interfaces) or elsewhere.

I would like to thank you a lot in advance for any help on this problem.

Best regards,
Frédéric

Attachment: signature.asc
Description: OpenPGP digital signature