Re: XSA-332 kernel patch - huge network performance on pfSense VMs

[Samuel Verschelde <samuel.verschelde@xxxxxxxx>]

Le 18/01/2021 à 11:03, Roger Pau Monné a écrit :
> On Fri, Jan 15, 2021 at 03:03:26PM +0000, Samuel Verschelde wrote:  >> Hi list, >> >> 
> Another "popular" thread on XCP-ng forum [1],
started in october >> 2020, allowed us to detect that patch 12 from the
XSA-332 advisory >> [2] had a very significant impact on network
performance in the >> case of pfSense VMs. >> >> We reproduced the issue
internally (well, we reproduced >> "something". The user setups in this
thread are diverse) and our >> findings seem to confirm what the users
reported. Running iperf3 >> from the pfSense VM to a debian VM gives
results around 5 times >> slower than before. Reverting this single
patch brings the >> performance back. On the debian to pfSense
direction, the drop is >> about 25%. > > pfSense is based on FreeBSD, so
I would bet that whatever performance > degradation you are seeing would
also happen with plain FreeBSD. I > would assume netfront in FreeBSD is
triggering the ratelimit on > Linux, and hence it gets throttled. > > Do
you think you have the bandwidth to look into the FreeBSD side and > try
to provide a fix? I'm happy to review and commit in upstream > FreeBSD,
but would be nice to have someone else also in the loop as > ATM I'm the
only one doing FreeBSD/Xen development AFAIK. >
I would personnally not be able to hack into either Xen, the linux
kernel or FreeBSD in any efficient way. My role here is limited to
packaging, testing and acting as a relay between users and developers.
We currently don't have anyone at Vates who would be able to hack into
FreeBSD either.

What currently puts FreeBSD into our radar is the large amount of users
who use FreeNAS/TrueNAS or pfSense VMs, and the recent bugs they
detected (XSA-360 and this performance drop).

Additionnally, regarding this performance issue, some users report an
impact of that same patch 12 on the network performance of their non-BSD
VMs [1][2], so I think the FreeBSD case might be helpful to help
identify what in that patch caused throttling (if that's what happens),
because it's easier to reproduce, but I'm not sure fixes would only need
to be made in FreeBSD.

Best regards,

Samuel Verschelde

[1] https://xcp-ng.org/forum/post/35521 mentions debian based Untangle
OS and inter-VLAN traffic
[2] https://xcp-ng.org/forum/post/35476 general slowdown affecting all
VMs (VM to workstation traffic), from the first user who identified
patch 12 as the cause.