Re: AMD EPYC virtual network performances

[Jürgen Groß <jgross@xxxxxxxx>]

On 13.11.24 18:25, Elliott Mitchell wrote:
> On Tue, Jul 09, 2024 at 08:36:18AM +0000, Andrei Semenov wrote:
> > After some investigations we notices a huge performances drop (perfs divided
> > by
> > factor of 5) starting from 5.10.88 Linux kernel version on the AMD EPYC
> > platforms. The patch introduced in this kernel version that allows to
> > pinpoint
> > the buggy behavior is :
> >
> >   “xen/netfront: harden netfront against event channel storms”
> > d31b3379179d64724d3bbfa87bd4ada94e3237de
> >
> > The patch basically binds the network frontend to the `xen_lateeoi_chip`
> > irq_chip (insead of `xen_dynamic_chip`) which allows to its clients to
> > inform
> > the chip if spurious interrupts are detected and so the delay in interrupt
> > treatment is introduced by the chip.
> I worry I'm being knave here.
>
> For the heck of it, I took a glance at b27d47950e48.  If my understanding
> is correct, b27d47950e48 is making a very basic (and wrong) assumption
> about timing/latency.
>
> In particular any time either side receive an event, it will handle
> X # of incoming payloads and Y # of acknowledged outgoing payloads.  As
> such if X + Y > 1, then up to X + Y - 1 spurious events may be detected.
> The issue is there is no synchronization between the event channel and
> the work queues.
>
> In particular the network back end could legitimately generate:
>
> work0   signal0 work1   signal1 work2   signal2 work3   signal3
>
> Whereas the network front end may handle this as:
>
> event0  work0   work1   work2   work3   event1  event2  event3
>
> Where b27d47950e48 would interpret events 1-3 as spurious, even though
> they're perfectly legitimate.  The same phenomenon could occur in both
> directions and also with the Xen block devices.
No.

For one, as long as event0 isn't EOI'd, the other events would just be
merged into a single one.

Additionally, as long as work0 isn't acknowledged by incrementing the
consumer index, additional queued work items should NOT result in
additional events being sent. An event is only sent if a work item is
queued to a ring buffer with consumer == producer.

> Ultimately how is the network portion of XSA-391 any different from any
> other network DoS?  If an interrupt is generated for every single packet
> of a series of runt frames, there will be heavy processor use for little
> network traffic.
The problem is that a steady stream of events could keep the other side
in IRQ handling for arbitrary amount of times, leading to hangups.

> AMD systems may fair worse than Intel systems due to differing cache
> coherence behavior/latency.  Perhaps AMD's NUMA implementation adds
> some latency.  (huh, suddenly the RAID1 issue comes to mind)
>
>
> Hopefully I'm not making knave speculation here.  Might this be the
> simplest of issues, just it was missed due to being too obvious?
I don't agree with your analysis, see above.


Juergen

Attachment: OpenPGP_0xB0DE9DD628BF132F.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature