Xen in ARM environment networking overload issue

[Oleshii Wood <oleshiiwood@xxxxxxxxx>]

Hello guys,

The same issue migh be found by link
https://xen.markmail.org/message/3w4oqeu5z7ryfbsb?q=xen_add_phys_to_mach_entry&page=1

DOM0/DOMU kernels version 5.15.19

Two XENified guests. They are equal. They both have the same configuration. It is enough to have one DomU and Dom0.

Part of DomU configuration.
type = "pvh"
memory = 1024
vcpus = 1

It is not necessary to repeat full custom board configuration with all the bridges.
It is enough to connect DomU with some external machine or with Dom0 through the network.

You should have a perf3 compiled package on both ends.
on Dom0 or external host you issue command:
perf3 -s &

on DomU you issue the command:
perf3 -c [Dom0 or external host ip address] -k 400K -b 0

After enough short time you will see in your DomU kernel messages or in your xen console something like
[ 2385.999011] xen_add_phys_to_mach_entry: cannot add pfn=0x000000000003e0de -> mfn=0x00000000008031f2: pfn=0x000000000003e0de -> mfn=0x00000000008044a0 already exists
[ 2355.968172] xen_add_phys_to_mach_entry: cannot add pfn=0x000000000003bfca -> mfn=0x000000000080319c: pfn=0x000000000003bfca -> mfn=0x0000000000803276 already exists
[ 2323.002652] xen_add_phys_to_mach_entry: cannot add pfn=0x000000000003b80e -> mfn=0x000000000080447d: pfn=0x000000000003b80e -> mfn=0x00000000008032a7 already exists
[ 2302.036336] xen_add_phys_to_mach_entry: cannot add pfn=0x000000000003e0de -> mfn=0x0000000000803105: pfn=0x000000000003e0de -> mfn=0x00000000008044a0 already exists
[ 2273.758169] xen_add_phys_to_mach_entry: cannot add pfn=0x000000000003b80e -> mfn=0x00000000008033fc: pfn=0x000000000003b80e -> mfn=0x00000000008032a7 already exists
[ 2252.254857] xen_add_phys_to_mach_entry: cannot add pfn=0x000000000003b80e -> mfn=0x00000000008032f0: pfn=0x000000000003b80e -> mfn=0x00000000008032a7 already exists

You will have a lot of those messages.

Involved files
arch/arm/xen/p2m.c
drivers/net/xen-netback/netback.c
drivers/net/xen-netback/common.h

This problem arrived in the p2m.c file in the xen_add_phys_to_mach_entry function
This function adds a new mapping XEN page pfn to the DomU gfn.
It does via red/black tree
Xen netback adapter structure placed in the common.h file. It contains xenvif_queue structure.
There are some involved members from this structure.

struct xenvif_queue {
...
struct page *mmap_pages[MAX_PENDING_REQS];
pending_ring_idx_t pending_cons;
...
u16 pending_ring[MAX_PENDING_REQS];
...
struct page *pages_to_map[MAX_PENDING_REQS];
...
}

All the pages are stored in the xenvif_queue->mmap_pages

They are allocated by their indexes. Those ones are stored
sequentially in the xenvif_queue->pending_ring.
Pages allocation depends on xenvif_queue->pending_cons.
This value rounded up to MAX_PENDING_REQS value.
Pages allocated for the request cyclically.
When an intensive network traffic is in progress especially
when a packets flow density has been growing, sooner or later
we will run into the case when we do not have enough free pages.
MAX_PENDING_REQS in our case is 256
This case arrives in the netback.c xenvif_tx_action->gnttab_map_refs call.
Main work is in the xenvif_tx_build_gops.
xenvif_tx_action is issued in NAPI context.
So we could say it is something like the interrupt bottom half.
This message is produced when the issued pfn is presented in the red/black tree.
It is produced unconditionally. In  the above mentioned condition this output
degrades the performance drastically.
I may offer a patch which decreases the amount of messages.

Regards,

Oleg