Issue with PCI passthrough ("swiotlb buffer is full") after Debian Dom0 kernel update to 5.10.178-3

[Paul Leiber <paul@xxxxxxxxxxxxxxxx>]

Hi list,

Since a Debian update of package linux-image-amd64 to version 5.10.0-22 
(kernel 5.10.178-3), I am experiencing an issue with PCI passthrough. 
The Wifi card is still visible in DomU, but it is not working anymore 
with the previous settings. I expect the card to continue working 
without any changes. Does anybody experience similar behavior? Any 
hints, pointers? Should I report this as a debian or a xen bug or 
something else?
I am running Xen with a Debian Bullseye Dom0 (standard packages) on an 
APU 4D4 system (https://pcengines.ch/apu4d4.htm). CPU is AMD Embedded G 
series GX-412TC, IOMMU is enabled in BIOS. A mPCIe WiFi card Compex 
WLE900VX is passed through to an OpenWRT DomU in PV mode. So far, the 
system has been running quite well (aside from some hickups in the early 
setup process, with incidentally a very similar behavior which I solved 
by increasing Dom0 memory, see my previous mail to the xen users list).
In DomU dmesg, there are several identical "swiotlb buffer is full" 
entries while failing to load the driver during the boot process:
[    7.979357] ath10k_pci 0000:00:00.0: Xen PCI mapped GSI40 to IRQ17
[    7.982856] ath10k_pci 0000:00:00.0: pci irq msi oper_irq_mode 2 
irq_mode 0 reset_mode 0
[    8.258462] ath10k_pci 0000:00:00.0: qca988x hw2.0 target 0x4100016c 
chip_id 0x043202ff sub 0000:0000
[    8.259117] ath10k_pci 0000:00:00.0: kconfig debug 0 debugfs 1 
tracing 0 dfs 1 testmode 0
[    8.261937] ath10k_pci 0000:00:00.0: firmware ver 10.2.4.70.70 api 5 
features no-p2p,raw-mode,mfp,allows-mesh-bcast crc32 3a226daa
[    8.349462] ath10k_pci 0000:00:00.0: board_file api 1 bmi_id N/A 
crc32 bebc7c08
[    9.533804] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112 
bytes), total 2048 (slots), used 2036 (slots)
[    9.534677] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112 
bytes), total 2048 (slots), used 2036 (slots)
[    9.535954] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112 
bytes), total 2048 (slots), used 2036 (slots)
[    9.537023] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112 
bytes), total 2048 (slots), used 2036 (slots)
[    9.546441] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112 
bytes), total 2048 (slots), used 2036 (slots)
[    9.556979] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112 
bytes), total 2048 (slots), used 2036 (slots)
[    9.557754] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112 
bytes), total 2048 (slots), used 2036 (slots)
[    9.558593] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112 
bytes), total 2048 (slots), used 2038 (slots)
[    9.559396] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112 
bytes), total 2048 (slots), used 2038 (slots)
[    9.560553] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112 
bytes), total 2048 (slots), used 2040 (slots)
[    9.563413] ath10k_pci 0000:00:00.0: failed to refill htt rx ring: -12
[    9.677728] ath10k_pci 0000:00:00.0: could not init core (-12)
[    9.678924] ath10k_pci 0000:00:00.0: could not probe fw (-12)
[    9.686262] kmodloader: done loading kernel modules from /etc/modul

What I tried:
-------------
Booting DomU with kernel option swiotlb=8192 leads to a crash/reboot of 
the complete system when the Openwrt boot process reaches loading the 
kernel module for the WiFi card (dmesg from DomU):
[    7.780624] ath10k_pci 0000:00:00.0: Xen PCI mapped GSI40 to IRQ17
[    7.785727] ath10k_pci 0000:00:00.0: pci irq msi oper_irq_mode 2 
irq_mode 0 reset_mode 0
[    7.903071] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[    8.023071] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[    8.142934] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[   11.798704] ath10k_pci 0000:00:00.0: failed to read device register, 
device is gone
[   11.799297] ath10k_pci 0000:00:00.0: failed to wait for target after 
cold reset: -5
[   11.800178] ath10k_pci 0000:00:00.0: failed to reset chip: -5


Booting DomU with kernel option swiotlb=16394 leads to error messages 
and a not working WiFi card (dmesg from DomU):
[    8.908062] ath10k_pci 0000:00:00.0: Xen PCI mapped GSI40 to IRQ17
[    8.912608] ath10k_pci 0000:00:00.0: pci irq msi oper_irq_mode 2 
irq_mode 0 reset_mode 0
[    9.025185] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[    9.145555] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[    9.265060] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[    9.465046] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[    9.585009] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[    9.705125] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[    9.794764] ath10k_pci 0000:00:00.0: Failed to get pcie state addr: -16
[    9.795310] ath10k_pci 0000:00:00.0: failed to setup init config: -16
[    9.795982] ath10k_pci 0000:00:00.0: could not power on hif bus (-16)
[    9.796472] ath10k_pci 0000:00:00.0: could not probe fw (-16)


Booting Dom0 with the previous kernel (package linux-image-amd64 
5.10.0-21) solves this issue. Booting with kernel option iommu=soft 
solves this issue (dmesg from DomU):
[    9.251808] ath10k_pci 0000:00:00.0: Xen PCI mapped GSI40 to IRQ17
[    9.255480] ath10k_pci 0000:00:00.0: pci irq msi oper_irq_mode 2 
irq_mode 0 reset_mode 0
[    9.377406] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[    9.496997] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[    9.617334] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[    9.877658] ath10k_pci 0000:00:00.0: qca988x hw2.0 target 0x4100016c 
chip_id 0x043202ff sub 0000:0000
[    9.878453] ath10k_pci 0000:00:00.0: kconfig debug 0 debugfs 1 
tracing 0 dfs 1 testmode 0
[    9.881475] ath10k_pci 0000:00:00.0: firmware ver 10.2.4.70.70 api 5 
features no-p2p,raw-mode,mfp,allows-mesh-bcast crc32 3a226daa
[    9.963705] ath10k_pci 0000:00:00.0: board_file api 1 bmi_id N/A 
crc32 bebc7c08
[   11.112605] ath10k_pci 0000:00:00.0: htt-ver 2.1 wmi-op 5 htt-op 2 
cal otp max-sta 128 raw 0 hwcrypto 1
dmesg in the Dom0 shows some corresponding entries in the working case:

[  303.813849] pciback 0000:05:00.0: xen_pciback: vpci: assign to 
virtual slot 0
[  303.819487] pciback 0000:05:00.0: registering for 5
[  304.887437] xen-blkback: backend/vbd/5/51712: using 1 queues, 
protocol 1 (x86_64-abi)
[  324.941391] xen-blkback: backend/vbd/5/51712: prepare for reconnect
[  326.400065] xen: registering gsi 40 triggering 0 polarity 1
[  326.400129] Already setup the GSI :40
[  326.403607] xen: registering gsi 40 triggering 0 polarity 1
[  326.403630] Already setup the GSI :40
[  326.406778] xen: registering gsi 40 triggering 0 polarity 1
[  326.406826] Already setup the GSI :40
[  326.409009] xen: registering gsi 40 triggering 0 polarity 1
[  326.409025] Already setup the GSI :40
[  326.411967] xen: registering gsi 40 triggering 0 polarity 1
[  326.411988] Already setup the GSI :40
[  326.414061] xen: registering gsi 40 triggering 0 polarity 1
[  326.414075] Already setup the GSI :40
[  326.415389] xen: registering gsi 40 triggering 0 polarity 1
[  326.415399] Already setup the GSI :40
[  326.497043] xen-blkback: backend/vbd/5/51712: using 1 queues, 
protocol 1 (x86_64-abi) persistent grants
Side information:
-----------------
There are some recent entries in the Debian package linux-image-amd64 
changelog which might be relevant, but I might also be completely wrong:
- [amd64] iommu/amd: Fix pci device refcount leak in ppr_notifier()
- [amd64] iommu/amd: Fix ivrs_acpihid cmdline parsing code
- [amd64] iommu/amd: Add PCI segment support for 
ivrs_[ioapic/hpet/acpihid] commands
- [amd64] iommu/amd: Fix ill-formed ivrs_ioapic, ivrs_hpet and 
ivrs_acpihid options
- [amd64] iommu/amd: Add a length limitation for the ivrs_acpihid 
command-line parameter
System information:
-------------------

xl dmesg:

(XEN) Xen version 4.14.5 (Debian 4.14.5+94-ge49571868d-1) 
(pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx) (x86_64-linux-gnu-gcc (Debian 
10.2.1-6) 10.2.1 20210110) debug=n  Thu Mar 23 19:40:49 UTC 2023
(XEN) build-id: c5e2deedc07540c4e17aa09e2a67b5aaf93d1806
(XEN) Bootloader: GRUB 2.06-3~deb11u5
(XEN) Command line: placeholder dom0_mem=768M,max:768M dom0_max_vcpus=2 
loglvl=all guest_loglvl=all iommu=verbose com1=115200,8n1,pci console=com1
(XEN) Xen image load base address: 0xcf600000
(XEN) Video information:
(XEN)  No VGA detected
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) CPU Vendor: AMD, Family 22 (0x16), Model 48 (0x30), Stepping 1 
(raw 00730f01)
(XEN) Xen-e820 RAM map:
(XEN)  [0000000000000000, 000000000009fbff] (usable)
(XEN)  [000000000009fc00, 000000000009ffff] (reserved)
(XEN)  [00000000000f0000, 00000000000fffff] (reserved)
(XEN)  [0000000000100000, 00000000cfe95fff] (usable)
(XEN)  [00000000cfe96000, 00000000cfffffff] (reserved)
(XEN)  [00000000f8000000, 00000000fbffffff] (reserved)
(XEN)  [00000000fec10000, 00000000fec10fff] (reserved)
(XEN)  [00000000fed40000, 00000000fed44fff] (reserved)
(XEN)  [0000000100000000, 000000012effffff] (usable)
(XEN)  [000000012f000000, 000000012fffffff] (reserved)
(XEN) ACPI: RSDP 000F2990, 0024 (r2 COREv4)
(XEN) ACPI: XSDT CFEB00E0, 007C (r1 COREv4 COREBOOT        0 CORE 20200925)
(XEN) ACPI: FACP CFEB1E10, 0114 (r6 COREv4 COREBOOT        0 CORE 20200925)
(XEN) ACPI: DSDT CFEB0280, 1B87 (r2 COREv4 COREBOOT    10001 INTL 20200925)
(XEN) ACPI: FACS CFEB0240, 0040
(XEN) ACPI: SSDT CFEB1F30, 01F3 (r2 COREv4 COREBOOT       2A CORE 20200925)
(XEN) ACPI: MCFG CFEB2130, 003C (r1 COREv4 COREBOOT        0 CORE 20200925)
(XEN) ACPI: TPM2 CFEB2170, 004C (r4 COREv4 COREBOOT        0 CORE 20200925)
(XEN) ACPI: APIC CFEB21C0, 007E (r3 COREv4 COREBOOT        0 CORE 20200925)
(XEN) ACPI: HEST CFEB2240, 01D0 (r1 COREv4 COREBOOT        0 CORE 20200925)
(XEN) ACPI: IVRS CFEB2410, 00E0 (r2 COREv4 COREBOOT        0 CORE 20200925)
(XEN) ACPI: SSDT CFEB24F0, 48A6 (r2 AMD    AGESA           2 MSFT  4000000)
(XEN) ACPI: SSDT CFEB6DA0, 07C8 (r1 AMD    AGESA           1 AMD         1)
(XEN) ACPI: DRTM CFEB7570, 007C (r1 COREv4 COREBOOT        0 CORE 20200925)
(XEN) ACPI: HPET CFEB75F0, 0038 (r1 COREv4 COREBOOT        0 CORE 20200925)
(XEN) System RAM: 4078MB (4176084kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-000000012f000000
(XEN) Domain heap initialised
(XEN) SMBIOS 3.0 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x818 (32 bits)
(XEN) ACPI: v5 SLEEP INFO: control[0:0], status[0:0]
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:804,1:0], pm1x_evt[1:800,1:0]
(XEN) ACPI: 32/64X FACS address mismatch in FADT - 
cfeb0240/0000000000000000, using 32
(XEN) ACPI:             wakeup_vec[cfeb024c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
(XEN) ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 4, version 33, address 0xfec00000, GSI 0-23
(XEN) ACPI: IOAPIC (id[0x05] address[0xfec20000] gsi_base[24])
(XEN) IOAPIC[1]: apic_id 5, version 33, address 0xfec20000, GSI 24-55
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 2 I/O APICs
(XEN) ACPI: HPET id: 0x10228201 base: 0xfed00000
(XEN) PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 3f
(XEN) PCI: MCFG area at f8000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) HEST: Table parsing has been initialized
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 4 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 56 GSI, 776 MSI/MSI-X
(XEN) CPU0: 1000 (600 ... 1400) MHz
(XEN) xstate: size: 0x340 and states: 0x7
(XEN) CPU0: AMD Fam16h machine check reporting enabled
(XEN) Speculative mitigation facilities:
(XEN)   Hardware hints:
(XEN)   Hardware features:
(XEN)   Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN)   Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: No, Other: 
BRANCH_HARDEN
(XEN)   Support for HVM VMs: RSB
(XEN)   Support for PV VMs: RSB
(XEN)   XPTI (64-bit PV only): Dom0 disabled, DomU disabled (without PCID)
(XEN)   PV L1TF shadowing: Dom0 disabled, DomU disabled
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN)  load_precision_shift: 18
(XEN)  load_window_shift: 30
(XEN)  underload_balance_tolerance: 0
(XEN)  overload_balance_tolerance: -3
(XEN)  runqueues arrangement: socket
(XEN)  cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Platform timer is 14.318MHz HPET
(XEN) Detected 998.134 MHz processor.
(XEN) alt table ffff82d040451290 -> ffff82d04045c342
(XEN) AMD-Vi: IOMMU Extended Features:
(XEN) - Peripheral Page Service Request
(XEN) - Invalidate All Command
(XEN) - Guest APIC
(XEN) - Performance Counters
(XEN) - Host Address Translation Size: 0x2
(XEN) - Guest Address Translation Size: 0
(XEN) - Guest CR3 Root Table Level: 0
(XEN) - Maximum PASID: 0x8
(XEN) - SMI Filter Register: 0x1
(XEN) - SMI Filter Register Count: 0x2
(XEN) - Guest Virtual APIC Modes: 0x1
(XEN) - Dual PPR Log: 0
(XEN) - Dual Event Log: 0
(XEN) - Device Table Segmentation: 0
(XEN) - Memory Access Routing and Control: 0
(XEN) - VM Table Size: 0
(XEN) AMD-Vi: Disabled HAP memory map sharing with IOMMU
(XEN) AMD-Vi: IOMMU 0 Enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) nr_sockets: 1
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=0 pin2=0
(XEN) Allocated console ring of 32 KiB.
(XEN) mwait-idle: does not run on family 22 model 48
(XEN) HVM: ASIDs enabled.
(XEN) SVM: Supported advanced features:
(XEN)  - Nested Page Tables (NPT)
(XEN)  - Last Branch Record (LBR) Virtualisation
(XEN)  - Next-RIP Saved on #VMEXIT
(XEN)  - DecodeAssists
(XEN)  - Pause-Intercept Filter
(XEN)  - Pause-Intercept Filter Threshold
(XEN)  - TSC Rate MSR
(XEN) HVM: SVM enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) alt table ffff82d040451290 -> ffff82d04045c342
(XEN) Brought up 4 CPUs
(XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) Adding cpu 0 to runqueue 0
(XEN)  First cpu on runqueue, activating
(XEN) Adding cpu 1 to runqueue 0
(XEN) Adding cpu 2 to runqueue 0
(XEN) Adding cpu 3 to runqueue 0
(XEN) MCA: Use hw thresholding to adjust polling frequency
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) NX (Execute Disable) protection active
(XEN) Dom0 has maximum 472 PIRQs
(XEN) *** Building a PV Dom0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x4600000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000118000000->0000000120000000 (155689 pages 
to be allocated)
(XEN)  Init. ramdisk: 000000012d029000->000000012efff2f7
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff84600000
(XEN)  Init. ramdisk: 0000000000000000->0000000000000000
(XEN)  Phys-Mach map: 0000008000000000->0000008000180000
(XEN)  Start info:    ffffffff84600000->ffffffff846004b8
(XEN)  Xenstore ring: 0000000000000000->0000000000000000
(XEN)  Console ring:  0000000000000000->0000000000000000
(XEN)  Page tables:   ffffffff84601000->ffffffff84628000
(XEN)  Boot stack:    ffffffff84628000->ffffffff84629000
(XEN)  TOTAL:         ffffffff80000000->ffffffff84800000
(XEN)  ENTRY ADDRESS: ffffffff82c441c0
(XEN) Dom0 has maximum 2 VCPUs
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 612kB init memory
(XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:00.2
(XEN) PCI add device 0000:00:02.0
(XEN) PCI add device 0000:00:02.1
(XEN) PCI add device 0000:00:02.2
(XEN) PCI add device 0000:00:02.3
(XEN) PCI add device 0000:00:02.4
(XEN) PCI add device 0000:00:02.5
(XEN) PCI add device 0000:00:08.0
(XEN) PCI add device 0000:00:10.0
(XEN) SR-IOV device 0000:00:11.0 has its virtual functions already 
enabled (01ab)
(XEN) PCI add device 0000:00:11.0
(XEN) PCI add device 0000:00:12.0
(XEN) PCI add device 0000:00:13.0
(XEN) PCI add device 0000:00:14.0
(XEN) PCI add device 0000:00:14.3
(XEN) PCI add device 0000:00:14.7
(XEN) PCI add device 0000:00:18.0
(XEN) PCI add device 0000:00:18.1
(XEN) PCI add device 0000:00:18.2
(XEN) PCI add device 0000:00:18.3
(XEN) PCI add device 0000:00:18.4
(XEN) PCI add device 0000:00:18.5
(XEN) PCI add device 0000:01:00.0
(XEN) PCI add device 0000:02:00.0
(XEN) PCI add device 0000:03:00.0
(XEN) PCI add device 0000:04:00.0
(XEN) PCI add device 0000:05:00.0
(XEN) HVM d3v0 save: CPU
(XEN) HVM d3v1 save: CPU
(XEN) HVM d3 save: PIC
(XEN) HVM d3 save: IOAPIC
(XEN) HVM d3v0 save: LAPIC
(XEN) HVM d3v1 save: LAPIC
(XEN) HVM d3v0 save: LAPIC_REGS
(XEN) HVM d3v1 save: LAPIC_REGS
(XEN) HVM d3 save: PCI_IRQ
(XEN) HVM d3 save: ISA_IRQ
(XEN) HVM d3 save: PCI_LINK
(XEN) HVM d3 save: PIT
(XEN) HVM d3 save: RTC
(XEN) HVM d3 save: HPET
(XEN) HVM d3 save: PMTIMER
(XEN) HVM d3v0 save: MTRR
(XEN) HVM d3v1 save: MTRR
(XEN) HVM d3 save: VIRIDIAN_DOMAIN
(XEN) HVM d3v0 save: CPU_XSAVE
(XEN) HVM d3v1 save: CPU_XSAVE
(XEN) HVM d3v0 save: VIRIDIAN_VCPU
(XEN) HVM d3v1 save: VIRIDIAN_VCPU
(XEN) HVM d3v0 save: VMCE_VCPU
(XEN) HVM d3v1 save: VMCE_VCPU
(XEN) HVM d3v0 save: TSC_ADJUST
(XEN) HVM d3v1 save: TSC_ADJUST
(XEN) HVM d3v0 save: CPU_MSR
(XEN) HVM d3v1 save: CPU_MSR
(XEN) HVM3 restore: CPU 0
(d3) HVM Loader
(d3) Detected Xen v4.14.5
(d3) Xenbus rings @0xfeffc000, event channel 1
(d3) System requested SeaBIOS
(d3) CPU speed is 998 MHz
(d3) Relocating guest memory for lowmem MMIO space disabled
(d3) PCI-ISA link 0 routed to IRQ5
(d3) PCI-ISA link 1 routed to IRQ10
(d3) PCI-ISA link 2 routed to IRQ11
(d3) PCI-ISA link 3 routed to IRQ5
(d3) pci dev 01:3 INTA->IRQ10
(d3) pci dev 02:0 INTA->IRQ11
(d3) pci dev 04:0 INTA->IRQ5
(d3) pci dev 05:0 INTA->IRQ10
(d3) pci dev 06:0 INTA->IRQ11
(d3) pci dev 07:0 INTA->IRQ5
(d3) No RAM in high memory; setting high_mem resource base to 100000000
(d3) pci dev 03:0 bar 10 size 002000000: 0f0000008
(d3) pci dev 02:0 bar 14 size 001000000: 0f2000008
(d3) pci dev 04:0 bar 30 size 000040000: 0f3000000
(d3) pci dev 05:0 bar 30 size 000040000: 0f3040000
(d3) pci dev 06:0 bar 30 size 000040000: 0f3080000
(d3) pci dev 03:0 bar 30 size 000010000: 0f30c0000
(d3) pci dev 03:0 bar 14 size 000001000: 0f30d0000
(d3) pci dev 07:0 bar 24 size 000001000: 0f30d1000
(d3) pci dev 02:0 bar 10 size 000000100: 00000c001
(d3) pci dev 04:0 bar 10 size 000000100: 00000c101
(d3) pci dev 04:0 bar 14 size 000000100: 0f30d2000
(d3) pci dev 05:0 bar 10 size 000000100: 00000c201
(d3) pci dev 05:0 bar 14 size 000000100: 0f30d2100
(d3) pci dev 06:0 bar 10 size 000000100: 00000c301
(d3) pci dev 06:0 bar 14 size 000000100: 0f30d2200
(d3) pci dev 07:0 bar 20 size 000000020: 00000c401
(d3) pci dev 01:1 bar 20 size 000000010: 00000c421
(d3) Multiprocessor initialisation:
(d3)  - CPU0 ... 40-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done.
(d3)  - CPU1 ... 40-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done.
(d3) Writing SMBIOS tables ...
(d3) Loading SeaBIOS ...
(d3) Creating MP tables ...
(d3) Loading ACPI ...
(d3) vm86 TSS at fc100280
(d3) BIOS map:
(d3)  10000-100e3: Scratch space
(d3)  c0000-fffff: Main BIOS
(d3) E820 table:
(d3)  [00]: 00000000:00000000 - 00000000:000a0000: RAM
(d3)  HOLE: 00000000:000a0000 - 00000000:000c0000
(d3)  [01]: 00000000:000c0000 - 00000000:00100000: RESERVED
(d3)  [02]: 00000000:00100000 - 00000000:7f800000: RAM
(d3)  HOLE: 00000000:7f800000 - 00000000:fc000000
(d3)  [03]: 00000000:fc000000 - 00000000:fc00b000: NVS
(d3)  [04]: 00000000:fc00b000 - 00000001:00000000: RESERVED
(d3) Invoking SeaBIOS ...
(d3) SeaBIOS (version 1.14.0-2)
(d3) BUILD: gcc: (Debian 10.2.0-23) 10.2.0 binutils: (GNU Binutils for 
Debian) 2.35.1
(d3)
(d3) Found Xen hypervisor signature at 40000000
(d3) Running on QEMU (i440fx)
(d3) xen: copy e820...
(d3) Relocating init from 0x000d6160 to 0x7f7ab680 (size 84192)
(d3) Found 10 PCI devices (max PCI bus is 00)
(d3) Allocated Xen hypercall page at 7f7ff000
(d3) Detected Xen v4.14.5
(d3) xen: copy BIOS tables...
(d3) Copying SMBIOS entry point from 0x00010020 to 0x000f5e20
(d3) Copying MPTABLE from 0xfc100180/fc100190 to 0x000f5d20
(d3) Copying PIR from 0x00010040 to 0x000f5ca0
(d3) Copying ACPI RSDP from 0x000100c0 to 0x000f5c70
(d3) table(50434146)=0xfc00a370 (via xsdt)
(d3) Using pmtimer, ioport 0xb008
(d3) table(50434146)=0xfc00a370 (via xsdt)
(d3) ACPI: parse DSDT at 0xfc001040 (len 37539)
(d3) parse_termlist: parse error, skip from 16/27641
(d3) parse_termlist: parse error, skip from 87/6041
(d3) Scan for VGA option rom
(d3) Running option rom at c000:0003
(d3) pmm call arg1=0
(d3) Turning on vga text mode console
(d3) SeaBIOS (version 1.14.0-2)
(d3) Machine UUID 029bc847-46ce-4395-b59a-96c6d6819ff2
(d3) ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
(d3) ATA controller 2 at 170/374/0 (irq 15 dev 9)
(d3) AHCI controller at 00:07.0, iobase 0xf30d1000, irq 5
(d3) Searching bootorder for: /pci@i0cf8/*@7/drive@0/disk@0
(d3) AHCI/0: Set transfer mode to UDMA-5
(d3) Searching bios-geometry for: /pci@i0cf8/*@7/drive@0/disk@0
(d3) AHCI/0: registering: "AHCI/0: QEMU HARDDISK ATA-7 Hard-Disk (16384 
MiBytes)"
(d3) PS2 keyboard initialized
(d3) Searching bootorder for: HALT
(d3) Found 0 lpt ports
(d3) Found 1 serial ports
(d3) All threads complete.
(d3) Scan for option roms
(d3) Running option rom at ca00:0003
(d3) pmm call arg1=1
(d3) pmm call arg1=0
(d3) pmm call arg1=1
(d3) pmm call arg1=0
(d3) Running option rom at cb00:0003
(d3) pmm call arg1=1
(d3) pmm call arg1=1
(d3) Running option rom at cc00:0003
(d3) pmm call arg1=1
(d3) pmm call arg1=1
(d3) Searching bootorder for: /pci@i0cf8/*@4
(d3) Searching bootorder for: /pci@i0cf8/*@5
(d3) Searching bootorder for: /pci@i0cf8/*@6
(d3)
(d3) Press ESC for boot menu.
(d3)
(d3) Searching bootorder for: HALT
(d3) drive 0x000f5bc0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 
s=33554432
(d3) Space available for UMB: cd000-e9000, f5640-f5bc0
(d3) Returned 253952 bytes of ZoneHigh
(d3) e820 map has 7 items:
(d3)   0: 0000000000000000 - 000000000009fc00 = 1 RAM
(d3)   1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
(d3)   2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
(d3)   3: 0000000000100000 - 000000007f7fe000 = 1 RAM
(d3)   4: 000000007f7fe000 - 000000007f800000 = 2 RESERVED
(d3)   5: 00000000fc000000 - 00000000fc00b000 = 4 NVS
(d3)   6: 00000000fc00b000 - 0000000100000000 = 2 RESERVED
(d3) enter handle_19:
(d3)   NULL
(d3) Booting from Hard Disk...
(d3) Booting from 0000:7c00
(XEN) d3v0: upcall vector 93
(XEN) d3v1: upcall vector 93



xl info:

host                   : xxx
release                : 5.10.0-22-amd64
version                : #1 SMP Debian 5.10.178-3 (2023-04-22)
machine                : x86_64
nr_cpus                : 4
max_cpu_id             : 3
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 1
cpu_mhz                : 998.134
hw_caps                : 
178bf3ff:36d8220b:2e500800:044037ff:00000001:00000008:00000000:00000100
virt_caps              : pv hvm hvm_directio pv_directio hap shadow
total_memory           : 4078
free_memory            : 413
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 14
xen_extra              : .5
xen_version            : 4.14.5
xen_caps               : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p 
hvm-3.0-x86_64
xen_scheduler          : credit2
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : placeholder dom0_mem=768M,max:768M 
dom0_max_vcpus=2 loglvl=all guest_loglvl=all iommu=verbose 
com1=115200,8n1,pci console=com1
cc_compiler            : x86_64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 
20210110
cc_compile_by          : pkg-xen-devel
cc_compile_domain      : lists.alioth.debian.org
cc_compile_date        : Thu Mar 23 19:40:49 UTC 2023
build_id               : c5e2deedc07540c4e17aa09e2a67b5aaf93d1806
xend_config_format     : 4


DomU configuration:

name = "xxx"
vcpus = 1
memory = 256
kernel      = '/usr/lib/grub-xen/grub-x86_64-xen.bin'
root        = ''
extra       = '(xen/xvda1)/boot/grub/grub.cfg'
disk = ['phy:/dev/vg0/xxx,xvda,w']
vif = [ 'ip=192.168.178.1 
fd00::1,mac=22:D0:7D:45:53:D7,bridge=xenbr0','ip=192.168.12.2 
fd10:0:0:12::2,mac=B6:33:0F:09:50:5F,bridge=xenbr2', 'ip=192.168.14.1 
fd00:0:0:14::1,mac=AA:47:D7:9B:42:E0,bridge=xenbr1', 
'ip=192.168.15.1,mac=EE:EF:C7:81:3E:65,bridge=xenbr5' ]
pci = ['05:00.0,permissive=1']
on_reboot = 'restart'
on_crash = 'destroy'


lspci -v on Dom0:

05:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac 
Wireless Network Adapter
        Flags: bus master, fast devsel, latency 0, IRQ 126
        Memory at d0000000 (64-bit, non-prefetchable) [size=2M]
        Expansion ROM at d0200000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/8 Maskable+ 64bit-
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 00-00-00-00-00-00-00-00
        Kernel driver in use: pciback
        Kernel modules: ath10k_pci