Re: [PATCH v1 01/23] pc/xen: Xen Q35 support: provide IRQ handling for PCI devices

[David Woodhouse <dwmw2@xxxxxxxxxxxxx>]

On Tue, 2023-06-20 at 13:24 -0400, Joel Upham wrote:
> The primary difference in PCI device IRQ management between Xen HVM and
> QEMU is that Xen PCI IRQs are "device-centric" while QEMU PCI IRQs are
> "chipset-centric". Namely, Xen uses PCI device BDF and INTx as coordinates
> to assert IRQ while QEMU finds out to which chipset PIRQ the IRQ is routed
> through the hierarchy of PCI buses and manages IRQ assertion on chipset
> side (as PIRQ inputs).
> 
> Two callback functions are used for this purpose: .map_irq and .set_irq
> (named after corresponding structure fields). Corresponding Xen-specific
> callback functions are piix3_set_irq() and pci_slot_get_pirq(). In Xen
> case these functions do not operate on pirq pin numbers. Instead, they use
> a specific value to pass BDF/INTx information between .map_irq and
> .set_irq -- PCI device devfn and INTx pin number are combined into
> pseudo-PIRQ in pci_slot_get_pirq, which piix3_set_irq later decodes back
> into devfn and INTx number for passing to *set_pci_intx_level() call.
> 
> For Xen on Q35 this scheme is still applicable, with the exception that
> function names are non-descriptive now and need to be renamed to show
> their common i440/Q35 nature. Proposed new names are:
> 
> xen_pci_slot_get_pirq --> xen_cmn_pci_slot_get_pirq
> xen_piix3_set_irq     --> xen_cmn_set_irq
> 
> Another IRQ-related difference between i440 and Q35 is the number of PIRQ
> inputs and PIRQ routers (PCI IRQ links in terms of ACPI) available. i440
> has 4 PCI interrupt links, while Q35 has 8 (PIRQA...PIRQH).
> Currently Xen have support for only 4 PCI links, so we describe only 4 of
> 8 PCI links in ACPI tables. Also, hvmloader disables PIRQ routing for
> PIRQE..PIRQH by writing 80h into corresponding PIRQ[n]_ROUT registers.
> 
> All this PCI interrupt routing stuff is largely an ancient legacy from PIC
> era. It's hardly worth to extend number of PCI links supported as we
> normally deal with APIC mode and/or MSI interrupts.
> 
> The only useful thing to do with PIRQE..PIRQH routing currently is to
> check if guest actually attempts to use it for some reason (despite ACPI
> PCI routing information provided). In this case, a warning is logged.
> 
> Things have changed a bit in modern Qemu, and more changes to the IRQ
> mapping had to be done inside the lpc_ich9 to write the irqs and setup
> the mappings.
> 
> Signed-off-by: Alexey Gerasimenko <x1917x@xxxxxxxxx>
> Signed-off-by: Joel Upham <jupham125@xxxxxxxxx>

Please ensure you test Linux guests with attached/emulated PCI devices
and (at least) the following command lines:

 • pci=nomsi
 • pci=nomsi noapic
 • xen_no_vector_callback pci=nomsi noapic
 • xen_no_vector_callback pci=nomsi

Obviously, with pci=nomsi you're actually *using* PCI INTx, and force
it not to use MSI.

With noapic you force it to use the i8259. The INTx should be routed to
a *different* interrupt number on the i8259 than the I/O APIC.
(I thought the IRQ should be asserted on both, but QEMU's
ich9_route_intx_pin_to_irq() seems to route to the I/O APIC *only* if
it's disabled in the i8259? Sadly, QEMU lacks a comment explaining if
that's a faithful emulation of the ICH9, or if it's just a workaround
for the fact that that function is only allowed one return value.)

This one has potential to be particularly entertaining because the
guest may route INTx IRQs to PIRQ event channels. And I think those are
supposed to use the I/O APIC pin numbering, *not* the PIC numbering. I
*think* you get away with it because of the workaround discussed above.

With xen_no_vector_callback you *also* use the emulated INTx of the Xen
platform PCI device, for event channel callbacks. We want to test that
via both the i8259 and the I/O APIC. And it also ensures that the INTx
of the real PCI devices is handled via the PIC or I/OAPIC instead of
letting the guest route them to PIRQs.

Attachment: smime.p7s
Description: S/MIME cryptographic signature