Re: [help] Xen 4.14.5 on Devuan 4.0 Chimaera, regression from Xen 4.0.1

[Jan Beulich <jbeulich@xxxxxxxx>]

On 09.03.2023 21:37, Andrew Cooper wrote:
> On 09/03/2023 7:34 pm, tachyon_gun@xxxxxx wrote:
>> Subject:
>> [help] Xen 4.14.5 on Devuan 4.0 Chimaera
>> From:
>> tachyon_gun@xxxxxx
>> Date:
>> 09/03/2023, 7:34 pm
>>
>> To:
>> xen-devel@xxxxxxxxxxxxxxxxxxxx
>>
>>
>> Hello.
>>
>> Following the advice of Andrew Cooper (thanks for helping out) over on
>> OFTC.net in #xen, I'll post this here.
>> If this is the wrong place, please move it to the right section of your
>> mailing lists.
>>  
>> I got some problems running Xen 4.14.5 on Devuan 4.0.
>>  
>> The AMD-Vi and I/O virtualisation are not being enabled when booting up
>> the host system with Xen.
>>  
>> Hardware used:
>> Mainboard: Gigabyte GA-890FXA-UD5, BIOS version F6 (2010.11.24, the
>> latest version)
>> CPU: AMD Phenom II X4 910e
>> Memory: 16GB
>> Storage: 2TB Crucial MX500
>>  
>> A short snippet of what I see when invoking "xl dmesg":
>>  
>> (XEN) No southbridge IO-APIC found in IVRS table
>> (XEN) AMD-Vi: Error initialization
>> (XEN) I/O virtualisation disabled 
>>  
>> What I would like to see (taken from Xen 4.0.1 running on Debian
>> Squeeze, in use since 2011):
>>  
>> (XEN) IOAPIC[0]: apic_id 8, version 33, address 0xfec00000, GSI 0-23
>> (XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
>> (XEN) Using scheduler: SMP Credit Scheduler (credit)
>> (XEN) Detected 2611.936 MHz processor.
>> (XEN) Initing memory sharing.
>> (XEN) HVM: ASIDs enabled.
>> (XEN) HVM: SVM enabled
>> (XEN) HVM: Hardware Assisted Paging detected.
>> (XEN) AMD-Vi: IOMMU 0 Enabled.
>> (XEN) I/O virtualisation enabled
>>  
>> My question would be if this is "normal" behaviour due to older hardware
>> being used with newer versions of Xen (compared to the old 4.0.1) or if
>> this is a bug.
>> If the latter, has this been addressed already in newer version (4.14+)?

No, the code there is still the same. The commit introducing the check
(06bbcaf48d09 ["AMD IOMMU: fail if there is no southbridge IO-APIC"])
specifically provided for a workaround: "iommu=no-intremap" on the Xen
command line. Could you give this a try? (As per below this could be
what we want to do "automatically" in such a situation, i.e. without
the need for a command line option. But you then still would face a
perceived regression of interrupt remapping being disabled on such a
system.)

The other possible workaround, "iommu=no-amd-iommu-perdev-intremap",
is something I rather wouldn't want to recommend, but you may still
want to give it a try.

>> I'll attach some log files (hypervisor.log, dom0.log, xl_info.log,
>> lspci_vvv.log, acpi.dmp, ivrs.dat, ivrs.dsl).
>>  
>> Thank you for your time.
> 
> Let me braindump the investigation so far before I forget it.
> 
> Xen requires that there is an IVRS special-device record describing an
> IO-APIC 00:14.0.  This check failing is the source of the "No
> southbridge" message, and the cause of the IOMMU(s) being turned off.
> 
> The MADT and IVRS tables agree that there is one IO-APIC in the system,
> but that's the northbridge IO-APIC, not the southbridge.
> 
> The block diagram for the southbridge does have a PIC/IO-APIC as part of
> the PCI bridge, so honestly I was expecting the MADT to describe 2
> IO-APICs.  But OTOH, I could see this legitimately not existing in
> configurations where the PCI bridge isn't in use.
> 
> `xl dmesg` does have a few unknown irqs, so there might be something
> down in the southbridge really generating interrupts.  Or there might be
> a IRQ misconfiguration elsewhere, and this is just a red herring.
> 
> However, a consequence of the northbridge and southbridge being separate
> chips means that all southbridge IO is fully encapsulated by the IOMMU
> in the northbridge.
> 
> So irrespective of whether there is ah IO-APIC operating properly in the
> southbridge, and whether or not it's properly described, I think Xen's
> insistence that there must be an IVRS special-device entry for it is bogus.
> 
> 
> Furthermore, Xen's decisions are monumentally stupid.  It takes a
> specifically safe (IOMMU-wise) system, and because it can't figure out a
> partial aspect of interrupt handling the southbridge, decided that the
> system can't be safe (a false conclusion) and turns the IOMMU off fully
> to compensate, which makes the system concretely less safe.

So this touches once again the area of the fuzzy split between the IOMMU
being disabled as a whole (meaning DMA+interrupt remapping off) vs only
one of the two being off (where presently we are unable to turn off just
DMA remapping). Indeed the original Linux commit, which our change was
inspired by, results in merely interrupt remapping getting turned off
(afaict), and that hasn't changed. (Would be nice to have this confirmed
for the system in question, i.e. without Xen underneath Linux.) It would
certainly be possible for us to do so too - it might be a one line change:

@@ ... @@ static int __init cf_check parse_ivrs_table(struct
     if ( !error && !sb_ioapic )
     {
         if ( amd_iommu_perdev_intremap )
-            error = -ENXIO;
+            iommu_intremap = false;
         printk("%sNo southbridge IO-APIC found in IVRS table\n",
                amd_iommu_perdev_intremap ? XENLOG_ERR : XENLOG_WARNING);
     }

but only if there aren't any earlier decisions that need undoing, and
only if all subsequent code will properly honor that turning off of
intremap. I'll try to get to doing the necessary audit some time next
week.

Jan