Re: [help] Xen 4.14.5 on Devuan 4.0 Chimaera, regression from Xen 4.0.1

[Denis <tachyon_gun@xxxxxx>]

On 10.03.2023 09:51, Jan Beulich wrote:
> Betreff: Re: [help] Xen 4.14.5 on Devuan 4.0 Chimaera, regression from Xen 
> 4.0.1
>
> On 09.03.2023 21:37, Andrew Cooper wrote:
> > On 09/03/2023 7:34 pm, tachyon_gun@xxxxxx wrote:
> >> Subject:
> >> [help] Xen 4.14.5 on Devuan 4.0 Chimaera
> >> From:
> >> tachyon_gun@xxxxxx
> >> Date:
> >> 09/03/2023, 7:34 pm
> >>
> >> To:
> >> xen-devel@xxxxxxxxxxxxxxxxxxxx
> >>
> >>
> >> Hello.
> >>
> >> Following the advice of Andrew Cooper (thanks for helping out) over on
> >> OFTC.net in #xen, I'll post this here.
> >> If this is the wrong place, please move it to the right section of your
> >> mailing lists.
> >>  
> >> I got some problems running Xen 4.14.5 on Devuan 4.0.
> >>  
> >> The AMD-Vi and I/O virtualisation are not being enabled when booting up
> >> the host system with Xen.
> >>  
> >> Hardware used:
> >> Mainboard: Gigabyte GA-890FXA-UD5, BIOS version F6 (2010.11.24, the
> >> latest version)
> >> CPU: AMD Phenom II X4 910e
> >> Memory: 16GB
> >> Storage: 2TB Crucial MX500
> >>  
> >> A short snippet of what I see when invoking "xl dmesg":
> >>  
> >> (XEN) No southbridge IO-APIC found in IVRS table
> >> (XEN) AMD-Vi: Error initialization
> >> (XEN) I/O virtualisation disabled 
> >>  
> >> What I would like to see (taken from Xen 4.0.1 running on Debian
> >> Squeeze, in use since 2011):
> >>  
> >> (XEN) IOAPIC[0]: apic_id 8, version 33, address 0xfec00000, GSI 0-23
> >> (XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
> >> (XEN) Using scheduler: SMP Credit Scheduler (credit)
> >> (XEN) Detected 2611.936 MHz processor.
> >> (XEN) Initing memory sharing.
> >> (XEN) HVM: ASIDs enabled.
> >> (XEN) HVM: SVM enabled
> >> (XEN) HVM: Hardware Assisted Paging detected.
> >> (XEN) AMD-Vi: IOMMU 0 Enabled.
> >> (XEN) I/O virtualisation enabled
> >>  
> >> My question would be if this is "normal" behaviour due to older hardware
> >> being used with newer versions of Xen (compared to the old 4.0.1) or if
> >> this is a bug.
> >> If the latter, has this been addressed already in newer version (4.14+)?
> 
> No, the code there is still the same. The commit introducing the check
> (06bbcaf48d09 ["AMD IOMMU: fail if there is no southbridge IO-APIC"])
> specifically provided for a workaround: "iommu=no-intremap" on the Xen
> command line. Could you give this a try? (As per below this could be
> what we want to do "automatically" in such a situation, i.e. without
> the need for a command line option. But you then still would face a
> perceived regression of interrupt remapping being disabled on such a
> system.)
> 
> The other possible workaround, "iommu=no-amd-iommu-perdev-intremap",
> is something I rather wouldn't want to recommend, but you may still
> want to give it a try.
 
Thanks for your reply.

I added the lines you suggested and it seems that "AMD-Vi: IOMMU 0" and
"I/O virtualisation" is enabled again.

There are only minor differences in the "xl dmesg" output.
In the one with "iommu=no-amd-iommu-perdev-intremap", 
the line "No southbridge IO-APIC found in IVRS table" is listed.

Though I yet have to test a HVM domU with passthrough.

I'll attach the two "xl dmesg" files and a third one from the old version of 
Xen.

> >> I'll attach some log files (hypervisor.log, dom0.log, xl_info.log,
> >> lspci_vvv.log, acpi.dmp, ivrs.dat, ivrs.dsl).
> >>  
> >> Thank you for your time.
> > 
> > Let me braindump the investigation so far before I forget it.
> > 
> > Xen requires that there is an IVRS special-device record describing an
> > IO-APIC 00:14.0.  This check failing is the source of the "No
> > southbridge" message, and the cause of the IOMMU(s) being turned off.
> > 
> > The MADT and IVRS tables agree that there is one IO-APIC in the system,
> > but that's the northbridge IO-APIC, not the southbridge.
> > 
> > The block diagram for the southbridge does have a PIC/IO-APIC as part of
> > the PCI bridge, so honestly I was expecting the MADT to describe 2
> > IO-APICs.  But OTOH, I could see this legitimately not existing in
> > configurations where the PCI bridge isn't in use.
> > 
> > `xl dmesg` does have a few unknown irqs, so there might be something
> > down in the southbridge really generating interrupts.  Or there might be
> > a IRQ misconfiguration elsewhere, and this is just a red herring.
> > 
> > However, a consequence of the northbridge and southbridge being separate
> > chips means that all southbridge IO is fully encapsulated by the IOMMU
> > in the northbridge.
> > 
> > So irrespective of whether there is ah IO-APIC operating properly in the
> > southbridge, and whether or not it's properly described, I think Xen's
> > insistence that there must be an IVRS special-device entry for it is bogus.
> > 
> > 
> > Furthermore, Xen's decisions are monumentally stupid.  It takes a
> > specifically safe (IOMMU-wise) system, and because it can't figure out a
> > partial aspect of interrupt handling the southbridge, decided that the
> > system can't be safe (a false conclusion) and turns the IOMMU off fully
> > to compensate, which makes the system concretely less safe.

Also, thank you Andrew for bringing this in.

> So this touches once again the area of the fuzzy split between the IOMMU
> being disabled as a whole (meaning DMA+interrupt remapping off) vs only
> one of the two being off (where presently we are unable to turn off just
> DMA remapping). Indeed the original Linux commit, which our change was
> inspired by, results in merely interrupt remapping getting turned off
> (afaict), and that hasn't changed. (Would be nice to have this confirmed
> for the system in question, i.e. without Xen underneath Linux.) It would
> certainly be possible for us to do so too - it might be a one line change:
 
Could you elaborate on that one?
Should I test something else?

> @@ ... @@ static int __init cf_check parse_ivrs_table(struct
>      if ( !error && !sb_ioapic )
>      {
>          if ( amd_iommu_perdev_intremap )
> -            error = -ENXIO;
> +            iommu_intremap = false;
>          printk("%sNo southbridge IO-APIC found in IVRS table\n",
>                 amd_iommu_perdev_intremap ? XENLOG_ERR : XENLOG_WARNING);
>      }
> 
> but only if there aren't any earlier decisions that need undoing, and
> only if all subsequent code will properly honor that turning off of
> intremap. I'll try to get to doing the necessary audit some time next
> week.
 
Denis

Attachment: hypervisor_iommu=no-amd-iommu-perdev-intremap.log
Description: Binary data

Attachment: hypervisor_iommu=no-intremap.log
Description: Binary data

Attachment: hypervisor_old_xen_iommu=debug_loglvl=all_guest_loglvl=all.log
Description: Binary data