Aw: Re: [help] Xen 4.14.5 on Devuan 4.0 Chimaera, regression from Xen 4.0.1

[Denis <tachyon_gun@xxxxxx>]

On 13.03.2023 10:36, Jan wrote
> On 10.03.2023 21:50, Denis wrote:
> > On 10.03.2023 09:51, Jan Beulich wrote:
> >> On 09.03.2023 21:37, Andrew Cooper wrote:
> >>> On 09/03/2023 7:34 pm, tachyon_gun@xxxxxx wrote:
> >>>> A short snippet of what I see when invoking "xl dmesg":
> >>>>  
> >>>> (XEN) No southbridge IO-APIC found in IVRS table
> >>>> (XEN) AMD-Vi: Error initialization
> >>>> (XEN) I/O virtualisation disabled 
> >>>>  
> >>>> What I would like to see (taken from Xen 4.0.1 running on Debian
> >>>> Squeeze, in use since 2011):
> >>>>  
> >>>> (XEN) IOAPIC[0]: apic_id 8, version 33, address 0xfec00000, GSI 0-23
> >>>> (XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
> >>>> (XEN) Using scheduler: SMP Credit Scheduler (credit)
> >>>> (XEN) Detected 2611.936 MHz processor.
> >>>> (XEN) Initing memory sharing.
> >>>> (XEN) HVM: ASIDs enabled.
> >>>> (XEN) HVM: SVM enabled
> >>>> (XEN) HVM: Hardware Assisted Paging detected.
> >>>> (XEN) AMD-Vi: IOMMU 0 Enabled.
> >>>> (XEN) I/O virtualisation enabled
> >>>>  
> >>>> My question would be if this is "normal" behaviour due to older hardware
> >>>> being used with newer versions of Xen (compared to the old 4.0.1) or if
> >>>> this is a bug.
> >>>> If the latter, has this been addressed already in newer version (4.14+)?
> >>
> >> No, the code there is still the same. The commit introducing the check
> >> (06bbcaf48d09 ["AMD IOMMU: fail if there is no southbridge IO-APIC"])
> >> specifically provided for a workaround: "iommu=no-intremap" on the Xen
> >> command line. Could you give this a try? (As per below this could be
> >> what we want to do "automatically" in such a situation, i.e. without
> >> the need for a command line option. But you then still would face a
> >> perceived regression of interrupt remapping being disabled on such a
> >> system.)
> >>
> >> The other possible workaround, "iommu=no-amd-iommu-perdev-intremap",
> >> is something I rather wouldn't want to recommend, but you may still
> >> want to give it a try.
> >  
> > Thanks for your reply.
> > 
> > I added the lines you suggested and it seems that "AMD-Vi: IOMMU 0" and
> > "I/O virtualisation" is enabled again.
> 
> Good - that'll have to do as a workaround for the time being.
> 
> > There are only minor differences in the "xl dmesg" output.
> > In the one with "iommu=no-amd-iommu-perdev-intremap", 
> > the line "No southbridge IO-APIC found in IVRS table" is listed.
> 
> That's as expected - the message is issued as a non-error one in this
> case.
> 
> > Though I yet have to test a HVM domU with passthrough.
> > 
> > I'll attach the two "xl dmesg" files and a third one from the old version 
> > of Xen.
> > 
> >>>> I'll attach some log files (hypervisor.log, dom0.log, xl_info.log,
> >>>> lspci_vvv.log, acpi.dmp, ivrs.dat, ivrs.dsl).
> >>>>  
> >>>> Thank you for your time.
> >>>
> >>> Let me braindump the investigation so far before I forget it.
> >>>
> >>> Xen requires that there is an IVRS special-device record describing an
> >>> IO-APIC 00:14.0.  This check failing is the source of the "No
> >>> southbridge" message, and the cause of the IOMMU(s) being turned off.
> >>>
> >>> The MADT and IVRS tables agree that there is one IO-APIC in the system,
> >>> but that's the northbridge IO-APIC, not the southbridge.
> >>>
> >>> The block diagram for the southbridge does have a PIC/IO-APIC as part of
> >>> the PCI bridge, so honestly I was expecting the MADT to describe 2
> >>> IO-APICs.  But OTOH, I could see this legitimately not existing in
> >>> configurations where the PCI bridge isn't in use.
> >>>
> >>> `xl dmesg` does have a few unknown irqs, so there might be something
> >>> down in the southbridge really generating interrupts.  Or there might be
> >>> a IRQ misconfiguration elsewhere, and this is just a red herring.
> >>>
> >>> However, a consequence of the northbridge and southbridge being separate
> >>> chips means that all southbridge IO is fully encapsulated by the IOMMU
> >>> in the northbridge.
> >>>
> >>> So irrespective of whether there is ah IO-APIC operating properly in the
> >>> southbridge, and whether or not it's properly described, I think Xen's
> >>> insistence that there must be an IVRS special-device entry for it is 
> >>> bogus.
> >>>
> >>>
> >>> Furthermore, Xen's decisions are monumentally stupid.  It takes a
> >>> specifically safe (IOMMU-wise) system, and because it can't figure out a
> >>> partial aspect of interrupt handling the southbridge, decided that the
> >>> system can't be safe (a false conclusion) and turns the IOMMU off fully
> >>> to compensate, which makes the system concretely less safe.
> > 
> > Also, thank you Andrew for bringing this in.
> > 
> >> So this touches once again the area of the fuzzy split between the IOMMU
> >> being disabled as a whole (meaning DMA+interrupt remapping off) vs only
> >> one of the two being off (where presently we are unable to turn off just
> >> DMA remapping). Indeed the original Linux commit, which our change was
> >> inspired by, results in merely interrupt remapping getting turned off
> >> (afaict), and that hasn't changed. (Would be nice to have this confirmed
> >> for the system in question, i.e. without Xen underneath Linux.) It would
> >> certainly be possible for us to do so too - it might be a one line change:
> >  
> > Could you elaborate on that one?
> 
> I guess I'd need to know what you're missing; the entire paragraph was
> intended more for Andrew and Roger (and others who are interested on the
> "development" side) rather than you. Specifically ...
> 
> > Should I test something else?
> 
> ... there was no request for any further testing here, for the moment.

ah...sorry, going by "Would be nice to have this confirmed forthe system 
in question, i.e. without Xen underneath Linux" I thought I could test 
something which might help shed some light on all of this.

Denis