Re: [help] Xen 4.14.5 on Devuan 4.0 Chimaera, regression from Xen 4.0.1

[Jan Beulich <jbeulich@xxxxxxxx>]

On 13.03.2023 12:43, Andrew Cooper wrote:
> On 13/03/2023 9:36 am, Jan Beulich wrote:
>> On 10.03.2023 21:50, Denis wrote:
>>> On 10.03.2023 09:51, Jan Beulich wrote:
>>>> On 09.03.2023 21:37, Andrew Cooper wrote:
>>>>> On 09/03/2023 7:34 pm, tachyon_gun@xxxxxx wrote:
>>>>>> A short snippet of what I see when invoking "xl dmesg":
>>>>>>  
>>>>>> (XEN) No southbridge IO-APIC found in IVRS table
>>>>>> (XEN) AMD-Vi: Error initialization
>>>>>> (XEN) I/O virtualisation disabled 
>>>>>>  
>>>>>> What I would like to see (taken from Xen 4.0.1 running on Debian
>>>>>> Squeeze, in use since 2011):
>>>>>>  
>>>>>> (XEN) IOAPIC[0]: apic_id 8, version 33, address 0xfec00000, GSI 0-23
>>>>>> (XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
>>>>>> (XEN) Using scheduler: SMP Credit Scheduler (credit)
>>>>>> (XEN) Detected 2611.936 MHz processor.
>>>>>> (XEN) Initing memory sharing.
>>>>>> (XEN) HVM: ASIDs enabled.
>>>>>> (XEN) HVM: SVM enabled
>>>>>> (XEN) HVM: Hardware Assisted Paging detected.
>>>>>> (XEN) AMD-Vi: IOMMU 0 Enabled.
>>>>>> (XEN) I/O virtualisation enabled
>>>>>>  
>>>>>> My question would be if this is "normal" behaviour due to older hardware
>>>>>> being used with newer versions of Xen (compared to the old 4.0.1) or if
>>>>>> this is a bug.
>>>>>> If the latter, has this been addressed already in newer version (4.14+)?
>>>> No, the code there is still the same. The commit introducing the check
>>>> (06bbcaf48d09 ["AMD IOMMU: fail if there is no southbridge IO-APIC"])
>>>> specifically provided for a workaround: "iommu=no-intremap" on the Xen
>>>> command line. Could you give this a try? (As per below this could be
>>>> what we want to do "automatically" in such a situation, i.e. without
>>>> the need for a command line option. But you then still would face a
>>>> perceived regression of interrupt remapping being disabled on such a
>>>> system.)
>>>>
>>>> The other possible workaround, "iommu=no-amd-iommu-perdev-intremap",
>>>> is something I rather wouldn't want to recommend, but you may still
>>>> want to give it a try.
>>>  
>>> Thanks for your reply.
>>>
>>> I added the lines you suggested and it seems that "AMD-Vi: IOMMU 0" and
>>> "I/O virtualisation" is enabled again.
>> Good - that'll have to do as a workaround for the time being.
> 
> Not really.  Booting this system with no-intremap is still a regression
> vs Xen 4.0.1

Well, "for the time being" meant untiol we figure out what exactly we can
do here.

> Disabling interrupt remapping on PCIe devices because we can't figure
> out interrupt handling around the PCI bridge is still bad behaviour.
> 
> 
> What we need to figure out here is how interrupts from the PCI bridge
> actually work.  The IVRS table does contain records covering the devices
> on the Southbridge, including the PCI bridge and it's entire subordinate
> range.
> 
> MSI/MSI-X interrupts from the PCI devices will work fine (they'll have a
> proper source id), so the only question is about line interrupts.  They
> ought to appear with the bridge's source id, and ought to be fine too.
> 
> 
> I see no evidence to suggest that the IVRS/MADT are incorrect or
> incomplete.  Xen's believe that there must be a southbridge IO-APIC
> special device seems to be the false entity here.

It may be possible to behave better here, but my reading of Linux sources
suggests that they would do exactly that on this system - disable
interrupt remapping (I had asked Denis for trying that out to double
check, but so far he provided back only hypervisor logs). Also my
understanding of the reason for the check is that for pin-based interrupts
of devices behind the southbridge there ought to be an I/O-APIC there. But
it may be one or both of my understanding and the (original) Linux change
(and thus ours) are wrong / misguided.

Do you have concrete suggestions towards logic which might eliminate the
need for the check for a present southbridge?

Jan