Re: AW: AW: AW: AW: Xen data from meta-virtualization layer

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>]

Adding Bertrand, Oleksandr, Julien, and others -- they have a more
recent experience with GICv3 ITS than me and might be able to help.
I am attaching the device tree Leo sent a few days ago for reference.


Typically when you can set the ethernet link up and no packets are
exchanged it is because of a missing interrupt. In this case a missing
MSI.

Bertrand, I believe you tried the GIC ITS driver with PCI devices
recently. It is expected to work correctly with MSIs in Dom0, right?



On Tue, 17 Nov 2020, Leo Krueger wrote:
> Hi,
> 
> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it 
> before...) but then had to add the following node to my device tree
> 
>       gic_lpi_base: syscon@0x80000000 {
>               compatible = "gic-lpi-base";
>               reg = <0x0 0x80000000 0x0 0x100000>;
>               max-gic-redistributors = <2>;
>       };
> 
> to somehow change something in regard to the ITS and MSI/MSI-X
> 
> (XEN) GICv3 initialization:
> (XEN)       gic_dist_addr=0x00000006000000
> (XEN)       gic_maintenance_irq=25
> (XEN)       gic_rdist_stride=0
> (XEN)       gic_rdist_regions=1
> (XEN)       redistributor regions:
> (XEN)         - region 0: 0x00000006040000 - 0x00000006080000
> (XEN) GICv3: using at most 57344 LPIs on the host.
> (XEN) GICv3: 288 lines, (IID 0001143b).
> (XEN) GICv3: Found ITS @0x6020000
> (XEN) using non-cacheable ITS command queue
> (XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000
> 
> [    0.000000] GICv3: Distributor has no Range Selector support
> [    0.000000] GICv3: no VLPI support, no direct LPI support
> [    0.000000] ITS [mem 0x06020000-0x0603ffff]
> [    0.000000] ITS@0x0000000006020000: allocated 65536 Devices @dc880000 
> (flat, esz 8, psz 64K, shr 1)
> [    0.000000] ITS@0x0000000006020000: allocated 32768 Interrupt Collections 
> @dc820000 (flat, esz 2, psz 64K, shr 1)
> [    0.000000] GIC: using LPI property table @0x00000000dc830000
> [    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000006040000
> [    0.000000] CPU0: using LPI pending table @0x00000000dc840000
> ...
> [    0.040080] Platform MSI: gic-its domain created
> [    0.040136] PCI/MSI: /interrupt-controller/gic-its domain created
> [    0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created
> 
> 
> Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X 
> might fail!" log messages for some PCI devices, but at least the on-board 
> ethernet ports (fsl_enetc ) are initialized.
> I can set the link up and a link is successfully established.
> 
> But (!) I cannot receive or transmit anything (no error message...) and there 
> seem to be no interrupts:
> 
> 29:          0   ITS-MSI   1 Edge      gbe0-rxtx0
>  32:          0   ITS-MSI 8192 Edge      ptp_qoriq
> 
> (from /proc/interrupts).
> 
> Any idea on this one? I keep digging and checking whether my device tree 
> needs some additional fixes.
> 
> Kind regards,
> Leo
> 
> --
> Leo Krüger, M.Sc.
> Senior Systems Engineer Distributed Systems
> Intelligent Digital Cabin
> 
> ZAL Zentrum für Angewandte Luftfahrtforschung GmbH
> Hein-Saß-Weg 22
> 21129 Hamburg
>  
> +49 (0) 40 248 595-154
> 
> zal.aero | twitter.com/ZALTechCenter | facebook.com/ZALTechCenter 
> 
> ZAL Zentrum für Angewandte Luftfahrtforschung GmbH 
> Sitz der Gesellschaft / Legal Domicile: Hamburg 
> Registergericht / Registration Court: Amtsgericht Hamburg HRB 110232
> Vorsitzender des Aufsichtsrates / Chairman of the Supervisory Board: StR 
> Andreas Rieckhof
> Geschäftsführung / Board of Management: Roland Gerhards
> 
> Disclaimer:
> This e-mail may contain confidential and/or privileged information. If you 
> are not the intended recipient (or have
> received this mail in error), please notify the sender immediately and 
> destroy this e-mail. Any unauthorised copying, 
> disclosure or distribution of the material in this e-mail is strictly 
> forbidden.
> 
> > -----Ursprüngliche Nachricht-----
> > Von: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > Gesendet: Dienstag, 17. November 2020 01:59
> > An: Leo Krueger <leo.krueger@xxxxxxxx>
> > Cc: Peng Fan <peng.fan@xxxxxxx>; Stefano Stabellini
> > <stefano.stabellini@xxxxxxxxxx>; brucea@xxxxxxxxxx; Cornelia Bruelhart
> > <cornelia.bruelhart@xxxxxxxx>
> > Betreff: Re: AW: AW: AW: Xen data from meta-virtualization layer
> > 
> > Replies inline below
> > 
> > 
> > On Sun, 15 Nov 2020, Leo Krueger wrote:
> > > Hi Peng, hi Stefano,
> > >
> > >
> > >
> > > sorry for the long silence…
> > >
> > >
> > >
> > > I tried the change suggested (and hope I didn’t do anything wrong
> > > while doing so…) on top of XEN 4.13.2 (before, I always tried with
> > > 4.12 but wanted to give 4.13.2 a try as well) but I do not see any 
> > > difference,
> > still the same “unhandled context fault” log entries pop up and I cannot
> > access my sdcard.
> > >
> > >
> > >
> > > As it seems to work without respectively disabled iommu, that would be
> > > fine for me for now. What I am worried about a bit more is PCIe or
> > MSI/MSIX to be exact.
> > >
> > >
> > >
> > > Here is the gic-v3 and its node from my device tree:
> > >
> > >
> > >
> > > interrupt-controller@6000000 {
> > >
> > >         compatible = "arm,gic-v3";
> > >
> > >         #address-cells = <0x2>;
> > >
> > >         #size-cells = <0x2>;
> > >
> > >         ranges;
> > >
> > >         reg = <0x0 0x6000000 0x0 0x10000 0x0 0x6040000 0x0 0x40000>;
> > >
> > >         #interrupt-cells = <0x3>;
> > >
> > >         interrupt-controller;
> > >
> > >         interrupts = <0x1 0x9 0xf08>;
> > >
> > >         phandle = <0x1>;
> > >
> > >
> > >
> > >         gic-its@6020000 {
> > >
> > >                 compatible = "arm,gic-v3-its";
> > >
> > >                 msi-controller;
> > >
> > >                 reg = <0x0 0x6020000 0x0 0x20000>;
> > >
> > >                 phandle = <0xd>;
> > >
> > >         };
> > >
> > > };
> > >
> > >
> > >
> > > And here are some kernel log excerpts related to GIC when booting
> > without (!) XEN:
> > >
> > >
> > >
> > > [    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
> > >
> > > [    0.000000] GICv3: Distributor has no Range Selector support
> > >
> > > [    0.000000] GICv3: no VLPI support, no direct LPI support
> > >
> > > [    0.000000] ITS [mem 0x06020000-0x0603ffff]
> > >
> > > [    0.000000] ITS@0x0000000006020000: allocated 65536 Devices
> > > @20fb880000 (flat, esz 8, psz 64K, shr 0)
> > >
> > > [    0.000000] ITS: using cache flushing for cmd queue
> > >
> > > [    0.000000] GIC: using LPI property table @0x00000020fb830000
> > >
> > > [    0.000000] GICv3: CPU0: found redistributor 0 region
> > > 0:0x0000000006040000
> > >
> > > [    0.000000] CPU0: using LPI pending table @0x00000020fb840000
> > >
> > > [    0.000000] GIC: using cache flushing for LPI property table
> > >
> > >
> > >
> > > However, when booting with XEN, only the following three lines are
> > contained in the kernel log:
> > >
> > >
> > >
> > > [    0.000000] GICv3: Distributor has no Range Selector support
> > >
> > > [    0.000000] GICv3: no VLPI support, no direct LPI support
> > >
> > > [    0.000000] GICv3: CPU0: found redistributor 0 region
> > > 0:0x0000000006040000
> > 
> > "no VLPI support" is very suspicious, it looks like Dom0 doesn't find any 
> > ITS
> > support.
> > 
> > Can you double check that you have the ITS driver in Xen built-in? That 
> > would
> > be CONFIG_HAS_ITS. If you do "make menuconfig" and enable "Configure
> > standard Xen features (expert users)" you should get a new option "GICv3
> > ITS MSI controller support" under "Architecture Features".
> > Make sure to enable it.
> > 
> > Let me know if that works!
>

Attachment: devicetree.dts
Description: Text document