AW: AW: AW: AW: AW: Xen data from meta-virtualization layer

[Leo Krueger <leo.krueger@xxxxxxxx>]

Hi Julien,

finally I could try out what you suggested, please find my answers inline.

> -----Ursprüngliche Nachricht-----
> Von: Julien Grall <julien@xxxxxxx>
> Gesendet: Mittwoch, 18. November 2020 13:24
> An: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; Leo Krueger
> <leo.krueger@xxxxxxxx>
> Cc: Peng Fan <peng.fan@xxxxxxx>; brucea@xxxxxxxxxx; Cornelia Bruelhart
> <cornelia.bruelhart@xxxxxxxx>; oleksandr_andrushchenko@xxxxxxxx; xen-
> devel@xxxxxxxxxxxxxxxxxxxx; Bertrand.Marquis@xxxxxxx
> Betreff: Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
> 
> Hi,
> 
> On 17/11/2020 23:53, Stefano Stabellini wrote:
> > Adding Bertrand, Oleksandr, Julien, and others -- they have a more
> > recent experience with GICv3 ITS than me and might be able to help.
> > I am attaching the device tree Leo sent a few days ago for reference.
> >
> >
> > Typically when you can set the ethernet link up and no packets are
> > exchanged it is because of a missing interrupt. In this case a missing
> > MSI.
> >
> > Bertrand, I believe you tried the GIC ITS driver with PCI devices
> > recently. It is expected to work correctly with MSIs in Dom0, right?
> 
> OSSTest has some hardware (e.g. Thunder-X) where ITS is required to boot
> Dom0. I haven't seen any failure on recent Xen. We are testing 4.11 and
> onwards on Thunder-X.
> 
> However, it may be possible that some more work is necessary for other
> hardware (e.g. workaround, missing code...). See more below.
> 
> >
> >
> >
> > On Tue, 17 Nov 2020, Leo Krueger wrote:
> >> Hi,
> >>
> >> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
> >> before...) but then had to add the following node to my device tree
> >>
> >>    gic_lpi_base: syscon@0x80000000 {
> >>            compatible = "gic-lpi-base";
> 
> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo, could
> you clarify which flavor/version of Linux you are using?

It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
While searching around the Internet for any solution, I came across [0] which 
contained the gic-lpi-base node.
So I just tried adding it (quite desperate I know) and voila, it at least 
brought me one step further (XEN exposing the ITS)...

> 
> >>            reg = <0x0 0x80000000 0x0 0x100000>;
> >>            max-gic-redistributors = <2>;
> >>    };
> >>
> >> to somehow change something in regard to the ITS and MSI/MSI-X
> >>
> >> (XEN) GICv3 initialization:
> >> (XEN)       gic_dist_addr=0x00000006000000
> >> (XEN)       gic_maintenance_irq=25
> >> (XEN)       gic_rdist_stride=0
> >> (XEN)       gic_rdist_regions=1
> >> (XEN)       redistributor regions:
> >> (XEN)         - region 0: 0x00000006040000 - 0x00000006080000
> >> (XEN) GICv3: using at most 57344 LPIs on the host.
> >> (XEN) GICv3: 288 lines, (IID 0001143b).
> >> (XEN) GICv3: Found ITS @0x6020000
> >> (XEN) using non-cacheable ITS command queue
> >> (XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000
> >>
> >> [    0.000000] GICv3: Distributor has no Range Selector support
> >> [    0.000000] GICv3: no VLPI support, no direct LPI support
> >> [    0.000000] ITS [mem 0x06020000-0x0603ffff]
> >> [    0.000000] ITS@0x0000000006020000: allocated 65536 Devices
> @dc880000 (flat, esz 8, psz 64K, shr 1)
> >> [    0.000000] ITS@0x0000000006020000: allocated 32768 Interrupt
> Collections @dc820000 (flat, esz 2, psz 64K, shr 1)
> >> [    0.000000] GIC: using LPI property table @0x00000000dc830000
> >> [    0.000000] GICv3: CPU0: found redistributor 0 region
> 0:0x0000000006040000
> >> [    0.000000] CPU0: using LPI pending table @0x00000000dc840000
> >> ...
> >> [    0.040080] Platform MSI: gic-its domain created
> >> [    0.040136] PCI/MSI: /interrupt-controller/gic-its domain created
> >> [    0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created
> >>
> >>
> >> Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X
> might fail!" log messages for some PCI devices, but at least the on-board
> ethernet ports (fsl_enetc ) are initialized.
> >> I can set the link up and a link is successfully established.
> 
> This message is normal. Xen on Arm is not yet aware of PCI devices and
> therefore the hypercalls to add PCI devices will return -EOPNOTSUPP.
> 
> However, this is not an issue because the virtual ITS implementation will
> allow dom0 to configure any devices.
> 
> >>
> >> But (!) I cannot receive or transmit anything (no error message...) and
> there seem to be no interrupts:
> >>
> >> 29:          0   ITS-MSI   1 Edge      gbe0-rxtx0
> >>   32:          0   ITS-MSI 8192 Edge      ptp_qoriq
> >>
> >> (from /proc/interrupts).
> >>
> >> Any idea on this one? I keep digging and checking whether my device tree
> needs some additional fixes.
> 
> Can you apply patch [1] and provide the logs? This will dump more
> information about the command received by the vITS and the one send to
> the host ITS.

For XEN 4.13.2 I had to adapt your patch slightly [1], see below (yes I know, 
quite ugly in parts).
Find attached the boot log and an output of "xl dmesg" which is truncated due 
to the large amount of messages.

When enabling the network interface (gbe0), the following output is visible:

root@kontron-sal28:~# ip link set up dev gbe0
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0c: 000000170000000c 0000000000000001 
0000000000000000 0000000000000000
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005 0000000000000000 
0000000000000000 0000000000000000
[   34.034598] Atheros 8031 ethernet 0000:00:00.3:05: attached PHY driver 
[Atheros 8031 ethernet] (mii_bus:phy_addr=0000:00:00.3:05, irq=POLL)
[   34.041111] 8021q: adding VLAN 0 to HW filter on device gbe0
[   34.041209] IPv6: ADDRCONF(NETDEV_UP): gbe0: link is not ready
root@kontron-sal28:~# [   35.041951] fsl_enetc 0000:00:00.0 gbe0: Link is Down
[   38.114426] fsl_enetc 0000:00:00.0 gbe0: Link is Up - 1Gbps/Full - flow 
control off
[   38.114508] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes ready

Does that tell you anything?

> 
> Note that Xen will need to be build with CONFIG_DEBUG=y in order to get
> some of the messages.
> 
> [...]
> 
> >>>> [    0.000000] GICv3: Distributor has no Range Selector support
> >>>>
> >>>> [    0.000000] GICv3: no VLPI support, no direct LPI support
> >>>>
> >>>> [    0.000000] GICv3: CPU0: found redistributor 0 region
> >>>> 0:0x0000000006040000
> >>>
> >>> "no VLPI support" is very suspicious, it looks like Dom0 doesn't
> >>> find any ITS support.
> VLPI is a feature that was introduced in GICv4 to directly inject LPI in the
> guest. So this is normal to see this message when running on Xen because
> we are going to only expose a GICv3 to a domain until at least we support
> nested virt.
> 
> However, you were right about that Xen didn't expose the ITS because the
> following lines were missing:
> 
> [    0.000000] ITS@0x0000000006020000: allocated 65536 Devices @dc880000
> (flat, esz 8, psz 64K, shr 1)
> 
> Cheers,

Best regards,
Leo

> 
> [1]
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c index
> 9558bad96ac3..8a0a02308e74 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -87,6 +87,10 @@ static int its_send_command(struct host_its *hw_its,
> const void *its_cmd)
>       /* No ITS commands from an interrupt handler (at the moment). */
>       ASSERT(!in_irq());
> 
> +    printk(XENLOG_WARNING, "pITS  cmd 0x%02lx: %016lx %016lx %016lx
> %016lx\n",
> +           its_cmd_get_command(command),
> +           command[0], command[1], command[2], command[3]);
> +
>       spin_lock(&hw_its->cmd_lock);
> 
>       do {
> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c index
> 869bc97fa1aa..e7c5bcd8d423 100644
> --- a/xen/arch/arm/gic-v3-lpi.c
> +++ b/xen/arch/arm/gic-v3-lpi.c
> @@ -183,7 +183,10 @@ void gicv3_do_LPI(unsigned int lpi)
>       /* Find out if a guest mapped something to this physical LPI. */
>       hlpip = gic_get_host_lpi(lpi);
>       if ( !hlpip )
> +    {
> +        printk("%s: Received LPI %u but it is not mapped?\n", __func__,
> lpi);
>           goto out;
> +    }
> 
>       hlpi.data = read_u64_atomic(&hlpip->data);
> 
> @@ -222,6 +225,9 @@ void gicv3_lpi_update_host_entry(uint32_t host_lpi,
> int domain_id,
>   {
>       union host_lpi *hlpip, hlpi;
> 
> +    printk("%s: host_lpi %u domain %d virq_lpi %u\n",
> +           __func__, host_lpi, domain_id, virq_lpi);
> +
>       ASSERT(host_lpi >= LPI_OFFSET);
> 
>       host_lpi -= LPI_OFFSET;
> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c index
> 58d939b85f92..89ef137b3e6b 100644
> --- a/xen/arch/arm/vgic-v3-its.c
> +++ b/xen/arch/arm/vgic-v3-its.c
> @@ -897,7 +897,7 @@ out_unlock:
> 
>   static void dump_its_command(uint64_t *command)
>   {
> -    gdprintk(XENLOG_WARNING, "  cmd 0x%02lx: %016lx %016lx %016lx
> %016lx\n",
> +    gdprintk(XENLOG_WARNING, "vITS  cmd 0x%02lx: %016lx %016lx %016lx
> %016lx\n",
>                its_cmd_get_command(command),
>                command[0], command[1], command[2], command[3]);
>   }
> @@ -926,6 +926,8 @@ static int vgic_its_handle_cmds(struct domain *d,
> struct virt_its *its)
>           if ( ret )
>               return ret;
> 
> +        dump_its_command(command);
> +
>           switch ( its_cmd_get_command(command) )
>           {
>           case GITS_CMD_CLEAR:
> 
> 
> --
> Julien Grall

[0] https://www.mail-archive.com/u-boot@xxxxxxxxxxxxx/msg379708.html
[1] 
diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 9558bad96a..d175ba52b0 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -87,6 +87,10 @@ static int its_send_command(struct host_its *hw_its, const 
void *its_cmd)
     /* No ITS commands from an interrupt handler (at the moment). */
     ASSERT(!in_irq());

+    printk(XENLOG_WARNING "pITS  cmd 0x%02lx: %016lx %016lx %016lx %016lx\n",
+        (((uint64_t *) its_cmd)[0] >> 0) & GENMASK(8 - 1, 0),
+        ((uint64_t *) its_cmd)[0], ((uint64_t *) its_cmd)[1], ((uint64_t *) 
its_cmd)[2], ((uint64_t *) its_cmd)[3]);
+
     spin_lock(&hw_its->cmd_lock);

     do {
diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
index 78b9521b21..2c3b0fc9e5 100644
--- a/xen/arch/arm/gic-v3-lpi.c
+++ b/xen/arch/arm/gic-v3-lpi.c
@@ -181,8 +181,10 @@ void gicv3_do_LPI(unsigned int lpi)

     /* Find out if a guest mapped something to this physical LPI. */
     hlpip = gic_get_host_lpi(lpi);
-    if ( !hlpip )
+    if ( !hlpip ) {
+        printk("%s: Received LPI %u but it is not mapped?\n", __func__, lpi);
         goto out;
+    }

     hlpi.data = read_u64_atomic(&hlpip->data);

@@ -221,6 +223,9 @@ void gicv3_lpi_update_host_entry(uint32_t host_lpi, int 
domain_id,
 {
     union host_lpi *hlpip, hlpi;

+    printk("%s: host_lpi %u domain %d virt_lpi %u\n",
+        __func__, host_lpi, domain_id, virt_lpi);
+
     ASSERT(host_lpi >= LPI_OFFSET);

     host_lpi -= LPI_OFFSET;
diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index 6e153c698d..dd5081ef80 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -897,7 +897,7 @@ out_unlock:

 static void dump_its_command(uint64_t *command)
 {
-    gdprintk(XENLOG_WARNING, "  cmd 0x%02lx: %016lx %016lx %016lx %016lx\n",
+    gdprintk(XENLOG_WARNING, "vITS  cmd 0x%02lx: %016lx %016lx %016lx 
%016lx\n",
              its_cmd_get_command(command),
              command[0], command[1], command[2], command[3]);
 }
@@ -926,6 +926,8 @@ static int vgic_its_handle_cmds(struct domain *d, struct 
virt_its *its)
         if ( ret )
             return ret;

+        dump_its_command(command);
+
         switch ( its_cmd_get_command(command) )
         {
         case GITS_CMD_CLEAR:

Attachment: boot-xendebug.log
Description: boot-xendebug.log

Attachment: xen-debug-output.txt
Description: xen-debug-output.txt