Re: [Xen-devel] [RFC PATCH v2 00/26] arm64: Dom0 ITS emulation

[Andre Przywara <andre.przywara@xxxxxxx>]

Hi Vijay,

On 19/01/17 12:26, Vijay Kilari wrote:
> Hi Andre,
> 
>    I see following issues when running on ThunderX platform with your patches.
> I have debugged and patched/workaround few issues. For issue (5) I
> need your  inputs.

thanks for the testing and the input. It seems that my anticipation of
issues when booting on hardware were right ;-)
It seems that some of the issues got fixed already by me rewriting some
parts, triggered by comments from Stefano.

> 1) Your code base fails to boot xen. Fails at dom0 memory allocation.
>     To overcome this I have rebased your patches on top of 4.8 stable
> release and this issue is not seen.

Can you try to rebase on latest xen master instead and see if that's
fixed there by any chance? Because that's where we eventually need to
base against anyway. 4.8 stable will not help.

> 2)  ITS is not initialized if GICv2 info is not found in GICv3 dt
> node. But having GICv2 info is not
>    mandatory. So in the below code if GICv2 info is not found ITS is
> not initialized.

Ah, right, good point. The model is v2 compatible.
As this looks like an issue independent from the ITS emulation, can you
make a patch? Or does Xen indeed rely on having v2 compat support at the
moment? I think you should be able to boot Dom0 with an initrd on
ThunderX already, can't you?

> static void __init gicv3_dt_init (void)
> {
>    ...
>     /*
>      * For GICv3 supporting GICv2, GICC and GICV base address will be
>      * provided.
>      */
>     res = dt_device_get_address(node, 1 + gicv3.rdist_count,
>                                 &cbase, &csize);
> 
> -    if ( res )    // <==  This check needs to be managed properly?
> -        return;
> 
>     dt_device_get_address(node, 1 + gicv3.rdist_count + 2,
>                           &vbase, &vsize);
> 
>     /* Check for ITS child nodes and build the host ITS list accordingly. */
>     gicv3_its_dt_init(node);
> 
> }
> 
> 3) Dom0 is not calling PHYSDEVOP_manage_pci_add for all the pci devices
>    instead it is making PHYSDEVOP_pci_device_add when segment exists.
>    So I have made below changes so that ITS mapping is called for all
> the devices.

Ah, OK. I didn't look at the code, just observed the physdevops and
implemented them. Will check this again.

> 
> 4) Also put_domain() is throws panic. So I have dropped it in below code.

Indeed, I found this by code inspection already.

> 
> --- a/xen/arch/arm/physdev.c
> +++ b/xen/arch/arm/physdev.c
> @@ -17,27 +17,68 @@
>  int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>  {
>      struct physdev_manage_pci manage;
> +    struct physdev_pci_device_add pci_add;
>      struct domain *dom0;
>      u32 devid;
>      int ret;
> 
>      switch (cmd)
>      {
>          case PHYSDEVOP_manage_pci_add:
>          case PHYSDEVOP_manage_pci_remove:
>              if ( copy_from_guest(&manage, arg, 1) != 0 )
>                  return -EFAULT;
> 
>              dom0 = hardware_domain;
>              devid = manage.bus << 8 | manage.devfn;
>              /* Allocate an ITS device table with space for 32 MSIs */
>              ret = gicv3_its_map_device(dom0, devid, devid, 5,
>                                         cmd == PHYSDEVOP_manage_pci_add);
> 
> -            put_domain(dom0);
>              return ret;
> +       case PHYSDEVOP_pci_device_add:
> +            if ( copy_from_guest(&pci_add, arg, 1) != 0 )
> +                return -EFAULT;
> +            dom0 = hardware_domain;
> +            devid = pci_add.seg << 16 | pci_add.bus << 8 | pci_add.devfn;
> +
> +            /* Allocate an ITS device table with space for 32 MSIs */
> +            ret = gicv3_its_map_device(dom0, devid, devid, 5,
> +                                       cmd == PHYSDEVOP_pci_device_add);
> +
> +            return ret;
>      }
> 
>      gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
>      return -ENOSYS;
>  }
> 
> 
> 5)  With the above patch, Now I see ITS HW is throwing up Error 10801
> where in it shows
> devid is out of range in MAPD command.
> 
> See below log with some debug prints, where I have printed
> GITS_IMP_CSER register for
> every MAPD command.
> (XEN) GITS_IMP_CSER 0x5e4000010801
> 
> This error is seen when seg is 1. This was not the case with previous
> version of patches.
> However GITS_TYPER from HW shows it support upto 20 bits.
> 
> pci 0000:01:10.1: [177d:a026] type 00 class 0x028000
> pci 0000:01:10.1: BAR 0: [mem 0x87e0e1000000-0x87e0e13fffff 64bit]
> (from Enhanced Allocation, properties 0x0)
> pci 0000:01:10.1: BAR 4: [mem 0x87e0e1400000-0x87e0e17fffff 64bit]
> (from Enhanced Allocation, properties 0x0)
> (XEN) In do_physdev_op PHYSDEVOP_pci_device_add
> (XEN) In do_physdev_op dev id 385 seg 0 bus 1 devfn 129
> (XEN) In gicv3_its_map_device send MAPD for  dev id 385
> (XEN) CMD[0] 0x18100000008 CMD[1] 0x4 CMD [2] 0x8000001ff6044200 CMD[3] 0x0
> (XEN) GITS_IMP_TSER 0x0
> (XEN) GITS_IMP_CSER 0x0
> iommu: Adding device 0000:01:10.1 to group 41
> pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can
> enable it with 'pcie_aspm=force'
> pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
> pci_bus 0000:02: busn_res: can not insert [bus 02-ff] under [bus
> 00-1f] (conflicts with (null) [bus 00-1f])
> pci 0000:02:00.0: [177d:a01d] type 00 class 0x010400
> pci 0000:02:00.0: BAR 0: [mem 0x870000000000-0x8700007fffff 64bit]
> (from Enhanced Allocation, properties 0x0)
> pci 0000:02:00.0: BAR 4: [mem 0x870000f00000-0x870000ffffff 64bit]
> (from Enhanced Allocation, properties 0x0)
> (XEN) In do_physdev_op PHYSDEVOP_pci_device_add
> (XEN) In do_physdev_op dev id 512 seg 0 bus 2 devfn 0
> (XEN) In gicv3_its_map_device send MAPD for  dev id 512
> (XEN) CMD[0] 0x20000000008 CMD[1] 0x4 CMD [2] 0x8000001ff6044400 CMD[3] 0x0
> (XEN) GITS_IMP_TSER 0x0
> (XEN) GITS_IMP_CSER 0x0
> iommu: Adding device 0000:02:00.0 to group 42
> pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device.  You can
> enable it with 'pcie_aspm=force'
> pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
> pci_bus 0000:03: busn_res: can not insert [bus 03-ff] under [bus
> 00-1f] (conflicts with (null) [bus 00-1f])
> pci 0000:03:00.0: [177d:a01a] type 00 class 0x120000
> pci 0000:03:00.0: BAR 0: [mem 0x838000000000-0x8380003fffff 64bit]
> (from Enhanced Allocation, properties 0x0)
> pci 0000:03:00.0: BAR 4: [mem 0x838000f00000-0x838000ffffff 64bit]
> (from Enhanced Allocation, properties 0x0)
> (XEN) In do_physdev_op PHYSDEVOP_pci_device_add
> (XEN) In do_physdev_op dev id 768 seg 0 bus 3 devfn 0
> (XEN) In gicv3_its_map_device send MAPD for  dev id 768
> (XEN) CMD[0] 0x30000000008 CMD[1] 0x4 CMD [2] 0x8000001ff6044500 CMD[3] 0x0
> (XEN) GITS_IMP_TSER 0x0
> (XEN) GITS_IMP_CSER 0x0
> iommu: Adding device 0000:03:00.0 to group 43
> pci 0000:03:00.0: disabling ASPM on pre-1.1 PCIe device.  You can
> enable it with 'pcie_aspm=force'
> pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
> pci_bus 0000:04: busn_res: can not insert [bus 04-ff] under [bus
> 00-1f] (conflicts with (null) [bus 00-1f])
> pci 0000:04:00.0: [177d:a019] type 00 class 0x120000
> pci 0000:04:00.0: BAR 0: [mem 0x846000000000-0x8467ffffffff 64bit]
> (from Enhanced Allocation, properties 0x0)
> pci 0000:04:00.0: BAR 4: [mem 0x846a00000000-0x846a000fffff 64bit]
> (from Enhanced Allocation, properties 0x0)
> (XEN) In do_physdev_op PHYSDEVOP_pci_device_add
> (XEN) In do_physdev_op dev id 1024 seg 0 bus 4 devfn 0
> (XEN) In gicv3_its_map_device send MAPD for  dev id 1024
> (XEN) CMD[0] 0x40000000008 CMD[1] 0x4 CMD [2] 0x8000001ff6044700 CMD[3] 0x0
> (XEN) GITS_IMP_TSER 0x0
> (XEN) GITS_IMP_CSER 0x0
> iommu: Adding device 0000:04:00.0 to group 44
> pci 0000:04:00.0: disabling ASPM on pre-1.1 PCIe device.  You can
> enable it with 'pcie_aspm=force'
> pci_bus 0000:04: busn_res: [bus 04-ff] end is updated to 04
> pci 0000:00:01.0: PCI bridge to [bus 01]
> pci 0000:00:14.0: PCI bridge to [bus 02]
> pci 0000:00:15.0: PCI bridge to [bus 03]
> pci 0000:00:16.0: PCI bridge to [bus 04]
> OF: PCI: host bridge /soc@0/pci@849000000000 ranges:
> OF: PCI:   MEM 0x810000000000..0x817fffffffff -> 0x810000000000
> pci-host-generic 849000000000.pci: ECAM at [mem
> 0x849000000000-0x849001ffffff] for [bus 00-1f]
> pci-host-generic 849000000000.pci: PCI host bridge to bus 0001:00
> pci_bus 0001:00: root bus resource [bus 00-1f]
> pci_bus 0001:00: root bus resource [mem 0x810000000000-0x817fffffffff]
> pci 0001:00:08.0: [177d:a01c] type 00 class 0x010601
> pci 0001:00:08.0: BAR 0: [mem 0x814000000000-0x8140001fffff 64bit]
> (from Enhanced Allocation, properties 0x0)
> pci 0001:00:08.0: BAR 4: [mem 0x814000200000-0x8140002fffff 64bit]
> (from Enhanced Allocation, properties 0x0)
> (XEN) In do_physdev_op PHYSDEVOP_pci_device_add
> (XEN) In do_physdev_op dev id 65600 seg 1 bus 0 devfn 64
> (XEN) In gicv3_its_map_device send MAPD for  dev id 65600
> (XEN) CMD[0] 0x1004000000008 CMD[1] 0x4 CMD [2] 0x8000001ff6044900 CMD[3] 0x0
> (XEN) GITS_IMP_TSER 0x0
> (XEN) GITS_IMP_CSER 0x5e4000010801  // <== MAPD deviceid Out of Range error.
> 
> 6) I have also seen panic in gicv3_assign_guest_event() when called
> from dom0 DISCARD ITS command.
>    Not debugged much.

I think my local fixes here address some of that already. I will send a
new version hopefully later this month, so just stay tuned.

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel