Re: [BUG]Add PCIE devie to SMMUv3 fail

[sisyphean <sisyphean@zlw.email>]

在 2022/12/15 09:30, sisyphean 写道:
> Hi,
>
> 在 2022/12/14 21:05, Rahul Singh 写道:
> > Hi Sisphean,
> >
> > > On 13 Dec 2022, at 1:18 am, sisyphean <sisyphean@zlw.email> wrote:
> > >
> > > Hi,
> > >
> > > 在 2022/12/13 00:30, Rahul Singh 写道:
> > > > Hi Sisyphean,
> > > >
> > > > > On 12 Dec 2022, at 5:49 am, sisyphean <sisyphean@zlw.email> wrote:
> > > > >
> > > > > Hi,
> > > > > 在 2022/12/9 17:50, Rahul Singh 写道:
> > > > > > Hi Sisyphean,
> > > > > >
> > > > > > > On 9 Dec 2022, at 6:15 am, sisyphean <sisyphean@zlw.email> wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I try to run XEN on my ARM board(Sorry, for some commercial 
> > > > > > > reasons, I can't tell you
> > > > > > > on which platform I run XEN)  and add PCIE device passthrough to 
> > > > > > > DomU.But an error
> > > > > > > occurred while adding the PCIE device to SMMUv3.
> > > > > > PCI passthrough support is not fully upstream to Xen on ARM. We 
> > > > > > have working
> > > > > > PCI passthrough branch that you can use to test it.
> > > > > >
> > > > > > https://gitlab.com/xen-project/fusa/xen-integration/-/commits/integration/pci-passthrough 
> > > > > >
> > > > > > > Through reading the code and tracing debugging, the error is 
> > > > > > > found in the function
> > > > > > > arm_smmu_add_device, which will obtain and determine whether the 
> > > > > > > fwspec of the
> > > > > > > device to be added to SMMU exists.But for the XEN of arm, the 
> > > > > > > fwspec of the device is
> > > > > > > created and added by judging whether the iommu field exists in 
> > > > > > > the device node when
> > > > > > > XEN parses the device tree.However, the PCIE device does not 
> > > > > > > appear in the device tree,
> > > > > > > so there will be no fwspec for all PCIE devices. When attempting 
> > > > > > > to add a PCIE device to
> > > > > > > SMMU, a ENODEV error will be returned.
> > > > > > As of now Xen doesn’t support to add PCI device to IOMMU on ARM.
> > > > > > > In addition, the code at xen/drivers/passthrough/pci.c also 
> > > > > > > verifies the above view.
> > > > > > > For PCIE devices, pdev is alloc in function pci_add_device by 
> > > > > > > alloc_pdev.However,
> > > > > > > the function alloc_pdev does not create and add fwspec to the 
> > > > > > > PCIE device.Therefore,
> > > > > > > when function pci_add_device executes to iommu_add_device,it 
> > > > > > > will get the error
> > > > > > > return of ENODEV.
> > > > > > >
> > > > > > > How can I resolve the above errors?
> > > > > > If you want to test the PCI passthrough please follow below steps.
> > > > > >
> > > > > > Xen setup:
> > > > > >      • A checkout of the “integration/pci-passthrough” branch 
> > > > > > from the  gitlab 
> > > > > > https://gitlab.com/xen-project/fusa/xen-integration/-/commits/integration/pci-passthrough
> > > > > >      • Pass iommu=yes  and pci-passthrough=on to Xen command line 
> > > > > > to enable PCI passthrough.
> > > > > >   Linux Kernel setup:
> > > > > >        • Some changes are required for the kernel to work with 
> > > > > > PCI passthrough. First are some configuration options, enable 
> > > > > > them in kernel config.
> > > > > >           CONFIG_XEN=y
> > > > > >           CONFIG_XEN_BACKEND=y
> > > > > >           CONFIG_XEN_PCIDEV_BACKEND=y
> > > > > >      • Then a patch needs to be applied for enabling the pciback 
> > > > > > driver. Patch is attached in this email.
> > > > > >   Using PCI passthrough:
> > > > > >        • In order to pass a device to a guest, you first need its 
> > > > > > PCI address(SBDF). You can either get it from a bare-metal
> > > > > >        Linux running on the platform or by having pciutils  
> > > > > > installed (if you are using a yocto-based dom0 or have apt 
> > > > > > available), which provides lspci.
> > > > > >         For example, let's pass one ethernet interface to the 
> > > > > > guest. Running lspci gives us this output (truncated) :
> > > > > >         0000:00:00.0 Host bridge: Ampere Computing, LLC Device e100
> > > > > >         0000:00:01.0 PCI bridge: Ampere Computing, LLC Device 
> > > > > > e101 (rev 04)
> > > > > >         0000:01:00.0 Ethernet controller: Intel Corporation 
> > > > > > Ethernet Controller X710/X557-AT 10GBASE-T (rev 01)
> > > > > >                 [...]
> > > > > >         We will pass one of the ethernet from the PCI network 
> > > > > > card : 0000:01:00.0 .
> > > > > >        • Add the following line to the guest configuration file :
> > > > > >            pci = ['0000:01:00.0']
> > > > > >        • Run the following command before starting the guest :
> > > > > >             xl pci-assignable-add 0000:01:00.0
> > > > > >      • Start the guest. The network interface should appear as 
> > > > > > 00:00.0  in the guest and be usable.
> > > > > >   Please let me know if you need more info.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Rahul
> > > > > Thank you for your reply.
> > > > > After setting XEN and kernel as above, I tried the following two 
> > > > > methods to add a PCIE device passthrough:
> > > > > 1. According to your suggestion, use the command xl 
> > > > > pci-assignable-add 0002:21:00.0 to set in the Dom0. But in function
> > > > > iommu_do_pci_domctl,  after device_assigned is called, ENODEV 
> > > > > error is obtained.
> > > > > 2. Add xen-pciback.hide=(0002:21:00.0) to dom0-bootargs in the 
> > > > > device tree, I encountered the same problem as before
> > > > > when initializing the kernel. In function pci_add_device, PCIE 
> > > > > devices cannot be added to SMMUv3.
> > > > It is hard to find what is happening without logs. Could you please 
> > > > share the Xen and Linux boot logs so that I can
> > > > check what is the root cause of this issue.
> > > >
> > > > > The kernel version I use is 5.10. Does this have an impact?
> > > > I am using the Linux version 5.15.44 but I don’t think is because 
> > > > of Linux.
> > > > Regards,
> > > > Rahul
> > > The attachment is a log file. I have to delete some content that may 
> > > involve commercial. Please forgive me.
> > > In addition, I have forgotten to tell you a very important 
> > > information. The PCIE controller used on my board
> > > is DesignWare. I referred to the code of ECAM under XEN and added 
> > > some support related to DesignWare
> > > (DBI space mapping and PCIE read/write).
> The kernel.log file is from Dom0 boot. The PCIE device I trying to 
> assign to domU is working fine in dom0.
> > What I understood from the logs is you shared the logs for domU boot. 
> > Could you please confirm that dom0 booted
> > fine with the PCI passthrough branch and the PCI device you trying to 
> > assign to domU is working fine in dom0.
> > In that case we can confirm that code you added to support DesignWare 
> > PCIE controller is fine.
> > > In addition, if needed ,I can pre initialized PCIE controller in the 
> > > uboot stage, so  I can scan the PCIE device
> > > in the uboot command line.
>
> In order to verify my previous conjecture (there is no fwspec for the 
> PCIE device,
> which makes it impossible to passthrough), I used the following code 
> to test.
> The result is that the PCIE device can passthrough to the DomU, but 
> the DomU kernel
> appears Data Abort when trying to read the PCIE device. This may be 
> because I directly add
> the fwspec of the PCIE controller to the PCIE device, which leads to 
> memory mapping problems.
> diff --git a/xen/drivers/passthrough/device_tree.c 
> b/xen/drivers/passthrough/device_tree.c
> index 98f2aa0..28283d2 100644
> --- a/xen/drivers/passthrough/device_tree.c
> +++ b/xen/drivers/passthrough/device_tree.c
> @@ -127,6 +127,7 @@ int iommu_release_dt_devices(struct domain *d)
>      return 0;
>  }
>
> +struct device *smmu_dev_for_test;
>  int iommu_add_dt_device(struct dt_device_node *np)
>  {
>      const struct iommu_ops *ops = iommu_get_ops();
> @@ -167,7 +168,8 @@ int iommu_add_dt_device(struct dt_device_node *np)
>          rc = iommu_fwspec_init(dev, &iommu_spec.np->dev);
>          if ( rc )
>              break;
> -
> +        else
> +            smmu_dev_for_test = &iommu_spec.np->dev;
>          /*
>           * Provide DT IOMMU specifier which describes the IOMMU master
>           * interfaces of that device (device IDs, etc) to the driver.
>
>
> diff --git a/xen/drivers/passthrough/pci.c 
> b/xen/drivers/passthrough/pci.c
> index da4ecda..cf8a099 100644
> --- a/xen/drivers/passthrough/pci.c
> +++ b/xen/drivers/passthrough/pci.c
> @@ -1316,6 +1316,8 @@ static int __init setup_dump_pcidevs(void)
>  }
>  __initcall(setup_dump_pcidevs);
>
> +extern int iommu_fwspec_init(struct device *dev, struct device 
> *iommu_dev);
> +extern struct device *smmu_dev_for_test;;
>  static int iommu_add_device(struct pci_dev *pdev)
>  {
>      const struct domain_iommu *hd;
> @@ -1331,6 +1333,10 @@ static int iommu_add_device(struct pci_dev *pdev)
>      if ( !is_iommu_enabled(pdev->domain) )
>          return 0;
>
> +    if (pdev->bus != 0x20)
> +    {
> +        iommu_fwspec_init(pci_to_dev(pdev), smmu_dev_for_test);
> +    }
>      rc = hd->platform_ops->add_device(pdev->devfn, pci_to_dev(pdev));
>      if ( rc || !pdev->phantom_stride )
>          return rc;
> @@ -1401,6 +1407,7 @@ static int device_assigned(u16 seg, u8 bus, u8 
> devfn)
>      ASSERT(pcidevs_locked());
>      pdev = pci_get_pdev(seg, bus, devfn);
> +    printk("%s:%p,%p,%p\n",__FUNCTION__, pdev->domain, 
> hardware_domain, dom_io);
>      if ( !pdev )
>          rc = -ENODEV;
The pci-passthrough  branch is xen-4.17-unstable, the xen tools does not 
work properly.
I can't use xl create command to create DomU. After the command is 
executed, it seems
to be suspended, and the kernel and terminal have no output.

> I will test the use of pci-passthrough  branch to verify the PCIE 
> passthrough function
> > Yes, we can try that and use the below branch to check if it works. 
> > Below branch supports to add PCI device
> >   during Xen boot as compared to previous branch where Linux inform 
> > Xen to add PCI devices.
> > You need to add "pci-scan=on” in Xen command line to add PCI devices 
> > during Xen boot.
> > https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-/tree/poc/pci-passthrough 
> >
> > Regards,
> > Rahul
> Cheers,
> --
> Sisyphean