Re: [BUG]Add PCIE devie to SMMUv3 fail

[sisyphean <sisyphean@zlw.email>]

Hi,

在 2022/12/13 00:30, Rahul Singh 写道:
> Hi Sisyphean,
>
> > On 12 Dec 2022, at 5:49 am, sisyphean <sisyphean@zlw.email> wrote:
> >
> > Hi,
> > 在 2022/12/9 17:50, Rahul Singh 写道:
> > > Hi Sisyphean,
> > >
> > > > On 9 Dec 2022, at 6:15 am, sisyphean <sisyphean@zlw.email> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I try to run XEN on my ARM board(Sorry, for some commercial reasons, I can't 
> > > > tell you
> > > > on which platform I run XEN)  and add PCIE device passthrough to DomU.But an 
> > > > error
> > > > occurred while adding the PCIE device to SMMUv3.
> > > PCI passthrough support is not fully upstream to Xen on ARM. We have working
> > > PCI passthrough branch that you can use to test it.
> > >
> > > https://gitlab.com/xen-project/fusa/xen-integration/-/commits/integration/pci-passthrough
> > >
> > > > Through reading the code and tracing debugging, the error is found in the 
> > > > function
> > > > arm_smmu_add_device, which will obtain and determine whether the fwspec of the
> > > > device to be added to SMMU exists.But for the XEN of arm, the fwspec of the 
> > > > device is
> > > > created and added by judging whether the iommu field exists in the device node 
> > > > when
> > > > XEN parses the device tree.However, the PCIE device does not appear in the 
> > > > device tree,
> > > > so there will be no fwspec for all PCIE devices. When attempting to add a PCIE 
> > > > device to
> > > > SMMU, a ENODEV error will be returned.
> > > As of now Xen doesn’t support to add PCI device to IOMMU on ARM.
> > > > In addition, the code at xen/drivers/passthrough/pci.c also verifies the above 
> > > > view.
> > > > For PCIE devices, pdev is alloc in function pci_add_device by 
> > > > alloc_pdev.However,
> > > > the function alloc_pdev does not create and add fwspec to the PCIE 
> > > > device.Therefore,
> > > > when function pci_add_device executes to iommu_add_device,it will get the error
> > > > return of ENODEV.
> > > >
> > > > How can I resolve the above errors?
> > > If you want to test the PCI passthrough please follow below steps.
> > >
> > > Xen setup:
> > >      • A checkout of the “integration/pci-passthrough” branch from the  gitlab 
> > > https://gitlab.com/xen-project/fusa/xen-integration/-/commits/integration/pci-passthrough
> > >      • Pass iommu=yes  and pci-passthrough=on to Xen command line to enable PCI 
> > > passthrough.
> > >
> > >   Linux Kernel setup:
> > >   
> > >      • Some changes are required for the kernel to work with PCI passthrough. First are some configuration options, enable them in kernel config.
> > >           CONFIG_XEN=y
> > >           CONFIG_XEN_BACKEND=y
> > >           CONFIG_XEN_PCIDEV_BACKEND=y
> > >      • Then a patch needs to be applied for enabling the pciback driver. Patch 
> > > is attached in this email.
> > >   
> > > Using PCI passthrough:
> > >   
> > >      • In order to pass a device to a guest, you first need its PCI address(SBDF). You can either get it from a bare-metal
> > >        Linux running on the platform or by having pciutils  installed (if you 
> > > are using a yocto-based dom0 or have apt available), which provides lspci.
> > >   
> > >       For example, let's pass one ethernet interface to the guest. Running lspci gives us this output (truncated) :
> > >         0000:00:00.0 Host bridge: Ampere Computing, LLC Device e100
> > >         0000:00:01.0 PCI bridge: Ampere Computing, LLC Device e101 (rev 04)
> > >         0000:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 
> > > X710/X557-AT 10GBASE-T (rev 01)
> > >                 [...]
> > >   
> > >       We will pass one of the ethernet from the PCI network card : 0000:01:00.0 .
> > >   
> > >      • Add the following line to the guest configuration file :
> > >            pci = ['0000:01:00.0']
> > >   
> > >      • Run the following command before starting the guest :
> > >             xl pci-assignable-add 0000:01:00.0
> > >      • Start the guest. The network interface should appear as 00:00.0  in the 
> > > guest and be usable.
> > >   
> > > Please let me know if you need more info.
> > >
> > >
> > > Regards,
> > > Rahul
> > Thank you for your reply.
> > After setting XEN and kernel as above, I tried the following two methods to add 
> > a PCIE device passthrough:
> > 1. According to your suggestion, use the command xl pci-assignable-add 
> > 0002:21:00.0 to set in the Dom0. But in function
> > iommu_do_pci_domctl,  after device_assigned is called, ENODEV error is obtained.
> > 2. Add xen-pciback.hide=(0002:21:00.0) to dom0-bootargs in the device tree, I 
> > encountered the same problem as before
> > when initializing the kernel. In function pci_add_device, PCIE devices cannot 
> > be added to SMMUv3.
> It is hard to find what is happening without logs. Could you please share the 
> Xen and Linux boot logs so that I can
> check what is the root cause of this issue.
>
> > The kernel version I use is 5.10. Does this have an impact?
> I am using the Linux version 5.15.44 but I don’t think is because of Linux.
>
> Regards,
> Rahul
The attachment is a log file. I have to delete some content that may 
involve commercial. Please forgive me.
In addition, I have forgotten to tell you a very important information. 
The PCIE controller used on my board
is DesignWare. I referred to the code of ECAM under XEN and added some 
support related to DesignWare
(DBI space mapping and PCIE read/write).

In addition, if needed ,I can pre initialized PCIE controller in the 
uboot stage, so  I can scan the PCIE device
in the uboot command line.

Cheers,

Attachment: xen.log
Description: Text Data

Attachment: kernel.log
Description: Text Data