Re: [BUG]Add PCIE devie to SMMUv3 fail

[Rahul Singh <Rahul.Singh@xxxxxxx>]

Hi Sisphean,

> On 13 Dec 2022, at 1:18 am, sisyphean <sisyphean@zlw.email> wrote:
> 
> Hi,
> 
> 在 2022/12/13 00:30, Rahul Singh 写道:
>> Hi Sisyphean,
>> 
>>> On 12 Dec 2022, at 5:49 am, sisyphean <sisyphean@zlw.email> wrote:
>>> 
>>> Hi,
>>> 在 2022/12/9 17:50, Rahul Singh 写道:
>>>> Hi Sisyphean,
>>>> 
>>>>> On 9 Dec 2022, at 6:15 am, sisyphean <sisyphean@zlw.email> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> I try to run XEN on my ARM board(Sorry, for some commercial reasons, I 
>>>>> can't tell you
>>>>> on which platform I run XEN)  and add PCIE device passthrough to DomU.But 
>>>>> an error
>>>>> occurred while adding the PCIE device to SMMUv3.
>>>> PCI passthrough support is not fully upstream to Xen on ARM. We have 
>>>> working
>>>> PCI passthrough branch that you can use to test it.
>>>> 
>>>> https://gitlab.com/xen-project/fusa/xen-integration/-/commits/integration/pci-passthrough
>>>> 
>>>>> Through reading the code and tracing debugging, the error is found in the 
>>>>> function
>>>>> arm_smmu_add_device, which will obtain and determine whether the fwspec 
>>>>> of the
>>>>> device to be added to SMMU exists.But for the XEN of arm, the fwspec of 
>>>>> the device is
>>>>> created and added by judging whether the iommu field exists in the device 
>>>>> node when
>>>>> XEN parses the device tree.However, the PCIE device does not appear in 
>>>>> the device tree,
>>>>> so there will be no fwspec for all PCIE devices. When attempting to add a 
>>>>> PCIE device to
>>>>> SMMU, a ENODEV error will be returned.
>>>> As of now Xen doesn’t support to add PCI device to IOMMU on ARM.
>>>>> In addition, the code at xen/drivers/passthrough/pci.c also verifies the 
>>>>> above view.
>>>>> For PCIE devices, pdev is alloc in function pci_add_device by 
>>>>> alloc_pdev.However,
>>>>> the function alloc_pdev does not create and add fwspec to the PCIE 
>>>>> device.Therefore,
>>>>> when function pci_add_device executes to iommu_add_device,it will get the 
>>>>> error
>>>>> return of ENODEV.
>>>>> 
>>>>> How can I resolve the above errors?
>>>> If you want to test the PCI passthrough please follow below steps.
>>>> 
>>>> Xen setup:
>>>>     • A checkout of the “integration/pci-passthrough” branch from the  
>>>> gitlab 
>>>> https://gitlab.com/xen-project/fusa/xen-integration/-/commits/integration/pci-passthrough
>>>>     • Pass iommu=yes  and pci-passthrough=on to Xen command line to enable 
>>>> PCI passthrough.
>>>> 
>>>>  Linux Kernel setup:
>>>>       • Some changes are required for the kernel to work with PCI 
>>>> passthrough. First are some configuration options, enable them in kernel 
>>>> config.
>>>>          CONFIG_XEN=y
>>>>          CONFIG_XEN_BACKEND=y
>>>>          CONFIG_XEN_PCIDEV_BACKEND=y
>>>>     • Then a patch needs to be applied for enabling the pciback driver. 
>>>> Patch is attached in this email.
>>>>  Using PCI passthrough:
>>>>       • In order to pass a device to a guest, you first need its PCI 
>>>> address(SBDF). You can either get it from a bare-metal
>>>>       Linux running on the platform or by having pciutils  installed (if 
>>>> you are using a yocto-based dom0 or have apt available), which provides 
>>>> lspci.
>>>>        For example, let's pass one ethernet interface to the guest. 
>>>> Running lspci gives us this output (truncated) :
>>>>        0000:00:00.0 Host bridge: Ampere Computing, LLC Device e100
>>>>        0000:00:01.0 PCI bridge: Ampere Computing, LLC Device e101 (rev 04)
>>>>        0000:01:00.0 Ethernet controller: Intel Corporation Ethernet 
>>>> Controller X710/X557-AT 10GBASE-T (rev 01)
>>>>                [...]
>>>>        We will pass one of the ethernet from the PCI network card : 
>>>> 0000:01:00.0 .
>>>>       • Add the following line to the guest configuration file :
>>>>           pci = ['0000:01:00.0']
>>>>       • Run the following command before starting the guest :
>>>>            xl pci-assignable-add 0000:01:00.0
>>>>     • Start the guest. The network interface should appear as 00:00.0  in 
>>>> the guest and be usable.
>>>>  Please let me know if you need more info.
>>>> 
>>>> 
>>>> 
>>>> Regards,
>>>> Rahul
>>>> 
>>> Thank you for your reply.
>>> After setting XEN and kernel as above, I tried the following two methods to 
>>> add a PCIE device passthrough:
>>> 1. According to your suggestion, use the command xl pci-assignable-add 
>>> 0002:21:00.0 to set in the Dom0. But in function
>>> iommu_do_pci_domctl,  after device_assigned is called, ENODEV error is 
>>> obtained.
>>> 2. Add xen-pciback.hide=(0002:21:00.0) to dom0-bootargs in the device tree, 
>>> I encountered the same problem as before
>>> when initializing the kernel. In function pci_add_device, PCIE devices 
>>> cannot be added to SMMUv3.
>> It is hard to find what is happening without logs. Could you please share 
>> the Xen and Linux boot logs so that I can
>> check what is the root cause of this issue.
>> 
>>> The kernel version I use is 5.10. Does this have an impact?
>> I am using the Linux version 5.15.44 but I don’t think is because of Linux.
>> 
>> Regards,
>> Rahul
> 
> The attachment is a log file. I have to delete some content that may involve 
> commercial. Please forgive me.
> 
> In addition, I have forgotten to tell you a very important information. The 
> PCIE controller used on my board
> is DesignWare. I referred to the code of ECAM under XEN and added some 
> support related to DesignWare
> (DBI space mapping and PCIE read/write).
> 

What I understood from the logs is you shared the logs for domU boot. Could you 
please confirm that dom0 booted
fine with the PCI passthrough branch and the PCI device you trying to assign to 
domU is working fine in dom0.

In that case we can confirm that code you added to support DesignWare PCIE 
controller is fine.

> In addition, if needed ,I can pre initialized PCIE controller in the uboot 
> stage, so  I can scan the PCIE device
> in the uboot command line.

Yes, we can try that and use the below branch to check if it works. Below 
branch supports to add PCI device
 during Xen boot as compared to previous branch where Linux inform Xen to add 
PCI devices.

You need to add "pci-scan=on” in Xen command line to add PCI devices during Xen 
boot.

https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-/tree/poc/pci-passthrough
 
 

Regards,
Rahul