[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit


  • To: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
  • From: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
  • Date: Fri, 14 May 2021 21:56:48 +0000
  • Accept-language: en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=evoleotech.com; dmarc=pass action=none header.from=evoleotech.com; dkim=pass header.d=evoleotech.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=s02SO17Nh348xED6QAliEYXIZEsfW14aV3IERY2bhLI=; b=EjY32/hdyThV2fvicmzOSv5W7yPq5kLv65jJ4NODktzXU8E+n7axLp5sVXZGWPUpy0B8puhi6iGTUP8bvV3v269rRepToBWt7M/nK6hz++weyNBJenYkuZJa2ZiLG0WiFSmpjMJgFgif4MAh20anHU1JxYolkYkPsHr1GL7xY0iQiELXB0DtaFhKaIjO7kf+fDSJ/MhgVyFbo+nSFIUTiiJVYqH5udqALCyT4O4RGwZF8UrtA7FmH6ZKCJQNb/zWrQ0rMkN6WFfnGqfWc2EFG5Ny735E0EUsk1Xhz2rbhMOrW/0sw/uwBqzifEpCpYqjGdagLQsHmYj8v/zlJYhX1A==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SHpGOxhvACm/82+ngd2X1MOGcsYR4we9EMmg04X9BI81Lq3gwdfu44UZSCcpUSGDd/Y6FAyp2GLOCVL9R1wHN49arggreVDDjDtRJm3HttQqSTVosUo58dKt1mI4nN5RjRrgG+iPkC3NWMa/ty1l4e1alNZ4iFJ7MpMa4WUMZxt9B0B0QxuqzbdXd3RkZR0hIHynu94V4ueeQWtOehXGY0emE8/YOW5Mgv+PbVuBXSJlf54Ni/IKtwou6q0pRZB87ITo05e1jpeFIl3p9lmJN0XFL7FaTAxeUFuvQV8F+kTfrS+n1KSFaelugsMfVpA+pzMtakE9NEVG5s0PRMlDVw==
  • Authentication-results: xilinx.com; dkim=none (message not signed) header.d=none;xilinx.com; dmarc=none action=none header.from=evoleotech.com;
  • Cc: Brian Woods <brian.woods@xxxxxxxxxx>, Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>, Julien Grall <julien@xxxxxxx>, Stefano Stabellini <sstabellini@xxxxxxxxxx>, "xen-users@xxxxxxxxxxxxxxxxxxxx" <xen-users@xxxxxxxxxxxxxxxxxxxx>, "fnuv@xxxxxxxxxx" <fnuv@xxxxxxxxxx>, "edgari@xxxxxxxxxx" <edgari@xxxxxxxxxx>, "bwoods288@xxxxxxxxx" <bwoods288@xxxxxxxxx>, Renato Costa Amorim <renato.amorim@xxxxxxxxxxxxxx>, "JOVITAC@xxxxxxxxxx" <JOVITAC@xxxxxxxxxx>
  • Delivery-date: Sat, 15 May 2021 13:49:09 +0000
  • List-id: Xen user discussion <xen-users.lists.xenproject.org>
  • Thread-index: AQHXOqMAPewhx/0ZVE2fbco7T/GOXKrdFmqAgACD+ICABAVYgIABrlVrgABJdgCAAATItw==
  • Thread-topic: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

Hi Stefano, 
Thanks for your reply. Yes the axidma application is a baremetal application from the xilinx vitis ide example. 
Also yes, I have already run hello world as DomU guest, with UART1.

I am using the simple mode using polling and not interrupts(they are disabled in the c code), which is Why I didn't include in the guest config file. This works well when I compile it normally (as EL3). When compiled as hypervisor guest, the execution stops before the function xil_dcacheflushrange(). Please find the c code attached in https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c 

I found the SATA example similar to my use case, also using DMA and SMMU. I just used it as a reference to create my pl.dtb to pass through in the guest config.(from pl.dtsi, which I manually created).

Thanks and Best Regards,
Prem

Get Outlook for Android


From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
Sent: Friday 14 May 2021, 23:16
To: Prem Harikrishnan
Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; xen-users@xxxxxxxxxxxxxxxxxxxx; fnuv@xxxxxxxxxx; edgari@xxxxxxxxxx; bwoods288@xxxxxxxxx; Renato Costa Amorim; JOVITAC@xxxxxxxxxx
Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
a baremetal application? Can you run it successfully, just printing
"hello world", if you only assign the UART and nothing else?

I am asking just to verify that you can run the baremetal VM
successfully, aside from the PL device assignment.


In regards to the PL device assignment, I'll let Brian answer on the
Vivado configuration and SMID calculation. But I noticed that you only
assigned IRQ 54 to the DomU which is the one for UART1. You haven't
assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
looks like there are two related IRQs: 121 and 122.

Finally, why did you mention SATA passthrough in your reply? I don't
follow: I cannot see any SATA controller assignment in your
configuration.


On Fri, 14 May 2021, Prem Harikrishnan wrote:
> Dear Brian, Stefano and Julien,
>
> Thank you very much for your replies.
> Hi Brian and Stefano, Can you please help me with the following questions?
>
> @Brian Woods Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling mode(previously i tried it
> with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only 1 master and
> I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have manually tied
> of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is correct? I can
> update you soon with an ILA trace on Vivado.
>
> @Stefano Stabellini Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually adding xen.dtsi.
> https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using this SATA
> passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in my xen
> configuration.
>
> Thanks and Best Regards,
> Prem
>
> ___________________________________________________________________________________________________________________________________________
> From: Brian Woods <brian.woods@xxxxxxxxxx>
> Sent: Thursday, May 13, 2021 17:13
> To: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> Cc: Julien Grall <julien@xxxxxxx>; Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>; Stefano Stabellini <sstabellini@xxxxxxxxxx>;
> xen-users@xxxxxxxxxxxxxxxxxxxx <xen-users@xxxxxxxxxxxxxxxxxxxx>; brian.woods@xxxxxxxxxx <brian.woods@xxxxxxxxxx>; fnuv@xxxxxxxxxx
> <fnuv@xxxxxxxxxx>; edgari@xxxxxxxxxx <edgari@xxxxxxxxxx>; bwoods288@xxxxxxxxx <bwoods288@xxxxxxxxx>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> Prem,
>
> Sorry for the late email, I've been under the weather and have been away
> from my computer.  In the Zynq SOCs this is how the SMMU IDs in PL are
> calculated:
>
> 14-10: TBU bits
> 9-6: Master ID bits
> 5-0: AXI ID bits
>
> https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> To look up the TBU value, see UG1085 and look for the "System Masters"
> table which is table 16-4 on page 409.
>
> For the master ID, see UG1085 and look for the "Master IDs List" table
> which is table 16-13 on page 429.
>
> The AXI ID bits is a bit trickery since it's going to be out in PL.
> See: https://www.xilinx.com/support/answers/69447.html
> With SmartConnect all the AXI IDs are 0, they aren't passed on.  There
> are work arounds.  Here's a couple of examples (with example numbers):
>
> AXI master -> PS slave port 0
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
>
> AXI master -> SmartConnect -> PS slave port 0
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
>
> AXI master -> SmartConnect -> PS slave port 0
>                            -> PS slave port 1
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
>         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
>
> Two AXI masters -> SmartConnect -> PS slave port 0
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
>
> Two AXI masters -> SmartConnect -> PS slave port 0
>                                 -> PS slave port 1
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
>         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)
>
> Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
>
> Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
>                                                         -> PS slave port 1
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
>         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
>         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)
>
> The "sideband -> SmartConenct -> sideband" can also be functionally
> replaced with "AXI Interconnect" to my knowledge.  So the AXI ID bits are
> fairly simple but with some caveats.
>
> You can also read in the SMMU faults from Xen and use:
> https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> to decode them. That should tell you the SMMU IDs as well.
>
> Let me know if this isn't clear or you have any other questions.
>
> Brian
>
> On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > On Mon, 10 May 2021, Julien Grall wrote:
> > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > Hello everyone,
> > >
> > > Hi Prem,
> > >
> > > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > > and i am trying to create a barematal application that uses the AXI DMA on
> > > > the programmable logic, using the scatter gather example from the xilinx
> > > > repository. This works perfectly well for the baremetal application. The
> > > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > > AXI ID cannot manually be set.
> > > >
> > > >
> > > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > >
> > > >
> > > > I followed this tutorial Baremetal + DomU
> > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > > image and use tftp to boot.
> > > >
> > > >  From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > section 3 and MPSOC SMMU
> > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > section 4, stream ids are required by the SMMU to get the correct address
> > > > translation.
> > > >
> > > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > > and how to set the dtdev config for my hypervisor guest.
> > > >
> > > >
> > > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > > which i can be useful to my application
> > > >
> > > >   *   1:1 memory mappings for Xen virtual machines, enabling device
> > > >     assignments without SMMU
> > > >   *
> > > >       Static assignment of PL blocks, including bus-mastering blocks, to
> > > >     Xen virtual machines
> > > >   *   Support for PL Device Passthrough to DomU in Xen
> > >
> > > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > > recommend to contact them directly.
> > >
> > > I have CCed Stefano who might be able to help you.
> >
> > Hi Julien,
> >
> > Many thanks for CC'ing me, I missed the email.
> >
> >
> > Hi Prem,
> >
> > Let me CC a few people that might be able to help.
>
>


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.