Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

[Brian Woods <brian.woods@xxxxxxxxxx>]

Prem,

Sorry for the late email, I've been under the weather and have been away
from my computer.  In the Zynq SOCs this is how the SMMU IDs in PL are
calculated:

14-10: TBU bits
9-6: Master ID bits
5-0: AXI ID bits

https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
To look up the TBU value, see UG1085 and look for the "System Masters"
table which is table 16-4 on page 409.

For the master ID, see UG1085 and look for the "Master IDs List" table
which is table 16-13 on page 429.

The AXI ID bits is a bit trickery since it's going to be out in PL.
See: https://www.xilinx.com/support/answers/69447.html
With SmartConnect all the AXI IDs are 0, they aren't passed on.  There
are work arounds.  Here's a couple of examples (with example numbers):

AXI master -> PS slave port 0
        TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001

AXI master -> SmartConnect -> PS slave port 0
        TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000

AXI master -> SmartConnect -> PS slave port 0
                           -> PS slave port 1
        TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
        TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000

Two AXI masters -> SmartConnect -> PS slave port 0
        TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)

Two AXI masters -> SmartConnect -> PS slave port 0
                                -> PS slave port 1
        TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
        TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)

Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
        TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
        TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)

Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
                                                        -> PS slave port 1
        TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
        TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
        TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
        TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)

The "sideband -> SmartConenct -> sideband" can also be functionally
replaced with "AXI Interconnect" to my knowledge.  So the AXI ID bits are
fairly simple but with some caveats.

You can also read in the SMMU faults from Xen and use:
https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
to decode them. That should tell you the SMMU IDs as well.

Let me know if this isn't clear or you have any other questions.

Brian

On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> On Mon, 10 May 2021, Julien Grall wrote:
> > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > Hello everyone,
> > 
> > Hi Prem,
> > 
> > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > and i am trying to create a barematal application that uses the AXI DMA on
> > > the programmable logic, using the scatter gather example from the xilinx
> > > repository. This works perfectly well for the baremetal application. The
> > > diagram below represents the information flow. The AXI DMA and the AXI 
> > > data
> > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box 
> > > IP
> > > from Vivado and uses a single channel (Does not use AxUSER) which means 
> > > the
> > > AXI ID cannot manually be set.
> > > 
> > > 
> > > Now i want to make this as a baremetal hypervisor guest, in future this 
> > > will
> > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > 
> > > 
> > > I followed this tutorial Baremetal + DomU
> > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > image and use tftp to boot.
> > > 
> > >  From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > section 3 and MPSOC SMMU
> > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > section 4, stream ids are required by the SMMU to get the correct address
> > > translation.
> > > 
> > > I am still unable to figure out how to get the iommu property for my AXI 
> > > DMA
> > > and how to set the dtdev config for my hypervisor guest.
> > > 
> > > 
> > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > which i can be useful to my application
> > > 
> > >   *   1:1 memory mappings for Xen virtual machines, enabling device
> > >     assignments without SMMU
> > >   *
> > >       Static assignment of PL blocks, including bus-mastering blocks, to
> > >     Xen virtual machines
> > >   *   Support for PL Device Passthrough to DomU in Xen
> > 
> > You are usiong a version of Xen that has been customised by Xilinx. So I 
> > would
> > recommend to contact them directly.
> > 
> > I have CCed Stefano who might be able to help you.
> 
> Hi Julien,
> 
> Many thanks for CC'ing me, I missed the email.
> 
> 
> Hi Prem,
> 
> Let me CC a few people that might be able to help.