Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>]

Great to hear! Thank you for reporting back.


On Tue, 18 May 2021, Prem Harikrishnan wrote:
> Dear Stefano,
> Thank you very much for your guidance and support. I finally fixed the issue. 
> I made an error in base memory address like you pointed out.
> I actually modified the base address correctly to 0x40000000 but the default 
> C code had an additional offset of 0x1100000 , and the DomU
> memory allocated was only 8MB. I either had to change the offset or allocate 
> more memory to the DomU guest which i missed until you pointed
> out. Additionally i had to set to AxPROT bits to 0x2 in Vivado and the SMID 
> required is 0xE80.
> 
> Thanks again i was able get my custom IP also working as an AXI master from 
> PL.
> 
> Best Regards,
> Prem
> 
> ___________________________________________________________________________________________________________________________________________
> From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> Sent: Tuesday, May 18, 2021 02:35
> To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> Cc: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; Brian Woods 
> <brian.woods@xxxxxxxxxx>; Julien Grall <julien@xxxxxxx>; Stefano
> Stabellini <sstabellini@xxxxxxxxxx>; xen-users@xxxxxxxxxxxxxxxxxxxx 
> <xen-users@xxxxxxxxxxxxxxxxxxxx>; fnuv@xxxxxxxxxx <fnuv@xxxxxxxxxx>;
> edgari@xxxxxxxxxx <edgari@xxxxxxxxxx>; bwoods288@xxxxxxxxx 
> <bwoods288@xxxxxxxxx>; Renato Costa Amorim <renato.amorim@xxxxxxxxxxxxxx>;
> JOVITAC@xxxxxxxxxx <JOVITAC@xxxxxxxxxx>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> masters in ZCU102 dev kit  
> Your configuration looks correct as far as I can tell.
> 
> You need dtdev because it triggers the SMMU configuration for your PL
> block. You don't need device_tree because your baremetal application is
> not going to read it, but it is not going to cause any issues either.
> 
> When you say "the execution does not complete the loop at line 244", it
> means that you see the execution entering the loop and then never
> leaving? Like an infinite loop?
> 
> If so, then I imagine the issue could be that the application is
> crashing when trying to access TxBufferPtr, which is pointing to
> TX_BUFFER_BASE, which is:
> 
> 
> #ifndef DDR_BASE_ADDR
> #warning CHECK FOR THE VALID DDR ADDRESS IN XPARAMETERS.H, \
>                  DEFAULT SET TO 0x01000000
> #define MEM_BASE_ADDR           0x01000000
> #else
> #define MEM_BASE_ADDR           (DDR_BASE_ADDR + 0x1000000)
> #endif
> 
> #define TX_BUFFER_BASE          (MEM_BASE_ADDR + 0x00100000)
> 
> 
> 
> Can you double check that TX_BUFFER_BASE is pointing to a correct
> address? Memory starts at 0x40000000 in a Xen DomU.
> 
> Maybe MEM_BASE_ADDR/DDR_BASE_ADDR is not set correctly?
> 
> 
> On Mon, 17 May 2021, Prem Harikrishnan wrote:
> > Hi Stefano,
> > Thanks again for your reply and your suggestion.  I did try to replace the 
> > Xil_DCacheFlushRange with Xil_DCacheInvalidateRange(works only
> > in EL3), I added printfs to debug, i can see that the execution does not 
> > complete the loop at line 244
> > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
> >   and
> > Xil_DCacheFlushRange is not reached. Please find attached my xen 
> > bootlog.txt.
> > I am still unable to find out why??
> >
> > Can you please atleast verify that my xen.dtsi, and pl.dtb is correct? and 
> > do i need dtdev and device_tree in my guest config file when
> it
> > is already added to xen.dtsi?
> >
> > Best Regards,
> > Prem
> >
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > Sent: Saturday, May 15, 2021 00:25
> > To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> > Cc: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; Brian Woods 
> > <brian.woods@xxxxxxxxxx>; Julien Grall <julien@xxxxxxx>; Stefano
> > Stabellini <sstabellini@xxxxxxxxxx>; xen-users@xxxxxxxxxxxxxxxxxxxx 
> > <xen-users@xxxxxxxxxxxxxxxxxxxx>; fnuv@xxxxxxxxxx <fnuv@xxxxxxxxxx>;
> > edgari@xxxxxxxxxx <edgari@xxxxxxxxxx>; bwoods288@xxxxxxxxx 
> > <bwoods288@xxxxxxxxx>; Renato Costa Amorim <renato.amorim@xxxxxxxxxxxxxx>;
> > JOVITAC@xxxxxxxxxx <JOVITAC@xxxxxxxxxx>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> > masters in ZCU102 dev kit  
> > Hi Prem,
> >
> > Some older versions of Xil_DCacheFlushRange were implemented using
> > deprecated set/way instructions that are known to cause problems in
> > virtualized environments.
> >
> > More recent implementations use the "dc civac" intruction that shouldn't
> > have any issues.
> >
> > Can you double-check that you are calling the 64-bit version implemented
> > by CIVAC?
> >
> >https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L414
> 
> >
> >
> >
> >
> > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > Hi Stefano, 
> > > Thanks for your reply. Yes the axidma application is a baremetal 
> > > application from the xilinx vitis ide example. 
> > > Also yes, I have already run hello world as DomU guest, with UART1.
> > >
> > > I am using the simple mode using polling and not interrupts(they are 
> > > disabled in the c code), which is Why I didn't include in the
> guest
> > > config file. This works well when I compile it normally (as EL3). When 
> > > compiled as hypervisor guest, the execution stops before the
> > > function xil_dcacheflushrange(). Please find the c code attached
> > > in 
> > > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
> > >  
> > >
> > > I found the SATA example similar to my use case, also using DMA and SMMU. 
> > > I just used it as a reference to create my pl.dtb to pass
> > through
> > > in the guest config.(from pl.dtsi, which I manually created).
> > >
> > > Thanks and Best Regards,
> > > Prem
> > >
> > > Get Outlook for Android
> > >
> >>_________________________________________________________________________________________________________________________________________
> _
> > _
> > > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > > Sent: Friday 14 May 2021, 23:16
> > > To: Prem Harikrishnan
> > > Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; 
> > > xen-users@xxxxxxxxxxxxxxxxxxxx; fnuv@xxxxxxxxxx;
> > edgari@xxxxxxxxxx;
> > > bwoods288@xxxxxxxxx; Renato Costa Amorim; JOVITAC@xxxxxxxxxx
> > > Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> > > masters in ZCU102 dev kit
> > >
> > > I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
> > > a baremetal application? Can you run it successfully, just printing
> > > "hello world", if you only assign the UART and nothing else?
> > >
> > > I am asking just to verify that you can run the baremetal VM
> > > successfully, aside from the PL device assignment.
> > >
> > >
> > > In regards to the PL device assignment, I'll let Brian answer on the
> > > Vivado configuration and SMID calculation. But I noticed that you only
> > > assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> > > assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
> > > looks like there are two related IRQs: 121 and 122.
> > >
> > > Finally, why did you mention SATA passthrough in your reply? I don't
> > > follow: I cannot see any SATA controller assignment in your
> > > configuration.
> > >
> > >
> > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > Dear Brian, Stefano and Julien,
> > > >
> > > > Thank you very much for your replies.
> > > > Hi Brian and Stefano, Can you please help me with the following 
> > > > questions?
> > > >
> > > > @Brian Woods Please find attached my vivado configuration, when i added 
> > > > my AXI DMA now running on simple polling mode(previously i
> > tried
> > > it
> > > > with Scatter Gather mode), Run Automation automatically generated, 1 
> > > > AXI Interconnect and 1 smart connect. Since there is only 1
> master
> > > and
> > > > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream 
> > > > IDs.
> > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters
> > > >  According to this link section 6.5 I have manually
> > tied
> > > > of AxPROT bits to value of 0x2 to create non secure transactions. Can 
> > > > you please confirm that my Vivado configuration is correct? I
> can
> > > > update you soon with an ILA trace on Vivado.
> > > >
> > > > @Stefano Stabellini Please find attached my pl.dtsi generated from 
> > > > petalinux and system.dtsi also generated after manually adding
> > > xen.dtsi.
> > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2
> > > >  Using this
> > SATA
> > > > passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I 
> > > > could really use your help to identify what is missing in my
> xen
> > > > configuration.
> > > >
> > > > Thanks and Best Regards,
> > > > Prem
> > > >
> >>>________________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > From: Brian Woods <brian.woods@xxxxxxxxxx>
> > > > Sent: Thursday, May 13, 2021 17:13
> > > > To: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > > > Cc: Julien Grall <julien@xxxxxxx>; Prem Harikrishnan 
> > > > <prem.harikrishnan@xxxxxxxxxxxxxx>; Stefano Stabellini 
> > > > <sstabellini@xxxxxxxxxx>;
> > > > xen-users@xxxxxxxxxxxxxxxxxxxx <xen-users@xxxxxxxxxxxxxxxxxxxx>; 
> > > > brian.woods@xxxxxxxxxx <brian.woods@xxxxxxxxxx>; fnuv@xxxxxxxxxx
> > > > <fnuv@xxxxxxxxxx>; edgari@xxxxxxxxxx <edgari@xxxxxxxxxx>; 
> > > > bwoods288@xxxxxxxxx <bwoods288@xxxxxxxxx>
> > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> > > > masters in ZCU102 dev kit  
> > > > Prem,
> > > >
> > > > Sorry for the late email, I've been under the weather and have been away
> > > > from my computer.  In the Zynq SOCs this is how the SMMU IDs in PL are
> > > > calculated:
> > > >
> > > > 14-10: TBU bits
> > > > 9-6: Master ID bits
> > > > 5-0: AXI ID bits
> > > >
> > > > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > > > To look up the TBU value, see UG1085 and look for the "System Masters"
> > > > table which is table 16-4 on page 409.
> > > >
> > > > For the master ID, see UG1085 and look for the "Master IDs List" table
> > > > which is table 16-13 on page 429.
> > > >
> > > > The AXI ID bits is a bit trickery since it's going to be out in PL.
> > > > See: https://www.xilinx.com/support/answers/69447.html
> > > > With SmartConnect all the AXI IDs are 0, they aren't passed on.  There
> > > > are work arounds.  Here's a couple of examples (with example numbers):
> > > >
> > > > AXI master -> PS slave port 0
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
> > > >
> > > > AXI master -> SmartConnect -> PS slave port 0
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > >
> > > > AXI master -> SmartConnect -> PS slave port 0
> > > >                            -> PS slave port 1
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
> > > >
> > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 
> > > > (both share)
> > > >
> > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > >                                 -> PS slave port 1
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 
> > > > (both share)
> > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 
> > > > (both share)
> > > >
> > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave 
> > > > port 0
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 
> > > > (master #1)
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 
> > > > (master #2)
> > > >
> > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave 
> > > > port 0
> > > >                                                         -> PS slave 
> > > > port 1
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 
> > > > (master #1)
> > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 
> > > > (master #1)
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 
> > > > (master #2)
> > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 
> > > > (master #2)
> > > >
> > > > The "sideband -> SmartConenct -> sideband" can also be functionally
> > > > replaced with "AXI Interconnect" to my knowledge.  So the AXI ID bits 
> > > > are
> > > > fairly simple but with some caveats.
> > > >
> > > > You can also read in the SMMU faults from Xen and use:
> > > > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > > > to decode them. That should tell you the SMMU IDs as well.
> > > >
> > > > Let me know if this isn't clear or you have any other questions.
> > > >
> > > > Brian
> > > >
> > > > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > > > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > > > Hello everyone,
> > > > > >
> > > > > > Hi Prem,
> > > > > >
> > > > > > > I am using the latest version of the xilinx mpsoc zcu102 
> > > > > > > development board
> > > > > > > and i am trying to create a barematal application that uses the 
> > > > > > > AXI DMA on
> > > > > > > the programmable logic, using the scatter gather example from the 
> > > > > > > xilinx
> > > > > > > repository. This works perfectly well for the baremetal 
> > > > > > > application. The
> > > > > > > diagram below represents the information flow. The AXI DMA and 
> > > > > > > the AXI data
> > > > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of 
> > > > > > > the box IP
> > > > > > > from Vivado and uses a single channel (Does not use AxUSER) which 
> > > > > > > means the
> > > > > > > AXI ID cannot manually be set.
> > > > > > >
> > > > > > >
> > > > > > > Now i want to make this as a baremetal hypervisor guest, in 
> > > > > > > future this will
> > > > > > > be replaced by a custom IP with cache colouring enabled. I want 
> > > > > > > to pass
> > > > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > > > >
> > > > > > >
> > > > > > > I followed this tutorial Baremetal + DomU
> > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > > > guest changed the base address to 0x4000000,(1GB allocated to 
> > > > > > > Dom-0) to
> > > > > > > create my EL1 binary. I used Petalinux 2020.2 to generate my 
> > > > > > > linux, xen
> > > > > > > image and use tftp to boot.
> > > > > > >
> > > > > > >  From what i understand from the xilinx tutorials 
> > > > > > >Xen+and+PL+Masters
> > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > > > section 3 and MPSOC SMMU
> > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > > > section 4, stream ids are required by the SMMU to get the correct 
> > > > > > > address
> > > > > > > translation.
> > > > > > >
> > > > > > > I am still unable to figure out how to get the iommu property for 
> > > > > > > my AXI DMA
> > > > > > > and how to set the dtdev config for my hypervisor guest.
> > > > > > >
> > > > > > >
> > > > > > > I can also see some new updates with the XEN 2020.1 and 2020.2 
> > > > > > > releases
> > > > > > > which i can be useful to my application
> > > > > > >
> > > > > > >   *   1:1 memory mappings for Xen virtual machines, enabling 
> > > > > > >device
> > > > > > >     assignments without SMMU
> > > > > > >   *
> > > > > > >       Static assignment of PL blocks, including bus-mastering 
> > > > > > >blocks, to
> > > > > > >     Xen virtual machines
> > > > > > >   *   Support for PL Device Passthrough to DomU in Xen
> > > > > >
> > > > > > You are usiong a version of Xen that has been customised by Xilinx. 
> > > > > > So I would
> > > > > > recommend to contact them directly.
> > > > > >
> > > > > > I have CCed Stefano who might be able to help you.
> > > > >
> > > > > Hi Julien,
> > > > >
> > > > > Many thanks for CC'ing me, I missed the email.
> > > > >
> > > > >
> > > > > Hi Prem,
> > > > >
> > > > > Let me CC a few people that might be able to help.
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> 
>