Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>]

We don't have a recipe ready. It is not something we test. However I
think it should be possible.

The first step would be try to generate a BOOT.BIN for QSPI booting and
write it to QSPI. The goal is to get to the u-boot prompt, with
everything booted from QSPI. Then the next step is to add Xen, Linux,
the DTB and Linux ramdisk. (Of course if you are OK with loading Xen and
the other binaries from somewhere else like the SD card then it is going
to work as is.)

You should be able to write each binary to QSPI as separate partitions.

At that point, you are only missing a boot.scr that can load them and
boot. ImageBuilder supports custom u-boot load commands by setting the
LOAD_CMD variable. In this case you'd need to set LOAD_CMD to something
like "sf read", but given that the other command arguments are not
filenames, ImageBuilder won't be able to handle it anyway.

But you can take an ImageBuilder-generated boot.source, substitute the
"tftpb" commands with the appropriate "sf read" commands and call
mkimage to get a boot.scr.

Now you should be able to run boot.scr from the u-boot prompt and load
everything from QSPI.

This is some relevant info:

https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842465/U-Boot+QSPI+Driver


I know it is a bit complex but I think it should work OK.



On Tue, 15 Jun 2021, Prem Harikrishnan wrote:
> Hi Stefano,
> 
> Is it possible to boot XEN from QSPI? Is there an image builder script 
> already available to generate boot.scr? Please let me know.
> Thanks and Best Regards,
> Prem
> 
> ___________________________________________________________________________________________________________________________________________
> From: Xen-users <xen-users-bounces@xxxxxxxxxxxxxxxxxxxx> on behalf of Prem 
> Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> Sent: Tuesday, June 1, 2021 18:07
> To: Mubin Usman Sayyed <MUBINUSM@xxxxxxxxxx>; Stefano Stabellini 
> <stefanos@xxxxxxxxxx>
> Cc: xen-users@xxxxxxxxxxxxxxxxxxxx <xen-users@xxxxxxxxxxxxxxxxxxxx>; Jovita 
> Castelino <JOVITAC@xxxxxxxxxx>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> masters in ZCU102 dev kit  
> Hi Stefano and Mubin,
> 
> Thank you very much for helping me setup up FreeRTOS and baremetal guests, 
> everything works as expected and I also get expected performance
> in both baremetal and hypervisor guests. Thank you again for your prompt 
> support during the past few weeks.
> 
> Best Regrards,
> Prem
> 
> ___________________________________________________________________________________________________________________________________________
> From: Mubin Usman Sayyed <MUBINUSM@xxxxxxxxxx>
> Sent: Tuesday, June 1, 2021 05:38
> To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>; Stefano Stabellini 
> <stefanos@xxxxxxxxxx>
> Cc: xen-users@xxxxxxxxxxxxxxxxxxxx <xen-users@xxxxxxxxxxxxxxxxxxxx>; Jovita 
> Castelino <JOVITAC@xxxxxxxxxx>
> Subject: RE: Help: Information required for AXI DMA passthrough for PL 
> masters in ZCU102 dev kit  
> 
> In case of FreeRTOS,  you need to use APIs from FreeRTOS port for interrupt 
> handling
>  
> (https://github.com/Xilinx/embeddedsw/blob/master/ThirdParty/bsp/freertos10_xilinx/examples/freertos_intr_example.c#L192
>  and line 194).
> Please make sure that you are taking care of that in DMA interrupt example .
> 
>  
> 
> Thanks,
> 
> Mubin
> 
>  
> 
> From: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> Sent: Tuesday, June 1, 2021 2:48 AM
> To: Stefano Stabellini <stefanos@xxxxxxxxxx>
> Cc: xen-users@xxxxxxxxxxxxxxxxxxxx; Jovita Castelino <JOVITAC@xxxxxxxxxx>; 
> Mubin Usman Sayyed <MUBINUSM@xxxxxxxxxx>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> masters in ZCU102 dev kit
> 
>  
> 
>  
> 
> Hi Stefano and Mubin,
> 
> Thank you very much for your replies.
> 
> @Stefano Stabellini Sorry to bother you again, and thank you for pointing 
> this out, the TTC0 is by default enabled for FreeRTOS to generate
> ticks, although this says only for R5 core, i would have never figured this 
> out. As soon as I added the irqs and iomem and gave a xen
> passthrough in the xen.dtsi as you have suggested, FreeRTOS guest started 
> working for other peripherals except for my DMA (FreeRTOS guest
> does not work with interrupt and DMA).
> 
>  
> 
> I can see my DMA is enabled because i have LEDs configured to blink for every 
> interrupt I get from the PL. I don't see my interrupts being
> processed, but when I destroy the guest, I get this output periodically when 
> an interrupt occurs.
> 
>  
> 
> (XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
> (XEN) smmu: /smmu@fd800000:     GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 
> 0x00000e80, GFSYNR2 0x00000000
> 
> (XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
> (XEN) smmu: /smmu@fd800000:     GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 
> 0x00000e80, GFSYNR2 0x00000000
> 
> (XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
> (XEN) smmu: /smmu@fd800000:     GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 
> 0x00000e80, GFSYNR2 0x00000000
> 
>  
> 
> From section 3.6 of the wiki 
> https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters
> 
> Xen and PL Masters - Xilinx Wiki - Confluence
> 
> 3.4 Stream IDs in the Device Tree. For Xen based systems the stream IDs of PL 
> masters must be added to the Linux Dom0 device tree.
> Automated device tree generation does not generate the stream IDs for the PL 
> masters so that the user must add them into the device tree
> manually.
> 
> xilinx-wiki.atlassian.net
> 
> The fault indicates an unidentified stream ID has been received by the SMMU 
> as shown in the GFSR register. The GFSYNR1 register contains
> the stream ID that was unidentified. A stream ID of 0xE80 was unidentified in 
> the above fault.
> 
>  
> 
> But I think i have already added that in my smmu and iommu in xen.dtsi 
> because my baremetal application works well as expected. I have
> attached my xen.dtsi, system.dtsi, baremetal.cfg(working) and 
> freertos.cfg(not working).
> 
> What could I be missing here? Do I have to enable the ttc0 in the device tree?
> 
> Also what does (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 99990 
> KHz mean when XEN boots up?
> 
> Thanks, I look forward for your reply.
> 
>  
> 
> Best Regards,
> 
> Prem
> 
>  
> 
>  
> 
> 
>  
> ________________________________________________________________________________________________________________________________________
> 
> 
> From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> Sent: Friday, May 28, 2021 20:43
> To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> Cc: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; 
> xen-users@xxxxxxxxxxxxxxxxxxxx <xen-users@xxxxxxxxxxxxxxxxxxxx>; 
> JOVITAC@xxxxxxxxxx
> <JOVITAC@xxxxxxxxxx>; MUBINUSM@xxxxxxxxxx <MUBINUSM@xxxxxxxxxx>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> masters in ZCU102 dev kit
> 
>  
> 
> From the gpa address, it looks like FreeRTOS is trying to access the TTC
> timer? If you look at device tree, the corresponding node is
> timer@ff110000.
> 
> If FreeRTOS is configured to access the TTC timer, then you need to
> assign it to it. Did you assign the TTC timer to the FreeRTOS domU by
> any chance?
> 
> You can do that by adding:
> 
> irqs = [ 68, 69, 70 ]
> iomem = [ "0xff110,1" ]
> 
> 
> On Fri, 28 May 2021, Prem Harikrishnan wrote:
> > Hi Stefano,
> > Thanks for your reply. I am pretty sure I followed the steps you mentioned 
> > below to run the FreeRTOS hello world example. I already added
> > printfs and i can see XEN cannot start the scheduler and i get the 
> > following out put on XEN.
> >
> > Parsing config from freertos.cfg (XEN) Dom2 colors: [ 8 ]
> > (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=2 
> > memflags=0xc0 (0 of 64)
> > (XEN) memory_map:add: dom2 gfn=ff010 mfn=ff010 nr=1 cache=0
> > (XEN) sched_null.c:344: 1 <-- d2v0
> > (XEN) traps.c:1973:d2v0 HSR=0x93820007 pc=0x00000040012b08 gva=0xff11000c 
> > gpa=0x000000ff11000c
> >
> >
> > I also see that according to this article
> > https://forums.xilinx.com/t5/Adaptable-Advantage-Blog/Partner-FreeRTOS-as-a-Xen-Virtual-Machine/ba-p/947030
> > [52662iFAD1733707609204?v=v2]
> > Partner: FreeRTOS as a Xen Virtual Machine - Community Forums
> > Contributed by Jeff Kubascik, Embedded Systems Engineer at DornerWorks What 
> > is FreeRTOS? The FreeRTOS kernel is a market leading
> real-time
> > operating system for embedded systems. As embedded applications grow in 
> > complexity, there is a growing need for a lightweight kernel
> > capable of task scheduling...
> > forums.xilinx.com
> > FreeRTOS does not run out of the box for XEN, although the article is 2 
> > years old, I am not sure if there are any updates on this after
> > that. Can you please confirm?
> >
> > Thanks and Best Regards,
> > Prem
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > Sent: Friday, May 28, 2021 02:51
> > To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> > Cc: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; 
> > xen-users@xxxxxxxxxxxxxxxxxxxx <xen-users@xxxxxxxxxxxxxxxxxxxx>;
> JOVITAC@xxxxxxxxxx
> > <JOVITAC@xxxxxxxxxx>; MUBINUSM@xxxxxxxxxx <MUBINUSM@xxxxxxxxxx>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> > masters in ZCU102 dev kit  
> > +Mubin
> >
> >
> > Hi Prem,
> >
> > Did you make sure to build FreeRTOS as "hypervisor guest"? I think
> > FreeRTOS is built to run at EL3 by default so if that is not changed
> > there are errors when run on any hypervisor. It is possible to build
> > FreeRTOS to run on Xen but it requires a couple of manual steps:
> >
> > - source settings64.sh from Vitis builds
> > - xsct
> > - set WORKSPACE .
> > - setws -switch $WORKSPACE
> > - app create -name freertos-hello-world -template {FreeRTOS Hello World} 
> > -proc psu_cortexa53_0 -hw {xsa path} -os freertos10_xilinx
> > - bsp config stdin psu_uart_1
> > - bsp config stdout psu_uart_1
> > - bsp config hypervisor_guest true
> > - Go to application source path and modify the linker script start address 
> > to 0x40000000,
> > - app build -name freertos-hello-world
> > - Run command to generate BIN file from executable 
> > "aarch64-linux-gnu-objcopy -O binary --gap-fill 0  <elf path> BOOT.BIN"
> >
> > The resulting BOOT.BIN can be started a Xen DomU guest. Mubin, CC'ed,
> > might be able to provide additional info.
> >
> >
> >
> > On Thu, 27 May 2021, Prem Harikrishnan wrote:
> > > Hi Stefano,
> > > Sorry to bother you again.
> > > I am currently trying to run my custom IP which sends periodic dma 
> > > interrupts to the PS with cache coloring enabled. I have now
> compiled
> > my
> > > guest running with FreeRTOS and it generates a trap.
> > > Petalinux and VITIS IDE Version is 2020.2.
> > >
> > > Please find the log attached. After the trap occurs, for every interrupt 
> > > XEN prints (XEN) smmu: /smmu@fd800000: Unhandled context
> fault:
> > > fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > >
> > > root@pilbara:/etc/xen# xl create -c apu_freertos.cfg Parsing config from 
> > > apu_freertos.cfg
> > > (XEN) Dom1 colors: [ 8 ]
> > > (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=1 
> > > memflags=0xc0 (0 of 64)
> > > (XEN) memory_map:add: dom1 gfn=ff010 mfn=ff010 nr=1 cache=0
> > > (XEN) memory_map:add: dom1 gfn=a0010 mfn=a0010 nr=1 cache=0
> > > (XEN) smmu: /smmu@fd800000: d1: p2maddr 0x000000087f554000
> > > (XEN) sched_null.c:344: 1 <-- d1v0
> > > (XEN) traps.c:1973:d1v0 HSR=0x93820007 pc=0x00000040013188 gva=0xff11000c 
> > > gpa=0x000000ff11000c
> > > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, 
> > > iova=0x41000000, fsynr=0x12, cb=1
> > > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, 
> > > iova=0x41000000, fsynr=0x12, cb=1
> > > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, 
> > > iova=0x41000000, fsynr=0x12, cb=1
> > > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, 
> > > iova=0x41000000, fsynr=0x12, cb=1
> > >
> > > The same application works as expected without FreeRTOS, i.e as a 
> > > baremetal guest. I look forward for your comment on this.
> > >
> > > Although i see performance degradation, normally i can measure delta time 
> > > for each interrupt configured to 1ms (without XEN), with XEN
> > for
> > > the same measurement shows 16 ms for each interval (but the performance 
> > > is not a priority for now).
> > > Thanks and Best Regards,
> > > Prem
> > >
> >>_________________________________________________________________________________________________________________________________________
> _
> > _
> > > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > > Sent: Tuesday, May 18, 2021 20:09
> > > To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> > > Cc: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; 
> > > xen-users@xxxxxxxxxxxxxxxxxxxx <xen-users@xxxxxxxxxxxxxxxxxxxx>;
> > JOVITAC@xxxxxxxxxx
> > > <JOVITAC@xxxxxxxxxx>
> > > Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> > > masters in ZCU102 dev kit  
> > > Great to hear! Thank you for reporting back.
> > >
> > >
> > > On Tue, 18 May 2021, Prem Harikrishnan wrote:
> > > > Dear Stefano,
> > > > Thank you very much for your guidance and support. I finally fixed the 
> > > > issue. I made an error in base memory address like you pointed
> > > out.
> > > > I actually modified the base address correctly to 0x40000000 but the 
> > > > default C code had an additional offset of 0x1100000 , and the
> > DomU
> > > > memory allocated was only 8MB. I either had to change the offset or 
> > > > allocate more memory to the DomU guest which i missed until you
> > > pointed
> > > > out. Additionally i had to set to AxPROT bits to 0x2 in Vivado and the 
> > > > SMID required is 0xE80.
> > > >
> > > > Thanks again i was able get my custom IP also working as an AXI master 
> > > > from PL.
> > > >
> > > > Best Regards,
> > > > Prem
> > > >
> >>>________________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > > > Sent: Tuesday, May 18, 2021 02:35
> > > > To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> > > > Cc: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; Brian Woods 
> > > > <brian.woods@xxxxxxxxxx>; Julien Grall <julien@xxxxxxx>; Stefano
> > > > Stabellini <sstabellini@xxxxxxxxxx>; xen-users@xxxxxxxxxxxxxxxxxxxx 
> > > > <xen-users@xxxxxxxxxxxxxxxxxxxx>; fnuv@xxxxxxxxxx
> > <fnuv@xxxxxxxxxx>;
> > > > edgari@xxxxxxxxxx <edgari@xxxxxxxxxx>; bwoods288@xxxxxxxxx 
> > > > <bwoods288@xxxxxxxxx>; Renato Costa Amorim 
> > > > <renato.amorim@xxxxxxxxxxxxxx>;
> > > > JOVITAC@xxxxxxxxxx <JOVITAC@xxxxxxxxxx>
> > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> > > > masters in ZCU102 dev kit  
> > > > Your configuration looks correct as far as I can tell.
> > > >
> > > > You need dtdev because it triggers the SMMU configuration for your PL
> > > > block. You don't need device_tree because your baremetal application is
> > > > not going to read it, but it is not going to cause any issues either.
> > > >
> > > > When you say "the execution does not complete the loop at line 244", it
> > > > means that you see the execution entering the loop and then never
> > > > leaving? Like an infinite loop?
> > > >
> > > > If so, then I imagine the issue could be that the application is
> > > > crashing when trying to access TxBufferPtr, which is pointing to
> > > > TX_BUFFER_BASE, which is:
> > > >
> > > >
> > > > #ifndef DDR_BASE_ADDR
> > > > #warning CHECK FOR THE VALID DDR ADDRESS IN XPARAMETERS.H, \
> > > >                  DEFAULT SET TO 0x01000000
> > > > #define MEM_BASE_ADDR           0x01000000
> > > > #else
> > > > #define MEM_BASE_ADDR           (DDR_BASE_ADDR + 0x1000000)
> > > > #endif
> > > >
> > > > #define TX_BUFFER_BASE          (MEM_BASE_ADDR + 0x00100000)
> > > >
> > > >
> > > >
> > > > Can you double check that TX_BUFFER_BASE is pointing to a correct
> > > > address? Memory starts at 0x40000000 in a Xen DomU.
> > > >
> > > > Maybe MEM_BASE_ADDR/DDR_BASE_ADDR is not set correctly?
> > > >
> > > >
> > > > On Mon, 17 May 2021, Prem Harikrishnan wrote:
> > > > > Hi Stefano,
> > > > > Thanks again for your reply and your suggestion.  I did try to 
> > > > > replace the Xil_DCacheFlushRange with
> Xil_DCacheInvalidateRange(works
> > > only
> > > > > in EL3), I added printfs to debug, i can see that the execution does 
> > > > > not complete the loop at line 244
> > > > > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
> > > > >   and
> > > > > Xil_DCacheFlushRange is not reached. Please find attached my xen 
> > > > > bootlog.txt.
> > > > > I am still unable to find out why??
> > > > >
> > > > > Can you please atleast verify that my xen.dtsi, and pl.dtb is 
> > > > > correct? and do i need dtdev and device_tree in my guest config file
> > when
> > > > it
> > > > > is already added to xen.dtsi?
> > > > >
> > > > > Best Regards,
> > > > > Prem
> > > > >
> > > > >
> >>>>_______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > > > > Sent: Saturday, May 15, 2021 00:25
> > > > > To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> > > > > Cc: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; Brian Woods 
> > > > > <brian.woods@xxxxxxxxxx>; Julien Grall <julien@xxxxxxx>;
> Stefano
> > > > > Stabellini <sstabellini@xxxxxxxxxx>; xen-users@xxxxxxxxxxxxxxxxxxxx 
> > > > > <xen-users@xxxxxxxxxxxxxxxxxxxx>; fnuv@xxxxxxxxxx
> > > <fnuv@xxxxxxxxxx>;
> > > > > edgari@xxxxxxxxxx <edgari@xxxxxxxxxx>; bwoods288@xxxxxxxxx 
> > > > > <bwoods288@xxxxxxxxx>; Renato Costa Amorim
> <renato.amorim@xxxxxxxxxxxxxx>;
> > > > > JOVITAC@xxxxxxxxxx <JOVITAC@xxxxxxxxxx>
> > > > > Subject: Re: Help: Information required for AXI DMA passthrough for 
> > > > > PL masters in ZCU102 dev kit  
> > > > > Hi Prem,
> > > > >
> > > > > Some older versions of Xil_DCacheFlushRange were implemented using
> > > > > deprecated set/way instructions that are known to cause problems in
> > > > > virtualized environments.
> > > > >
> > > > > More recent implementations use the "dc civac" intruction that 
> > > > > shouldn't
> > > > > have any issues.
> > > > >
> > > > > Can you double-check that you are calling the 64-bit version 
> > > > > implemented
> > > > > by CIVAC?
> > > > >
> >>>>https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L
> 4
> > 1
> > > 4
> > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > > Hi Stefano, 
> > > > > > Thanks for your reply. Yes the axidma application is a baremetal 
> > > > > > application from the xilinx vitis ide example. 
> > > > > > Also yes, I have already run hello world as DomU guest, with UART1.
> > > > > >
> > > > > > I am using the simple mode using polling and not interrupts(they 
> > > > > > are disabled in the c code), which is Why I didn't include in
> the
> > > > guest
> > > > > > config file. This works well when I compile it normally (as EL3). 
> > > > > > When compiled as hypervisor guest, the execution stops before
> the
> > > > > > function xil_dcacheflushrange(). Please find the c code attached
> > > > > > in 
> > > > > > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
> > > > > >  
> > > > > >
> > > > > > I found the SATA example similar to my use case, also using DMA and 
> > > > > > SMMU. I just used it as a reference to create my pl.dtb to
> pass
> > > > > through
> > > > > > in the guest config.(from pl.dtsi, which I manually created).
> > > > > >
> > > > > > Thanks and Best Regards,
> > > > > > Prem
> > > > > >
> > > > > > Get Outlook for Android
> > > > > >
> >>>>>______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > _
> > > > > > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > > > > > Sent: Friday 14 May 2021, 23:16
> > > > > > To: Prem Harikrishnan
> > > > > > Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano 
> > > > > > Stabellini; xen-users@xxxxxxxxxxxxxxxxxxxx; fnuv@xxxxxxxxxx;
> > > > > edgari@xxxxxxxxxx;
> > > > > > bwoods288@xxxxxxxxx; Renato Costa Amorim; JOVITAC@xxxxxxxxxx
> > > > > > Subject: Re: Help: Information required for AXI DMA passthrough for 
> > > > > > PL masters in ZCU102 dev kit
> > > > > >
> > > > > > I take that the guest kernel 
> > > > > > "/bin/xaxidma_example_simple_poll_1.bin" is
> > > > > > a baremetal application? Can you run it successfully, just printing
> > > > > > "hello world", if you only assign the UART and nothing else?
> > > > > >
> > > > > > I am asking just to verify that you can run the baremetal VM
> > > > > > successfully, aside from the PL device assignment.
> > > > > >
> > > > > >
> > > > > > In regards to the PL device assignment, I'll let Brian answer on the
> > > > > > Vivado configuration and SMID calculation. But I noticed that you 
> > > > > > only
> > > > > > assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> > > > > > assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, 
> > > > > > it
> > > > > > looks like there are two related IRQs: 121 and 122.
> > > > > >
> > > > > > Finally, why did you mention SATA passthrough in your reply? I don't
> > > > > > follow: I cannot see any SATA controller assignment in your
> > > > > > configuration.
> > > > > >
> > > > > >
> > > > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > > > Dear Brian, Stefano and Julien,
> > > > > > >
> > > > > > > Thank you very much for your replies.
> > > > > > > Hi Brian and Stefano, Can you please help me with the following 
> > > > > > > questions?
> > > > > > >
> > > > > > > @Brian Woods Please find attached my vivado configuration, when i 
> > > > > > > added my AXI DMA now running on simple polling
> mode(previously
> > i
> > > > > tried
> > > > > > it
> > > > > > > with Scatter Gather mode), Run Automation automatically 
> > > > > > > generated, 1 AXI Interconnect and 1 smart connect. Since there is 
> > > > > > > only
> 1
> > > > master
> > > > > > and
> > > > > > > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the 
> > > > > > > stream IDs.
> > > > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters
> > > > > > >  According to this link section 6.5 I have
> > > manually
> > > > > tied
> > > > > > > of AxPROT bits to value of 0x2 to create non secure transactions. 
> > > > > > > Can you please confirm that my Vivado configuration is
> correct?
> > I
> > > > can
> > > > > > > update you soon with an ILA trace on Vivado.
> > > > > > >
> > > > > > > @Stefano Stabellini Please find attached my pl.dtsi generated 
> > > > > > > from petalinux and system.dtsi also generated after manually
> adding
> > > > > > xen.dtsi.
> > > > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2
> > > > > > >  Using
> > this
> > > > > SATA
> > > > > > > passthrough, i made my own pl.dtsi for passthrough used in 
> > > > > > > dma.cfg. I could really use your help to identify what is missing 
> > > > > > > in
> > my
> > > > xen
> > > > > > > configuration.
> > > > > > >
> > > > > > > Thanks and Best Regards,
> > > > > > > Prem
> > > > > > >
> >>>>>>_____________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > _
> > > > > > _
> > > > > > > From: Brian Woods <brian.woods@xxxxxxxxxx>
> > > > > > > Sent: Thursday, May 13, 2021 17:13
> > > > > > > To: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > > > > > > Cc: Julien Grall <julien@xxxxxxx>; Prem Harikrishnan 
> > > > > > > <prem.harikrishnan@xxxxxxxxxxxxxx>; Stefano Stabellini
> > > <sstabellini@xxxxxxxxxx>;
> > > > > > > xen-users@xxxxxxxxxxxxxxxxxxxx <xen-users@xxxxxxxxxxxxxxxxxxxx>; 
> > > > > > > brian.woods@xxxxxxxxxx <brian.woods@xxxxxxxxxx>;
> fnuv@xxxxxxxxxx
> > > > > > > <fnuv@xxxxxxxxxx>; edgari@xxxxxxxxxx <edgari@xxxxxxxxxx>; 
> > > > > > > bwoods288@xxxxxxxxx <bwoods288@xxxxxxxxx>
> > > > > > > Subject: Re: Help: Information required for AXI DMA passthrough 
> > > > > > > for PL masters in ZCU102 dev kit  
> > > > > > > Prem,
> > > > > > >
> > > > > > > Sorry for the late email, I've been under the weather and have 
> > > > > > > been away
> > > > > > > from my computer.  In the Zynq SOCs this is how the SMMU IDs in 
> > > > > > > PL are
> > > > > > > calculated:
> > > > > > >
> > > > > > > 14-10: TBU bits
> > > > > > > 9-6: Master ID bits
> > > > > > > 5-0: AXI ID bits
> > > > > > >
> > > > > > > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > > > > > > To look up the TBU value, see UG1085 and look for the "System 
> > > > > > > Masters"
> > > > > > > table which is table 16-4 on page 409.
> > > > > > >
> > > > > > > For the master ID, see UG1085 and look for the "Master IDs List" 
> > > > > > > table
> > > > > > > which is table 16-13 on page 429.
> > > > > > >
> > > > > > > The AXI ID bits is a bit trickery since it's going to be out in 
> > > > > > > PL.
> > > > > > > See: https://www.xilinx.com/support/answers/69447.html
> > > > > > > With SmartConnect all the AXI IDs are 0, they aren't passed on.  
> > > > > > > There
> > > > > > > are work arounds.  Here's a couple of examples (with example 
> > > > > > > numbers):
> > > > > > >
> > > > > > > AXI master -> PS slave port 0
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > 0000001
> > > > > > >
> > > > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > 0000000
> > > > > > >
> > > > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > > >                            -> PS slave port 1
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > 0000000
> > > > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 
> > > > > > > 0000000
> > > > > > >
> > > > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > 0000000 (both share)
> > > > > > >
> > > > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > > >                                 -> PS slave port 1
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > 0000000 (both share)
> > > > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 
> > > > > > > 0000000 (both share)
> > > > > > >
> > > > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS 
> > > > > > > slave port 0
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > 0000001 (master #1)
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > 0000002 (master #2)
> > > > > > >
> > > > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS 
> > > > > > > slave port 0
> > > > > > >                                                         -> PS 
> > > > > > > slave port 1
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > 0000001 (master #1)
> > > > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 
> > > > > > > 0000001 (master #1)
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > 0000002 (master #2)
> > > > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 
> > > > > > > 0000002 (master #2)
> > > > > > >
> > > > > > > The "sideband -> SmartConenct -> sideband" can also be 
> > > > > > > functionally
> > > > > > > replaced with "AXI Interconnect" to my knowledge.  So the AXI ID 
> > > > > > > bits are
> > > > > > > fairly simple but with some caveats.
> > > > > > >
> > > > > > > You can also read in the SMMU faults from Xen and use:
> > > > > > > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > > > > > > to decode them. That should tell you the SMMU IDs as well.
> > > > > > >
> > > > > > > Let me know if this isn't clear or you have any other questions.
> > > > > > >
> > > > > > > Brian
> > > > > > >
> > > > > > > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini 
> > > > > > > wrote:
> > > > > > > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > > > > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > > > > > > Hello everyone,
> > > > > > > > >
> > > > > > > > > Hi Prem,
> > > > > > > > >
> > > > > > > > > > I am using the latest version of the xilinx mpsoc zcu102 
> > > > > > > > > > development board
> > > > > > > > > > and i am trying to create a barematal application that uses 
> > > > > > > > > > the AXI DMA on
> > > > > > > > > > the programmable logic, using the scatter gather example 
> > > > > > > > > > from the xilinx
> > > > > > > > > > repository. This works perfectly well for the baremetal 
> > > > > > > > > > application. The
> > > > > > > > > > diagram below represents the information flow. The AXI DMA 
> > > > > > > > > > and the AXI data
> > > > > > > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is 
> > > > > > > > > > out of the box IP
> > > > > > > > > > from Vivado and uses a single channel (Does not use AxUSER) 
> > > > > > > > > > which means the
> > > > > > > > > > AXI ID cannot manually be set.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Now i want to make this as a baremetal hypervisor guest, in 
> > > > > > > > > > future this will
> > > > > > > > > > be replaced by a custom IP with cache colouring enabled. I 
> > > > > > > > > > want to pass
> > > > > > > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I followed this tutorial Baremetal + DomU
> > > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > > > > > > guest changed the base address to 0x4000000,(1GB allocated 
> > > > > > > > > > to Dom-0) to
> > > > > > > > > > create my EL1 binary. I used Petalinux 2020.2 to generate 
> > > > > > > > > > my linux, xen
> > > > > > > > > > image and use tftp to boot.
> > > > > > > > > >
> > > > > > > > > >  From what i understand from the xilinx tutorials 
> > > > > > > > > >Xen+and+PL+Masters
> > > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > > > > > > section 3 and MPSOC SMMU
> > > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > > > > > > section 4, stream ids are required by the SMMU to get the 
> > > > > > > > > > correct address
> > > > > > > > > > translation.
> > > > > > > > > >
> > > > > > > > > > I am still unable to figure out how to get the iommu 
> > > > > > > > > > property for my AXI DMA
> > > > > > > > > > and how to set the dtdev config for my hypervisor guest.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I can also see some new updates with the XEN 2020.1 and 
> > > > > > > > > > 2020.2 releases
> > > > > > > > > > which i can be useful to my application
> > > > > > > > > >
> > > > > > > > > >   *   1:1 memory mappings for Xen virtual machines, 
> > > > > > > > > >enabling device
> > > > > > > > > >     assignments without SMMU
> > > > > > > > > >   *
> > > > > > > > > >       Static assignment of PL blocks, including 
> > > > > > > > > >bus-mastering blocks, to
> > > > > > > > > >     Xen virtual machines
> > > > > > > > > >   *   Support for PL Device Passthrough to DomU in Xen
> > > > > > > > >
> > > > > > > > > You are usiong a version of Xen that has been customised by 
> > > > > > > > > Xilinx. So I would
> > > > > > > > > recommend to contact them directly.
> > > > > > > > >
> > > > > > > > > I have CCed Stefano who might be able to help you.
> > > > > > > >
> > > > > > > > Hi Julien,
> > > > > > > >
> > > > > > > > Many thanks for CC'ing me, I missed the email.
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi Prem,
> > > > > > > >
> > > > > > > > Let me CC a few people that might be able to help.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
> 
> 
>