Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>]

Excellent! Thanks for letting me know.


On Fri, 18 Jun 2021, Prem Harikrishnan wrote:
> Hi Stefano,
> 
> Thank you very much for your detailed reply.  I now have xen booting from 
> qspi, i followed your instructions, the only additional thing i
> had to change is the boot script offset in petalinux u boot config. Thank you 
> again for your support.
> 
> Best Regards,
> Prem 
> 
> ___________________________________________________________________________________________________________________________________________
> From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> Sent: Tuesday, June 15, 2021 22:17
> To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> Cc: Stefano Stabellini <stefanos@xxxxxxxxxx>; xen-users@xxxxxxxxxxxxxxxxxxxx 
> <xen-users@xxxxxxxxxxxxxxxxxxxx>; JOVITAC@xxxxxxxxxx
> <JOVITAC@xxxxxxxxxx>; fnuv@xxxxxxxxxx <fnuv@xxxxxxxxxx>; 
> brian.woods@xxxxxxxxxx <brian.woods@xxxxxxxxxx>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> masters in ZCU102 dev kit  
> We don't have a recipe ready. It is not something we test. However I
> think it should be possible.
> 
> The first step would be try to generate a BOOT.BIN for QSPI booting and
> write it to QSPI. The goal is to get to the u-boot prompt, with
> everything booted from QSPI. Then the next step is to add Xen, Linux,
> the DTB and Linux ramdisk. (Of course if you are OK with loading Xen and
> the other binaries from somewhere else like the SD card then it is going
> to work as is.)
> 
> You should be able to write each binary to QSPI as separate partitions.
> 
> At that point, you are only missing a boot.scr that can load them and
> boot. ImageBuilder supports custom u-boot load commands by setting the
> LOAD_CMD variable. In this case you'd need to set LOAD_CMD to something
> like "sf read", but given that the other command arguments are not
> filenames, ImageBuilder won't be able to handle it anyway.
> 
> But you can take an ImageBuilder-generated boot.source, substitute the
> "tftpb" commands with the appropriate "sf read" commands and call
> mkimage to get a boot.scr.
> 
> Now you should be able to run boot.scr from the u-boot prompt and load
> everything from QSPI.
> 
> This is some relevant info:
> 
> https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842465/U-Boot+QSPI+Driver
> 
> 
> I know it is a bit complex but I think it should work OK.
> 
> 
> 
> On Tue, 15 Jun 2021, Prem Harikrishnan wrote:
> > Hi Stefano,
> >
> > Is it possible to boot XEN from QSPI? Is there an image builder script 
> > already available to generate boot.scr? Please let me know.
> > Thanks and Best Regards,
> > Prem
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Xen-users <xen-users-bounces@xxxxxxxxxxxxxxxxxxxx> on behalf of Prem 
> > Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> > Sent: Tuesday, June 1, 2021 18:07
> > To: Mubin Usman Sayyed <MUBINUSM@xxxxxxxxxx>; Stefano Stabellini 
> > <stefanos@xxxxxxxxxx>
> > Cc: xen-users@xxxxxxxxxxxxxxxxxxxx <xen-users@xxxxxxxxxxxxxxxxxxxx>; Jovita 
> > Castelino <JOVITAC@xxxxxxxxxx>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> > masters in ZCU102 dev kit  
> > Hi Stefano and Mubin,
> >
> > Thank you very much for helping me setup up FreeRTOS and baremetal guests, 
> > everything works as expected and I also get expected
> performance
> > in both baremetal and hypervisor guests. Thank you again for your prompt 
> > support during the past few weeks.
> >
> > Best Regrards,
> > Prem
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Mubin Usman Sayyed <MUBINUSM@xxxxxxxxxx>
> > Sent: Tuesday, June 1, 2021 05:38
> > To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>; Stefano 
> > Stabellini <stefanos@xxxxxxxxxx>
> > Cc: xen-users@xxxxxxxxxxxxxxxxxxxx <xen-users@xxxxxxxxxxxxxxxxxxxx>; Jovita 
> > Castelino <JOVITAC@xxxxxxxxxx>
> > Subject: RE: Help: Information required for AXI DMA passthrough for PL 
> > masters in ZCU102 dev kit  
> >
> > In case of FreeRTOS,  you need to use APIs from FreeRTOS port for interrupt 
> > handling
> >  
> > (https://github.com/Xilinx/embeddedsw/blob/master/ThirdParty/bsp/freertos10_xilinx/examples/freertos_intr_example.c#L192
> >  and line 194).
> > Please make sure that you are taking care of that in DMA interrupt example .
> >
> >  
> >
> > Thanks,
> >
> > Mubin
> >
> >  
> >
> > From: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> > Sent: Tuesday, June 1, 2021 2:48 AM
> > To: Stefano Stabellini <stefanos@xxxxxxxxxx>
> > Cc: xen-users@xxxxxxxxxxxxxxxxxxxx; Jovita Castelino <JOVITAC@xxxxxxxxxx>; 
> > Mubin Usman Sayyed <MUBINUSM@xxxxxxxxxx>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> > masters in ZCU102 dev kit
> >
> >  
> >
> >  
> >
> > Hi Stefano and Mubin,
> >
> > Thank you very much for your replies.
> >
> > @Stefano Stabellini Sorry to bother you again, and thank you for pointing 
> > this out, the TTC0 is by default enabled for FreeRTOS to
> generate
> > ticks, although this says only for R5 core, i would have never figured this 
> > out. As soon as I added the irqs and iomem and gave a xen
> > passthrough in the xen.dtsi as you have suggested, FreeRTOS guest started 
> > working for other peripherals except for my DMA (FreeRTOS guest
> > does not work with interrupt and DMA).
> >
> >  
> >
> > I can see my DMA is enabled because i have LEDs configured to blink for 
> > every interrupt I get from the PL. I don't see my interrupts
> being
> > processed, but when I destroy the guest, I get this output periodically 
> > when an interrupt occurs.
> >
> >  
> >
> > (XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
> > (XEN) smmu: /smmu@fd800000:     GFSR 0x80000002, GFSYNR0 0x00000002, 
> > GFSYNR1 0x00000e80, GFSYNR2 0x00000000
> >
> > (XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
> > (XEN) smmu: /smmu@fd800000:     GFSR 0x80000002, GFSYNR0 0x00000002, 
> > GFSYNR1 0x00000e80, GFSYNR2 0x00000000
> >
> > (XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
> > (XEN) smmu: /smmu@fd800000:     GFSR 0x80000002, GFSYNR0 0x00000002, 
> > GFSYNR1 0x00000e80, GFSYNR2 0x00000000
> >
> >  
> >
> > From section 3.6 of the wiki 
> > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters
> >
> > Xen and PL Masters - Xilinx Wiki - Confluence
> >
> > 3.4 Stream IDs in the Device Tree. For Xen based systems the stream IDs of 
> > PL masters must be added to the Linux Dom0 device tree.
> > Automated device tree generation does not generate the stream IDs for the 
> > PL masters so that the user must add them into the device tree
> > manually.
> >
> > xilinx-wiki.atlassian.net
> >
> > The fault indicates an unidentified stream ID has been received by the SMMU 
> > as shown in the GFSR register. The GFSYNR1 register contains
> > the stream ID that was unidentified. A stream ID of 0xE80 was unidentified 
> > in the above fault.
> >
> >  
> >
> > But I think i have already added that in my smmu and iommu in xen.dtsi 
> > because my baremetal application works well as expected. I have
> > attached my xen.dtsi, system.dtsi, baremetal.cfg(working) and 
> > freertos.cfg(not working).
> >
> > What could I be missing here? Do I have to enable the ttc0 in the device 
> > tree?
> >
> > Also what does (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 99990 
> > KHz mean when XEN boots up?
> >
> > Thanks, I look forward for your reply.
> >
> >  
> >
> > Best Regards,
> >
> > Prem
> >
> >  
> >
> >  
> >
> >
> >  
> >________________________________________________________________________________________________________________________________________
> >
> >
> > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > Sent: Friday, May 28, 2021 20:43
> > To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> > Cc: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; 
> > xen-users@xxxxxxxxxxxxxxxxxxxx <xen-users@xxxxxxxxxxxxxxxxxxxx>;
> JOVITAC@xxxxxxxxxx
> > <JOVITAC@xxxxxxxxxx>; MUBINUSM@xxxxxxxxxx <MUBINUSM@xxxxxxxxxx>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> > masters in ZCU102 dev kit
> >
> >  
> >
> > From the gpa address, it looks like FreeRTOS is trying to access the TTC
> > timer? If you look at device tree, the corresponding node is
> > timer@ff110000.
> >
> > If FreeRTOS is configured to access the TTC timer, then you need to
> > assign it to it. Did you assign the TTC timer to the FreeRTOS domU by
> > any chance?
> >
> > You can do that by adding:
> >
> > irqs = [ 68, 69, 70 ]
> > iomem = [ "0xff110,1" ]
> >
> >
> > On Fri, 28 May 2021, Prem Harikrishnan wrote:
> > > Hi Stefano,
> > > Thanks for your reply. I am pretty sure I followed the steps you 
> > > mentioned below to run the FreeRTOS hello world example. I already
> added
> > > printfs and i can see XEN cannot start the scheduler and i get the 
> > > following out put on XEN.
> > >
> > > Parsing config from freertos.cfg (XEN) Dom2 colors: [ 8 ]
> > > (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=2 
> > > memflags=0xc0 (0 of 64)
> > > (XEN) memory_map:add: dom2 gfn=ff010 mfn=ff010 nr=1 cache=0
> > > (XEN) sched_null.c:344: 1 <-- d2v0
> > > (XEN) traps.c:1973:d2v0 HSR=0x93820007 pc=0x00000040012b08 gva=0xff11000c 
> > > gpa=0x000000ff11000c
> > >
> > >
> > > I also see that according to this article
> > > https://forums.xilinx.com/t5/Adaptable-Advantage-Blog/Partner-FreeRTOS-as-a-Xen-Virtual-Machine/ba-p/947030
> > > [52662iFAD1733707609204?v=v2]
> > > Partner: FreeRTOS as a Xen Virtual Machine - Community Forums
> > > Contributed by Jeff Kubascik, Embedded Systems Engineer at DornerWorks 
> > > What is FreeRTOS? The FreeRTOS kernel is a market leading
> > real-time
> > > operating system for embedded systems. As embedded applications grow in 
> > > complexity, there is a growing need for a lightweight kernel
> > > capable of task scheduling...
> > > forums.xilinx.com
> > > FreeRTOS does not run out of the box for XEN, although the article is 2 
> > > years old, I am not sure if there are any updates on this after
> > > that. Can you please confirm?
> > >
> > > Thanks and Best Regards,
> > > Prem
> > >
> >>_________________________________________________________________________________________________________________________________________
> _
> > _
> > > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > > Sent: Friday, May 28, 2021 02:51
> > > To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> > > Cc: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; 
> > > xen-users@xxxxxxxxxxxxxxxxxxxx <xen-users@xxxxxxxxxxxxxxxxxxxx>;
> > JOVITAC@xxxxxxxxxx
> > > <JOVITAC@xxxxxxxxxx>; MUBINUSM@xxxxxxxxxx <MUBINUSM@xxxxxxxxxx>
> > > Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> > > masters in ZCU102 dev kit  
> > > +Mubin
> > >
> > >
> > > Hi Prem,
> > >
> > > Did you make sure to build FreeRTOS as "hypervisor guest"? I think
> > > FreeRTOS is built to run at EL3 by default so if that is not changed
> > > there are errors when run on any hypervisor. It is possible to build
> > > FreeRTOS to run on Xen but it requires a couple of manual steps:
> > >
> > > - source settings64.sh from Vitis builds
> > > - xsct
> > > - set WORKSPACE .
> > > - setws -switch $WORKSPACE
> > > - app create -name freertos-hello-world -template {FreeRTOS Hello World} 
> > > -proc psu_cortexa53_0 -hw {xsa path} -os freertos10_xilinx
> > > - bsp config stdin psu_uart_1
> > > - bsp config stdout psu_uart_1
> > > - bsp config hypervisor_guest true
> > > - Go to application source path and modify the linker script start 
> > > address to 0x40000000,
> > > - app build -name freertos-hello-world
> > > - Run command to generate BIN file from executable 
> > > "aarch64-linux-gnu-objcopy -O binary --gap-fill 0  <elf path> BOOT.BIN"
> > >
> > > The resulting BOOT.BIN can be started a Xen DomU guest. Mubin, CC'ed,
> > > might be able to provide additional info.
> > >
> > >
> > >
> > > On Thu, 27 May 2021, Prem Harikrishnan wrote:
> > > > Hi Stefano,
> > > > Sorry to bother you again.
> > > > I am currently trying to run my custom IP which sends periodic dma 
> > > > interrupts to the PS with cache coloring enabled. I have now
> > compiled
> > > my
> > > > guest running with FreeRTOS and it generates a trap.
> > > > Petalinux and VITIS IDE Version is 2020.2.
> > > >
> > > > Please find the log attached. After the trap occurs, for every 
> > > > interrupt XEN prints (XEN) smmu: /smmu@fd800000: Unhandled context
> > fault:
> > > > fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > > >
> > > > root@pilbara:/etc/xen# xl create -c apu_freertos.cfg Parsing config 
> > > > from apu_freertos.cfg
> > > > (XEN) Dom1 colors: [ 8 ]
> > > > (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=1 
> > > > memflags=0xc0 (0 of 64)
> > > > (XEN) memory_map:add: dom1 gfn=ff010 mfn=ff010 nr=1 cache=0
> > > > (XEN) memory_map:add: dom1 gfn=a0010 mfn=a0010 nr=1 cache=0
> > > > (XEN) smmu: /smmu@fd800000: d1: p2maddr 0x000000087f554000
> > > > (XEN) sched_null.c:344: 1 <-- d1v0
> > > > (XEN) traps.c:1973:d1v0 HSR=0x93820007 pc=0x00000040013188 
> > > > gva=0xff11000c gpa=0x000000ff11000c
> > > > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, 
> > > > iova=0x41000000, fsynr=0x12, cb=1
> > > > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, 
> > > > iova=0x41000000, fsynr=0x12, cb=1
> > > > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, 
> > > > iova=0x41000000, fsynr=0x12, cb=1
> > > > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, 
> > > > iova=0x41000000, fsynr=0x12, cb=1
> > > >
> > > > The same application works as expected without FreeRTOS, i.e as a 
> > > > baremetal guest. I look forward for your comment on this.
> > > >
> > > > Although i see performance degradation, normally i can measure delta 
> > > > time for each interrupt configured to 1ms (without XEN), with
> XEN
> > > for
> > > > the same measurement shows 16 ms for each interval (but the performance 
> > > > is not a priority for now).
> > > > Thanks and Best Regards,
> > > > Prem
> > > >
> >>>________________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > > > Sent: Tuesday, May 18, 2021 20:09
> > > > To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> > > > Cc: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; 
> > > > xen-users@xxxxxxxxxxxxxxxxxxxx <xen-users@xxxxxxxxxxxxxxxxxxxx>;
> > > JOVITAC@xxxxxxxxxx
> > > > <JOVITAC@xxxxxxxxxx>
> > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> > > > masters in ZCU102 dev kit  
> > > > Great to hear! Thank you for reporting back.
> > > >
> > > >
> > > > On Tue, 18 May 2021, Prem Harikrishnan wrote:
> > > > > Dear Stefano,
> > > > > Thank you very much for your guidance and support. I finally fixed 
> > > > > the issue. I made an error in base memory address like you
> pointed
> > > > out.
> > > > > I actually modified the base address correctly to 0x40000000 but the 
> > > > > default C code had an additional offset of 0x1100000 , and the
> > > DomU
> > > > > memory allocated was only 8MB. I either had to change the offset or 
> > > > > allocate more memory to the DomU guest which i missed until you
> > > > pointed
> > > > > out. Additionally i had to set to AxPROT bits to 0x2 in Vivado and 
> > > > > the SMID required is 0xE80.
> > > > >
> > > > > Thanks again i was able get my custom IP also working as an AXI 
> > > > > master from PL.
> > > > >
> > > > > Best Regards,
> > > > > Prem
> > > > >
> >>>>_______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > > > > Sent: Tuesday, May 18, 2021 02:35
> > > > > To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> > > > > Cc: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; Brian Woods 
> > > > > <brian.woods@xxxxxxxxxx>; Julien Grall <julien@xxxxxxx>;
> Stefano
> > > > > Stabellini <sstabellini@xxxxxxxxxx>; xen-users@xxxxxxxxxxxxxxxxxxxx 
> > > > > <xen-users@xxxxxxxxxxxxxxxxxxxx>; fnuv@xxxxxxxxxx
> > > <fnuv@xxxxxxxxxx>;
> > > > > edgari@xxxxxxxxxx <edgari@xxxxxxxxxx>; bwoods288@xxxxxxxxx 
> > > > > <bwoods288@xxxxxxxxx>; Renato Costa Amorim
> <renato.amorim@xxxxxxxxxxxxxx>;
> > > > > JOVITAC@xxxxxxxxxx <JOVITAC@xxxxxxxxxx>
> > > > > Subject: Re: Help: Information required for AXI DMA passthrough for 
> > > > > PL masters in ZCU102 dev kit  
> > > > > Your configuration looks correct as far as I can tell.
> > > > >
> > > > > You need dtdev because it triggers the SMMU configuration for your PL
> > > > > block. You don't need device_tree because your baremetal application 
> > > > > is
> > > > > not going to read it, but it is not going to cause any issues either.
> > > > >
> > > > > When you say "the execution does not complete the loop at line 244", 
> > > > > it
> > > > > means that you see the execution entering the loop and then never
> > > > > leaving? Like an infinite loop?
> > > > >
> > > > > If so, then I imagine the issue could be that the application is
> > > > > crashing when trying to access TxBufferPtr, which is pointing to
> > > > > TX_BUFFER_BASE, which is:
> > > > >
> > > > >
> > > > > #ifndef DDR_BASE_ADDR
> > > > > #warning CHECK FOR THE VALID DDR ADDRESS IN XPARAMETERS.H, \
> > > > >                  DEFAULT SET TO 0x01000000
> > > > > #define MEM_BASE_ADDR           0x01000000
> > > > > #else
> > > > > #define MEM_BASE_ADDR           (DDR_BASE_ADDR + 0x1000000)
> > > > > #endif
> > > > >
> > > > > #define TX_BUFFER_BASE          (MEM_BASE_ADDR + 0x00100000)
> > > > >
> > > > >
> > > > >
> > > > > Can you double check that TX_BUFFER_BASE is pointing to a correct
> > > > > address? Memory starts at 0x40000000 in a Xen DomU.
> > > > >
> > > > > Maybe MEM_BASE_ADDR/DDR_BASE_ADDR is not set correctly?
> > > > >
> > > > >
> > > > > On Mon, 17 May 2021, Prem Harikrishnan wrote:
> > > > > > Hi Stefano,
> > > > > > Thanks again for your reply and your suggestion.  I did try to 
> > > > > > replace the Xil_DCacheFlushRange with
> > Xil_DCacheInvalidateRange(works
> > > > only
> > > > > > in EL3), I added printfs to debug, i can see that the execution 
> > > > > > does not complete the loop at line 244
> > > > > > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
> > > > > >   and
> > > > > > Xil_DCacheFlushRange is not reached. Please find attached my xen 
> > > > > > bootlog.txt.
> > > > > > I am still unable to find out why??
> > > > > >
> > > > > > Can you please atleast verify that my xen.dtsi, and pl.dtb is 
> > > > > > correct? and do i need dtdev and device_tree in my guest config
> file
> > > when
> > > > > it
> > > > > > is already added to xen.dtsi?
> > > > > >
> > > > > > Best Regards,
> > > > > > Prem
> > > > > >
> > > > > >
> >>>>>______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > _
> > > > > > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > > > > > Sent: Saturday, May 15, 2021 00:25
> > > > > > To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> > > > > > Cc: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; Brian Woods 
> > > > > > <brian.woods@xxxxxxxxxx>; Julien Grall <julien@xxxxxxx>;
> > Stefano
> > > > > > Stabellini <sstabellini@xxxxxxxxxx>; xen-users@xxxxxxxxxxxxxxxxxxxx 
> > > > > > <xen-users@xxxxxxxxxxxxxxxxxxxx>; fnuv@xxxxxxxxxx
> > > > <fnuv@xxxxxxxxxx>;
> > > > > > edgari@xxxxxxxxxx <edgari@xxxxxxxxxx>; bwoods288@xxxxxxxxx 
> > > > > > <bwoods288@xxxxxxxxx>; Renato Costa Amorim
> > <renato.amorim@xxxxxxxxxxxxxx>;
> > > > > > JOVITAC@xxxxxxxxxx <JOVITAC@xxxxxxxxxx>
> > > > > > Subject: Re: Help: Information required for AXI DMA passthrough for 
> > > > > > PL masters in ZCU102 dev kit  
> > > > > > Hi Prem,
> > > > > >
> > > > > > Some older versions of Xil_DCacheFlushRange were implemented using
> > > > > > deprecated set/way instructions that are known to cause problems in
> > > > > > virtualized environments.
> > > > > >
> > > > > > More recent implementations use the "dc civac" intruction that 
> > > > > > shouldn't
> > > > > > have any issues.
> > > > > >
> > > > > > Can you double-check that you are calling the 64-bit version 
> > > > > > implemented
> > > > > > by CIVAC?
> > > > > >
> >>>>>https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#
> L
> > 4
> > > 1
> > > > 4
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > > > Hi Stefano, 
> > > > > > > Thanks for your reply. Yes the axidma application is a baremetal 
> > > > > > > application from the xilinx vitis ide example. 
> > > > > > > Also yes, I have already run hello world as DomU guest, with 
> > > > > > > UART1.
> > > > > > >
> > > > > > > I am using the simple mode using polling and not interrupts(they 
> > > > > > > are disabled in the c code), which is Why I didn't include in
> > the
> > > > > guest
> > > > > > > config file. This works well when I compile it normally (as EL3). 
> > > > > > > When compiled as hypervisor guest, the execution stops before
> > the
> > > > > > > function xil_dcacheflushrange(). Please find the c code attached
> > > > > > > in 
> > > > > > > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
> > > > > > >  
> > > > > > >
> > > > > > > I found the SATA example similar to my use case, also using DMA 
> > > > > > > and SMMU. I just used it as a reference to create my pl.dtb to
> > pass
> > > > > > through
> > > > > > > in the guest config.(from pl.dtsi, which I manually created).
> > > > > > >
> > > > > > > Thanks and Best Regards,
> > > > > > > Prem
> > > > > > >
> > > > > > > Get Outlook for Android
> > > > > > >
> >>>>>>_____________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > _
> > > > > > _
> > > > > > > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > > > > > > Sent: Friday 14 May 2021, 23:16
> > > > > > > To: Prem Harikrishnan
> > > > > > > Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano 
> > > > > > > Stabellini; xen-users@xxxxxxxxxxxxxxxxxxxx; fnuv@xxxxxxxxxx;
> > > > > > edgari@xxxxxxxxxx;
> > > > > > > bwoods288@xxxxxxxxx; Renato Costa Amorim; JOVITAC@xxxxxxxxxx
> > > > > > > Subject: Re: Help: Information required for AXI DMA passthrough 
> > > > > > > for PL masters in ZCU102 dev kit
> > > > > > >
> > > > > > > I take that the guest kernel 
> > > > > > > "/bin/xaxidma_example_simple_poll_1.bin" is
> > > > > > > a baremetal application? Can you run it successfully, just 
> > > > > > > printing
> > > > > > > "hello world", if you only assign the UART and nothing else?
> > > > > > >
> > > > > > > I am asking just to verify that you can run the baremetal VM
> > > > > > > successfully, aside from the PL device assignment.
> > > > > > >
> > > > > > >
> > > > > > > In regards to the PL device assignment, I'll let Brian answer on 
> > > > > > > the
> > > > > > > Vivado configuration and SMID calculation. But I noticed that you 
> > > > > > > only
> > > > > > > assigned IRQ 54 to the DomU which is the one for UART1. You 
> > > > > > > haven't
> > > > > > > assigned any IRQs for axi_dma_0. Looking at 
> > > > > > > pl_from_petalinux.dtsi, it
> > > > > > > looks like there are two related IRQs: 121 and 122.
> > > > > > >
> > > > > > > Finally, why did you mention SATA passthrough in your reply? I 
> > > > > > > don't
> > > > > > > follow: I cannot see any SATA controller assignment in your
> > > > > > > configuration.
> > > > > > >
> > > > > > >
> > > > > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > > > > Dear Brian, Stefano and Julien,
> > > > > > > >
> > > > > > > > Thank you very much for your replies.
> > > > > > > > Hi Brian and Stefano, Can you please help me with the following 
> > > > > > > > questions?
> > > > > > > >
> > > > > > > > @Brian Woods Please find attached my vivado configuration, when 
> > > > > > > > i added my AXI DMA now running on simple polling
> > mode(previously
> > > i
> > > > > > tried
> > > > > > > it
> > > > > > > > with Scatter Gather mode), Run Automation automatically 
> > > > > > > > generated, 1 AXI Interconnect and 1 smart connect. Since there 
> > > > > > > > is
> only
> > 1
> > > > > master
> > > > > > > and
> > > > > > > > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for 
> > > > > > > > the stream IDs.
> > > > > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters
> > > > > > > >  According to this link section 6.5 I have
> > > > manually
> > > > > > tied
> > > > > > > > of AxPROT bits to value of 0x2 to create non secure 
> > > > > > > > transactions. Can you please confirm that my Vivado 
> > > > > > > > configuration is
> > correct?
> > > I
> > > > > can
> > > > > > > > update you soon with an ILA trace on Vivado.
> > > > > > > >
> > > > > > > > @Stefano Stabellini Please find attached my pl.dtsi generated 
> > > > > > > > from petalinux and system.dtsi also generated after manually
> > adding
> > > > > > > xen.dtsi.
> > > > > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2
> Using
> > > this
> > > > > > SATA
> > > > > > > > passthrough, i made my own pl.dtsi for passthrough used in 
> > > > > > > > dma.cfg. I could really use your help to identify what is 
> > > > > > > > missing
> in
> > > my
> > > > > xen
> > > > > > > > configuration.
> > > > > > > >
> > > > > > > > Thanks and Best Regards,
> > > > > > > > Prem
> > > > > > > >
> >>>>>>>____________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > _
> > > > > > _
> > > > > > > _
> > > > > > > > From: Brian Woods <brian.woods@xxxxxxxxxx>
> > > > > > > > Sent: Thursday, May 13, 2021 17:13
> > > > > > > > To: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > > > > > > > Cc: Julien Grall <julien@xxxxxxx>; Prem Harikrishnan 
> > > > > > > > <prem.harikrishnan@xxxxxxxxxxxxxx>; Stefano Stabellini
> > > > <sstabellini@xxxxxxxxxx>;
> > > > > > > > xen-users@xxxxxxxxxxxxxxxxxxxx 
> > > > > > > > <xen-users@xxxxxxxxxxxxxxxxxxxx>; brian.woods@xxxxxxxxxx 
> > > > > > > > <brian.woods@xxxxxxxxxx>;
> > fnuv@xxxxxxxxxx
> > > > > > > > <fnuv@xxxxxxxxxx>; edgari@xxxxxxxxxx <edgari@xxxxxxxxxx>; 
> > > > > > > > bwoods288@xxxxxxxxx <bwoods288@xxxxxxxxx>
> > > > > > > > Subject: Re: Help: Information required for AXI DMA passthrough 
> > > > > > > > for PL masters in ZCU102 dev kit  
> > > > > > > > Prem,
> > > > > > > >
> > > > > > > > Sorry for the late email, I've been under the weather and have 
> > > > > > > > been away
> > > > > > > > from my computer.  In the Zynq SOCs this is how the SMMU IDs in 
> > > > > > > > PL are
> > > > > > > > calculated:
> > > > > > > >
> > > > > > > > 14-10: TBU bits
> > > > > > > > 9-6: Master ID bits
> > > > > > > > 5-0: AXI ID bits
> > > > > > > >
> > > > > > > > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > > > > > > > To look up the TBU value, see UG1085 and look for the "System 
> > > > > > > > Masters"
> > > > > > > > table which is table 16-4 on page 409.
> > > > > > > >
> > > > > > > > For the master ID, see UG1085 and look for the "Master IDs 
> > > > > > > > List" table
> > > > > > > > which is table 16-13 on page 429.
> > > > > > > >
> > > > > > > > The AXI ID bits is a bit trickery since it's going to be out in 
> > > > > > > > PL.
> > > > > > > > See: https://www.xilinx.com/support/answers/69447.html
> > > > > > > > With SmartConnect all the AXI IDs are 0, they aren't passed on. 
> > > > > > > >  There
> > > > > > > > are work arounds.  Here's a couple of examples (with example 
> > > > > > > > numbers):
> > > > > > > >
> > > > > > > > AXI master -> PS slave port 0
> > > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > > 0000001
> > > > > > > >
> > > > > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > > 0000000
> > > > > > > >
> > > > > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > > > >                            -> PS slave port 1
> > > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > > 0000000
> > > > > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 
> > > > > > > > 0000000
> > > > > > > >
> > > > > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > > 0000000 (both share)
> > > > > > > >
> > > > > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > > > >                                 -> PS slave port 1
> > > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > > 0000000 (both share)
> > > > > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 
> > > > > > > > 0000000 (both share)
> > > > > > > >
> > > > > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS 
> > > > > > > > slave port 0
> > > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > > 0000001 (master #1)
> > > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > > 0000002 (master #2)
> > > > > > > >
> > > > > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS 
> > > > > > > > slave port 0
> > > > > > > >                                                         -> PS 
> > > > > > > > slave port 1
> > > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > > 0000001 (master #1)
> > > > > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 
> > > > > > > > 0000001 (master #1)
> > > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 
> > > > > > > > 0000002 (master #2)
> > > > > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 
> > > > > > > > 0000002 (master #2)
> > > > > > > >
> > > > > > > > The "sideband -> SmartConenct -> sideband" can also be 
> > > > > > > > functionally
> > > > > > > > replaced with "AXI Interconnect" to my knowledge.  So the AXI 
> > > > > > > > ID bits are
> > > > > > > > fairly simple but with some caveats.
> > > > > > > >
> > > > > > > > You can also read in the SMMU faults from Xen and use:
> > > > > > > > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > > > > > > > to decode them. That should tell you the SMMU IDs as well.
> > > > > > > >
> > > > > > > > Let me know if this isn't clear or you have any other questions.
> > > > > > > >
> > > > > > > > Brian
> > > > > > > >
> > > > > > > > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini 
> > > > > > > > wrote:
> > > > > > > > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > > > > > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > > > > > > > Hello everyone,
> > > > > > > > > >
> > > > > > > > > > Hi Prem,
> > > > > > > > > >
> > > > > > > > > > > I am using the latest version of the xilinx mpsoc zcu102 
> > > > > > > > > > > development board
> > > > > > > > > > > and i am trying to create a barematal application that 
> > > > > > > > > > > uses the AXI DMA on
> > > > > > > > > > > the programmable logic, using the scatter gather example 
> > > > > > > > > > > from the xilinx
> > > > > > > > > > > repository. This works perfectly well for the baremetal 
> > > > > > > > > > > application. The
> > > > > > > > > > > diagram below represents the information flow. The AXI 
> > > > > > > > > > > DMA and the AXI data
> > > > > > > > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is 
> > > > > > > > > > > out of the box IP
> > > > > > > > > > > from Vivado and uses a single channel (Does not use 
> > > > > > > > > > > AxUSER) which means the
> > > > > > > > > > > AXI ID cannot manually be set.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Now i want to make this as a baremetal hypervisor guest, 
> > > > > > > > > > > in future this will
> > > > > > > > > > > be replaced by a custom IP with cache colouring enabled. 
> > > > > > > > > > > I want to pass
> > > > > > > > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I followed this tutorial Baremetal + DomU
> > > > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > > > > > > > guest changed the base address to 0x4000000,(1GB 
> > > > > > > > > > > allocated to Dom-0) to
> > > > > > > > > > > create my EL1 binary. I used Petalinux 2020.2 to generate 
> > > > > > > > > > > my linux, xen
> > > > > > > > > > > image and use tftp to boot.
> > > > > > > > > > >
> > > > > > > > > > >  From what i understand from the xilinx tutorials 
> > > > > > > > > > >Xen+and+PL+Masters
> > > > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > > > > > > > section 3 and MPSOC SMMU
> > > > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > > > > > > > section 4, stream ids are required by the SMMU to get the 
> > > > > > > > > > > correct address
> > > > > > > > > > > translation.
> > > > > > > > > > >
> > > > > > > > > > > I am still unable to figure out how to get the iommu 
> > > > > > > > > > > property for my AXI DMA
> > > > > > > > > > > and how to set the dtdev config for my hypervisor guest.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I can also see some new updates with the XEN 2020.1 and 
> > > > > > > > > > > 2020.2 releases
> > > > > > > > > > > which i can be useful to my application
> > > > > > > > > > >
> > > > > > > > > > >   *   1:1 memory mappings for Xen virtual machines, 
> > > > > > > > > > >enabling device
> > > > > > > > > > >     assignments without SMMU
> > > > > > > > > > >   *
> > > > > > > > > > >       Static assignment of PL blocks, including 
> > > > > > > > > > >bus-mastering blocks, to
> > > > > > > > > > >     Xen virtual machines
> > > > > > > > > > >   *   Support for PL Device Passthrough to DomU in Xen
> > > > > > > > > >
> > > > > > > > > > You are usiong a version of Xen that has been customised by 
> > > > > > > > > > Xilinx. So I would
> > > > > > > > > > recommend to contact them directly.
> > > > > > > > > >
> > > > > > > > > > I have CCed Stefano who might be able to help you.
> > > > > > > > >
> > > > > > > > > Hi Julien,
> > > > > > > > >
> > > > > > > > > Many thanks for CC'ing me, I missed the email.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Prem,
> > > > > > > > >
> > > > > > > > > Let me CC a few people that might be able to help.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
> >
> 
>