Xen project Mailing List

On Wed, Jun 22, 2016 at 12:37 PM, Edgar E. Iglesias <edgar.iglesias@xxxxxxxxxx> wrote:

On Wed, Jun 22, 2016 at 12:11:53PM -0400, Jarvis Roach wrote:
> Hi Julien,
>
> Thanks for taking a look, and please excuse any faux paus of mailing list
> etiquette I am about to commit...
>
>
> On Wed, Jun 22, 2016 at 10:43 AM, Julien Grall <julien.grall@xxxxxxx> wrote:
>
> > Hello Jarvis,
> >
> > I have CCed Edgar who did the bring of Xen on Xilinx Zinqmp
>
>
> >
> > On 21/06/16 15:40, Jarvis Roach wrote:
> >
> >> I've been working on passing an ethernet controller through to a bare
> >> metal guest running on an Xilinx Z US+ MPSoC platform. I got it working,
> >> but only by using the system's physical address (PA) for DMA buffers and
> >> descriptors instead of using the guest's intermediate physical address
> >> (IPA), aka guest phyiscal address.
> >>
> >> I am aware of Julien's presentation on passthrough, and I am successful
> >> in passing the ethernet device through to a Linux guest, but when I
> >> follow the same steps for my bare metal guest, the IPA->PA translation
> >> doesn't seem to be happening. I would expect that if the SMMU was active
> >> for that ethernet controller then a) using PA for DMA buffers and
> >> descriptors should result in a translation fault, and b) using IPA for
> >> the same would work, neither of which is true.
> >>
> >
> > Can you provide the guest configuration you are using?
>
>
> I'll start with the revelant snippets. Please let me know if you'd like to
> see the entire files.
>
> Relevant xen.dts:
>
> smmu@fd800000 {
> compatible = "arm,mmu-500";
> reg = <0x0 0xfd800000 0x0 0x20000>;
> #global-interrupts = <0x1>;
> interrupt-parent = <0x1>;
> interrupts = <0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b
> 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0
> 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4
> 0x0 0x9b 0x4>;
> mmu-masters = <&ser1 0x876 &ttc1 0x877 &gem3 0x878>;

Hi Jarvis,

These don't look correct.

These are the GEM master-ids:
&gem0 0x874 &gem1 0x875 &gem2 0x876 &gem3 0x877

The serial ports and ttc timers are not DMA capable and shouldn't be listed there.
Also these master-ids are tied to a device in HW, it's not possible to edit/change these in SW or DTS at choice.

Here's a list the DTS in linux:
mmu-masters = < &gem0 0x874
&gem1 0x875
&gem2 0x876
&gem3 0x877
&usb0 0x860
&usb1 0x861
&qspi 0x873
&lpd_dma_chan1 0x868
&lpd_dma_chan2 0x869
&lpd_dma_chan3 0x86a
&lpd_dma_chan4 0x86b
&lpd_dma_chan5 0x86c
&lpd_dma_chan6 0x86d
&lpd_dma_chan7 0x86e
&lpd_dma_chan8 0x86f
&fpd_dma_chan1 0x14e8
&fpd_dma_chan2 0x14e9
&fpd_dma_chan3 0x14ea
&fpd_dma_chan4 0x14eb
&fpd_dma_chan5 0x14ec
&fpd_dma_chan6 0x14ed
&fpd_dma_chan7 0x14ee
&fpd_dma_chan8 0x14ef
&sdhci0 0x870
&sdhci1 0x871
&nand0 0x872>;

Best regards,
Edgar

>
> };
> .
> .
> .
> gem3:ethernet@ff0e0000 {
> #stream-id-cells = <0x1>;
> compatible = "cdns,gem";
> status = "disabled";
> interrupt-parent = <0x1>;
> interrupts = <0x0 0x3f 0x4 0x0 0x3f 0x4>;
> reg = <0x0 0xff0e0000 0x1000>;
> clock-names = "pclk", "hclk", "tx_clk";
> #address-cells = <0x1>;
> #size-cells = <0x0>;
> jumbo-max-len = <0x2800>;
> jumbo-supported;
> clocks = <0x3 0x3 0x3>;
> phy-mode = "rgmii-id";
> xlnx,ptp-enet-clock = <0x0>;
> local-mac-address = [00 0a 35 00 22 01];
> phy-handle = <0x4>;
> xen,passthrough = <0x1>;
> linux,phandle = <0x12>;
> phandle = <0x12>;
>
> phy@21 {
> reg = <0x15>;
> ti,rx-internal-delay = <0x8>;
> ti,tx-internal-delay = <0xa>;
> ti,fifo-depth = <0x1>;
> linux,phandle = <0x4>;
> phandle = <0x4>;
> };
> };
>
>
>
> Guest configuration:
>
> name = "bare1"
> kernel = "bare.img"
> memory = 8
> vcpus = 1
> cpus = [1]
>
> dtdev = [ "/amba/serial@ff010000", "/amba/ethernet@ff0e0000",
> "/amba/timer@ff110000" ]
> #device_tree = "partial.dtb"
> irqs = [ 54, 95, 68, 69, 70 ]
> iomem = [ "0xff010,1", "0xff0e0,1", "0xff110,1" ]
>
>
> Is the partial.dtb only for the guest's benefit? I had assumed that was the
> case, and since my guest didn't speak device-tree I left it out initially,
> but I did go back and try it with a partial just in case there was some
> addtional configuration triggered by the presence of it in the .cfg file
> without any apparent change in behavior.
>
>
> >> It seems to me that there is something extra is being done by, or for,
> >> the Linux guest that isn't done by, or for, my bare metal guest that
> >> enables the SMMU for that ethernet controller. I haven't been able to
> >> figure what that something extra is by searching the internet, reading
> >> through the available documentation, or digging into the code. Any help
> >> (better search terms, useful URLs, detailed description of the what's
> >> supposed to happen for guests generically or what actually happens for a
> >> Linux guest, a working bare metal example, etc) would be appreciated.
> >>
> >
> > The SMMU is configured with Stage-2 page table and not exposed to the
> > guest. I am not aware of any configuration required in the guest, except
> > having a working network driver.
>
>
> >
> "Working" can be a tricky term to nail down. The driver "works", as in I
> can telnet/ping/nc to my target if I pass in PAs instead of IPAs to the
> DMA, but maybe "working" involves some step that the PV-aware Linux guest
> is doing that I'm not; that something occurs as a by-product of Linux guest
> behavior, like maybe a page grant that occurs as part of a DMA buffer
> allocation, that completes the SMMU configuration, but I haven't been able
> to find anything like yet. If there's any documentation on what guests
> should be doing in this regard I'd welcome a reference to it.
>
>
>
> > Regards,
> >
> > --
> > Julien Grall
> >
>
> Thanks again for your time and attention.
>
> -J

To summarize for posterity, after working with Edgar offline, it turned out that I did not have the passthrough for my Linux guest working after all. The reason behind that were some TrustZone register settings that had to be set differently, which Edgar graciously provided to me. While I haven't had the opportunity to do the same for my bare metal guest yet (the cause of my original post) I'm feeling pretty good that it'll work. I'll write again if it turns out *not* to be the case (in case anyone else is having the same problem). Big thanks to Edgar for helping out.

Regards,

-Jarvis

Re: [Xen-users] Enabling SMMU for a bare metal guest