Xen project Mailing List

Hi Julien,

Thanks for taking a look, and please excuse any faux paus of mailing list etiquette I am about to commit...

On Wed, Jun 22, 2016 at 10:43 AM, Julien Grall <julien.grall@xxxxxxx> wrote:

Hello Jarvis,

I have CCed Edgar who did the bring of Xen on Xilinx Zinqmp

On 21/06/16 15:40, Jarvis Roach wrote:

I've been working on passing an ethernet controller through to a bare
metal guest running on an Xilinx Z US+ MPSoC platform. I got it working,
but only by using the system's physical address (PA) for DMA buffers and
descriptors instead of using the guest's intermediate physical address
(IPA), aka guest phyiscal address.

I am aware of Julien's presentation on passthrough, and I am successful
in passing the ethernet device through to a Linux guest, but when I
follow the same steps for my bare metal guest, the IPA->PA translation
doesn't seem to be happening. I would expect that if the SMMU was active
for that ethernet controller then a) using PA for DMA buffers and
descriptors should result in a translation fault, and b) using IPA for
the same would work, neither of which is true.

Can you provide the guest configuration you are using?

I'll start with the revelant snippets. Please let me know if you'd like to see the entire files.

Relevant xen.dts:

smmu@fd800000 {

compatible = "arm,mmu-500";

reg = <0x0 0xfd800000 0x0 0x20000>;

#global-interrupts = <0x1>;

interrupt-parent = <0x1>;

interrupts = <0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4 0x0 0x9b 0x4>;

mmu-masters = <&ser1 0x876 &ttc1 0x877 &gem3 0x878>;

};

gem3:ethernet@ff0e0000 {

#stream-id-cells = <0x1>;

compatible = "cdns,gem";

status = "disabled";

interrupt-parent = <0x1>;

interrupts = <0x0 0x3f 0x4 0x0 0x3f 0x4>;

reg = <0x0 0xff0e0000 0x1000>;

clock-names = "pclk", "hclk", "tx_clk";

#address-cells = <0x1>;

#size-cells = <0x0>;

jumbo-max-len = <0x2800>;

jumbo-supported;

clocks = <0x3 0x3 0x3>;

phy-mode = "rgmii-id";

xlnx,ptp-enet-clock = <0x0>;

local-mac-address = [00 0a 35 00 22 01];

phy-handle = <0x4>;

xen,passthrough = <0x1>;

linux,phandle = <0x12>;

phandle = <0x12>;

phy@21 {

reg = <0x15>;

ti,rx-internal-delay = <0x8>;

ti,tx-internal-delay = <0xa>;

ti,fifo-depth = <0x1>;

linux,phandle = <0x4>;

phandle = <0x4>;

};

Guest configuration:

name = "bare1"

kernel = "bare.img"

memory = 8

vcpus = 1

cpus = [1]

dtdev = [ "/amba/serial@ff010000", "/amba/ethernet@ff0e0000", "/amba/timer@ff110000" ]

#device_tree = "partial.dtb"

irqs = [ 54, 95, 68, 69, 70 ]

iomem = [ "0xff010,1", "0xff0e0,1", "0xff110,1" ]

Is the partial.dtb only for the guest's benefit? I had assumed that was the case, and since my guest didn't speak device-tree I left it out initially, but I did go back and try it with a partial just in case there was some addtional configuration triggered by the presence of it in the .cfg file without any apparent change in behavior.

It seems to me that there is something extra is being done by, or for,
the Linux guest that isn't done by, or for, my bare metal guest that
enables the SMMU for that ethernet controller. I haven't been able to
figure what that something extra is by searching the internet, reading
through the available documentation, or digging into the code. Any help
(better search terms, useful URLs, detailed description of the what's
supposed to happen for guests generically or what actually happens for a
Linux guest, a working bare metal example, etc) would be appreciated.

The SMMU is configured with Stage-2 page table and not exposed to the guest. I am not aware of any configuration required in the guest, except having a working network driver.

"Working" can be a tricky term to nail down. The driver "works", as in I can telnet/ping/nc to my target if I pass in PAs instead of IPAs to the DMA, but maybe "working" involves some step that the PV-aware Linux guest is doing that I'm not; that something occurs as a by-product of Linux guest behavior, like maybe a page grant that occurs as part of a DMA buffer allocation, that completes the SMMU configuration, but I haven't been able to find anything like yet. If there's any documentation on what guests should be doing in this regard I'd welcome a reference to it.

Regards,

--
Julien Grall

Thanks again for your time and attention.

-J

Re: [Xen-users] Enabling SMMU for a bare metal guest