Re: [Xen-devel] Question about DMA on 1:1 mapping dom0 of arm64

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

On Sat, 2015-04-18 at 00:41 +0800, Chen Baozi wrote:
> On Fri, Apr 17, 2015 at 02:21:45PM +0100, Ian Campbell wrote:
> > On Fri, 2015-04-17 at 19:24 +0800, Chen Baozi wrote:
> > > Hi all,
> > > 
> > > According to my recent experience, there might be some problems of swiotlb
> > > dma map on 1:1 mapping arm64 dom0 with large memory. The issue is like 
> > > below:
> > > 
> > > For those arm64 server with large memory, it is possible to set dom0_mem >
> > > 4G (e.g. I have one set with 16G). In this case, according to my 
> > > understanding,
> > > there is chance that the dom0 kernel needs to map some buffers above 4G 
> > > to do
> > 
> >                                                                  ^below?
> > 
> > > DMA operations (e.g. in snps,dwmac ethernet driver). However, most DMA 
> > > engines
> > > support only 32-bit physical address, thus aren't able to operate 
> > > directly on
> > > those memory.
> > 
> > Even on arm64 systems with RAM above 4GB? That seems.... short-sighted.
> > Oh well, I suppose we have to live with it.
> 
> I understand for most ARM SoCs, the DMA engines come from third party IP 
> companies
> which is arm32/arm64 independent. Thus, 32-bit address DMA engine should be 
> common
> even on arm64 system.

Yes, I suppose that will be true, but I stand by my "shortsighted"
comment ;-). (What's the point of a DMA engine which can only access 1/4
of the system's RAM and therefore requires bounce buffering before it
can be used...)

>  The preferred way is to use/enable SMMU(IOMMU). However, we
> are focusing on 1:1 mapping right now...
> 
> > 
> > >  IIUC, swiotlb is implemented to solve this (using bounce buffer),
> > > if there is no IOMMU or IOMMU is not enabled on the system. Sadly, it 
> > > seems
> > > that xen_swiotlb_map_page in my dom0 kernel allocates
> > > (start_dma_addr = 0x944800000) the buffers for DMA above 4G which fails
> > > dma_capable() checking and was then unable to return from 
> > > xen_swiotlb_map_page()
> > > successfully.
> > 
> > The swiotlb bounce buffer have been allocated below 4GB? 
> 
> I have no idea (about the exact behavior of bounce buffer). But I don't think
> it has been allocated below 4GB on my board, for in that case it won't fail
> dma_capable() in the end of xen_swiotlb_map_page().
> 
> > I suspect that
> > xen_swiotlb_init is buggy for ARM -- it allocates some random pages and
> > then swizzles the backing pages for ones < 4G, but that won't work on an
> > ARM dom0 with a 1:1 mapping, I don't think. Do you see error messages
> > along those lines?
> > 
> > Essentially I think either xen_swiotlb_fixup is unable to work on ARM,
> > or the following:
> >         start_dma_addr = xen_virt_to_bus(xen_io_tlb_start);
> > is returning 1:1 and not reflecting the fixup.
> 
> Yes. It seems very likely what happened in my system.

Stefano suggested that xen_swiotlb_fixup is a NOP on arm (which doesn't
surprise me, that made sense until this issue was identified), which
will be the root cause here.

> > > If I set dom0_mem to a small value (e.g. 512M), which makes all physical 
> > > memory
> > > of dom0 below 4G, everything goes fine.
> > 
> > So you are getting allocated memory below 4G?
> 
> If all the banks of memory that xen populate to dom0 is below 4G, yes. 
> However,
> if some banks of memory for dom0 is above 4G, usually not.
> 
> > 
> > You message on IRC suggested you weren't, did you hack around this?
> 
> Yes. I did some hacks to help understand my situation earlier. What I have 
> done
> and observed is as below:
> 
> 1. At the very beginning, I used the default dom0_mem value to boot the 
> system,
> which is 128M. And I didn't realize the DMA buffer problem.
> 
> 2. I started to try more dom0_mem (16G). Then the ethernet driver reported
> that it cannot initiate rx buffers (DMA buffers). And I found out that
> allocate_memory_11 didn't populate any banks of memory below 4G for dom0.
> At that time, I guessed the failure might be introduced because there is no
> memory banks below 4G was populated. (there is only a 2GB address space below 
> 4G
> for physical memory on my platform, and there is a hole for PCI memory address
> space above 4G before the memory address space continue.)
> 
> 3. So I did some hacks to let lowmem=true manually in allocate_memory_11, 
> which
> made xen on arm64 acts similar as it is on arm32 that populates at least one
> bank of memory below 4G to dom0. (this is the point when I send you message
> on IRC.) I thought that can solve the problem, but it doesn't.
> 
> 4. Then I found out once xen populated any banks of memory which is above 4G,
> the ethernet driver would have chances (very likely, almost every time if
> dom0_mem=16G) to use buffers above 4G, regardless whether dom0 has banks of
> memory below 4G.
> 
> > 
> > I think we have two options, either xen_swiotlb_init allocates pages
> > below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> > to actually work even on a 1:1 dom0.
> > 
> > Although the first option seems preferable at first glance it has the
> > short coming that it requires dom0 to have some memory below 4GB, which
> > might not necessarily be the case. The second option seems like it might
> > be uglier but doesn't suffer from this issue.
> > 
> > Can you please look and find out if the IPA at 0x944800000 is actually
> > backed by 1:1 RAM or if xen_swiotlb_fixup has done it's job and updated
> > things such that the associated PAs are below 4GB?
> 
> I am at home now and will check it out tomorrow. But I guess it should be
> the first situation you mentioned.

I think given the information Stefano has provided there's likely not
much point in this experiment, since we now know that the fixup function
is a nop on ARM.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel