Re: [Xen-devel] Question about DMA on 1:1 mapping dom0 of arm64

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxxxxx>]

On Fri, 17 Apr 2015, Chen Baozi wrote:
> Hi all,
> 
> According to my recent experience, there might be some problems of swiotlb
> dma map on 1:1 mapping arm64 dom0 with large memory. The issue is like below:
> 
> For those arm64 server with large memory, it is possible to set dom0_mem >
> 4G (e.g. I have one set with 16G). In this case, according to my 
> understanding,
> there is chance that the dom0 kernel needs to map some buffers above 4G to do
> DMA operations (e.g. in snps,dwmac ethernet driver). However, most DMA engines
> support only 32-bit physical address, thus aren't able to operate directly on
> those memory. IIUC, swiotlb is implemented to solve this (using bounce 
> buffer),
> if there is no IOMMU or IOMMU is not enabled on the system. Sadly, it seems
> that xen_swiotlb_map_page in my dom0 kernel allocates
> (start_dma_addr = 0x944800000) the buffers for DMA above 4G which fails
> dma_capable() checking and was then unable to return from 
> xen_swiotlb_map_page()
> successfully.
>
> If I set dom0_mem to a small value (e.g. 512M), which makes all physical 
> memory
> of dom0 below 4G, everything goes fine.
> 
> I am not familiar with swiotlb-xen, so there would be misunderstanding about
> the current situation. Fix me if I did/understood anything wrong.
> 
> Any ideas?

I think that the problem is that xen_swiotlb_init doesn't necessarely allocate
memory under 4G on arm/arm64.

xen_swiotlb_init calls __get_free_pages to allocate memory, so the pages
could easily be above 4G.  Subsequently xen_swiotlb_fixup is called on
the allocated memory range, calling xen_create_contiguous_region and
passing an address_bits mask. However xen_create_contiguous_region
doesn't actually do anything at all on ARM.

I think that given that dom0 is mapped 1:1 on ARM, the easiest and best
fix would be to simply allocate memory under 4G to begin with. Something
like (maybe with an ifdef ARM around it):

diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index 810ad41..22ac33a 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -235,7 +235,7 @@ retry:
 #define SLABS_PER_PAGE (1 << (PAGE_SHIFT - IO_TLB_SHIFT))
 #define IO_TLB_MIN_SLABS ((1<<20) >> IO_TLB_SHIFT)
                while ((SLABS_PER_PAGE << order) > IO_TLB_MIN_SLABS) {
-                       xen_io_tlb_start = (void 
*)__get_free_pages(__GFP_NOWARN, order);
+                       xen_io_tlb_start = (void 
*)__get_free_pages(__GFP_NOWARN|__GFP_DMA32, order);
                        if (xen_io_tlb_start)
                                break;
                        order--;

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel