Re: [Xen-devel] Hikey: Enable Xen + Mainline Linux Kernel

[Julien Grall <julien.grall@xxxxxxxxx>]

On Tue, Jan 15, 2019 at 10:07 AM Leo Yan <leo.yan@xxxxxxxxxx> wrote:
>
> Hi all,

Hi Leo,

Thank you for the report.

> On Tue, Jan 15, 2019 at 10:49:58AM +0800, Leo Yan wrote:
>
> [...]
>
> > [    1.807591] Modules linked in:
> > [    1.810717] CPU: 4 PID: 1 Comm: swapper/0 Not tainted 
> > 5.0.0-rc2-00001-g5b47dea3757c #3
> > [    1.818691] Hardware name: HiKey Development Board (DT)
> > [    1.823983] pstate: 40000005 (nZcv daif -PAN -UAO)
> > [    1.828848] pc : xen_swiotlb_alloc_coherent+0x64/0x1e8
> > [    1.834044] lr : dma_alloc_attrs+0xf4/0x110
> > [    1.838289] sp : ffff000010073a50
> > [    1.841671] x29: ffff000010073a50 x28: 0000000000000007
> > [    1.847047] x27: ffff000011150068 x26: ffff80001b6ddd60
> > [    1.852429] x25: ffff000010caaa70 x24: 0000000000000000
> > [    1.857800] x23: 0000000000001000 x22: 0000000000001000
> > [    1.863177] x21: 0000000000000000 x20: ffff80001c2edc10
> > [    1.868553] x19: ffff0000111fd000 x18: ffffffffffffffff
> > [    1.873930] x17: 0000000000000000 x16: 0000000000000000
> > [    1.879306] x15: ffff0000111fd6c8 x14: ffff0000900737b7
> > [    1.884683] x13: ffff0000100737c5 x12: ffff000011215000
> > [    1.890060] x11: 0000000005f5e0ff x10: ffff0000111fd940
> > [    1.895436] x9 : 0000000000000000 x8 : ffff80001bb0e700
> > [    1.900813] x7 : 0000000000000000 x6 : 0000000000000000
> > [    1.906189] x5 : ffff0000105bdbb8 x4 : 0000000000000000
> > [    1.911566] x3 : 00000000006000c0 x2 : ffff80001b6ddd60
> > [    1.916943] x1 : 0000000000001000 x0 : 0000000000000000
> > [    1.922326] Process swapper/0 (pid: 1, stack limit = 0x(____ptrval____))
> > [    1.929084] Call trace:
> > [    1.931602]  xen_swiotlb_alloc_coherent+0x64/0x1e8
> > [    1.936456]  dma_alloc_attrs+0xf4/0x110
> > [    1.940359]  dmam_alloc_attrs+0x64/0xb8
> > [    1.944264]  dw_mci_probe+0x5f8/0xb00
> > [    1.947990]  dw_mci_pltfm_register+0xa0/0xd0
> > [    1.952327]  dw_mci_k3_probe+0x2c/0x38
>
> Some update for this issue after dig a bit for related code; with
> below simple hacking, the kernel can boot successfully to rootFS:
>
> diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
> index f6ded992c183..31c7e17f0fe5 100644
> --- a/include/linux/dma-mapping.h
> +++ b/include/linux/dma-mapping.h
> @@ -196,7 +196,8 @@ static inline int dma_mmap_from_global_coherent(struct 
> vm_area_struct *vma,
>
>  static inline bool dma_is_direct(const struct dma_map_ops *ops)
>  {
> -       return likely(!ops);
> +       return true;
> +       //return likely(!ops);
>  }
>
> Though this minor code tweaking can workaround the kernel panic, but
> it's not a formal fixing; if we look into the kernel code, we can see
> firstly the kernel will initialize dma operation pointer in
> arch/arm64/mm/dma-mapping.c, arch_setup_dma_ops():
>
> void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
>                         const struct iommu_ops *iommu, bool coherent)
> {
>         dev->dma_coherent = coherent;
>         __iommu_setup_dma_ops(dev, dma_base, size, iommu);
>
> #ifdef CONFIG_XEN
>         if (xen_initial_domain()) {
>                 dev->archdata.dev_dma_ops = dev->dma_ops;   // since 
> 'dev->dma_ops' is NULL,
>                                                             // so 
> dev->archdata.dev_dma_ops
>                                                             // will be 
> initialized as NULL
>                 dev->dma_ops = xen_dma_ops;
>         }
> #endif
> }
>
> So we can see 'dev->archdata.dev_dma_ops' will be NULL and
> 'dev->dma_ops' is assigned to xen_dma_ops;
>
> In dw mmc driver init function, it will run into below flow:
>
>   dw_mci_init_dma()
>     `> dmam_alloc_coherent()
>          `-> dmam_alloc_attrs()
>                `-> dma_alloc_attrs()
>                      `-> xen_dma_ops::alloc()
>                            `-> xen_swiotlb_alloc_coherent()
>                                  `-> xen_alloc_coherent_pages()
>                                        `-> xen_get_dma_ops()
>
> So xen_get_dma_ops() will try to retrieve pointer from
> 'dev->archdata.dev_dma_ops' but because it's NULL so at the end
> introduces kernel panic will NULL pointer.
>
> Seems to me, we should check two pointers in dma_is_direct(), one
> is 'dev->dma_ops' and another is 'dev->archdata.dev_dma_ops', should
> both of them are not NULL pointers then we can run into xen_alloc_xxx
> related function, otherwise it should fallback to use
> dma_direct_alloc() to allocate dma pages?
>
> Also very welcome if you could work on formal fixing and at my side
> I am glad to test it!

I actually reported a very similar bug on linux-iommu today [1]. It happens
to be an issue with the recent change in the DMA API. The IOMMU
maintainer suggested a patch that should fix both of our issues.
I haven't yet tried the patch [2]. Would you mind to give it go?

[1] https://lists.xen.org/archives/html/xen-devel/2019-01/msg01351.html
[2] https://lists.xen.org/archives/html/xen-devel/2019-01/msg01358.html

Best regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel