[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: xen-swiotlb issue when NVMe driver is enabled in Dom0 on ARM
Hi Stefano,
> On 18 Apr 2022, at 9:04 pm, Stefano Stabellini <sstabellini@xxxxxxxxxx> wrote: > > On Sun, 17 Apr 2022, Rahul Singh wrote: >>> On 15 Apr 2022, at 6:40 pm, Stefano Stabellini <sstabellini@xxxxxxxxxx> wrote: >>> On Fri, 15 Apr 2022, Christoph Hellwig wrote: >>>> On Thu, Apr 14, 2022 at 01:39:23PM -0700, Stefano Stabellini wrote: >>>>> OK, now we know that the code path with Xen is correct and it is the >>>>> same code path taken (dma_alloc_direct) as when !CONFIG_XEN and !SMMU. >>>>> That is how it should be. >>>>> >>>>> I cannot explain why dma_alloc_direct() would fail when called from >>>>> xen_swiotlb_alloc_coherent(), but it would succeed when called from >>>>> dma_alloc_attrs() without Xen. >>>>> >>>>> I am not aware of any restrictions that xen or swiotlb-xen would >>>>> introduce in that regard. Unless you are just running out of memory >>>>> because dom0_mem too low. >>>> >>>> The crash is deep down in the page allocator. Even if memory was low >>>> it should no crash. So there is some odd interaction between Xen >>>> and the page allocator going on. I think nvme and dma-direct really >>>> are only the messenger here. >>> >>> >>> I cannot think of anything but if that is the case I guess it is more >>> likely related to reserved-memory not properly advertised or ACPI tables >>> not properly populated. >> >> I am not sure if it is true as we are able to boot with the same reserved memory or >> the same ACPI table populated if we boot without swiotlb-xen dma ops. >> >>> >>> >>> Rahul, >>> >>> What happens if you boot Linux on Xen with swiotlb-xen disabled? >> >> Linux boots fine without any issue if we disable swiotlb-xen as mentioned below. > > The plot thinkens. > > Without swiotlb-xen, Linux boots fine. With swiotlb-xen it crashes. > However, in both cases, the very same memory allocation function is > used: dma_direct_alloc. In one case it works, in the other case it > crashes. Everything else is the same. > > There are a couple of questionable things with dma masks in > xen_swiotlb_alloc_coherent, but they are *after* the call to > xen_alloc_coherent_pages, which is the one that crashes. So they cannot > be the cause of the crash. > > Before the call to xen_alloc_coherent_pages, there is only: > > 1) flags &= ~(__GFP_DMA | __GFP_HIGHMEM); > 2) size = 1UL << (order + XEN_PAGE_SHIFT); > > > 1) is already done by dma_alloc_attrs, so it is superfluous. I couldn't > explain how 2) could possibly trigger the crash. XEN_PAGE_SHIFT is > always 12 even on 64K pages kernels. You can try removing 2) from > xen_swiotlb_alloc_coherent, but we are really wandering in the dark > here. I tried removing the 2) but after that also issue remains. > > Then there is xen_swiotlb_init() which allocates some memory for > swiotlb-xen at boot. It could lower the total amount of memory > available, but if you disabled swiotlb-xen like I suggested, > xen_swiotlb_init() still should get called and executed anyway at boot > (it is called from arch/arm/xen/mm.c:xen_mm_init). So xen_swiotlb_init() > shouldn't be the one causing problems. > > That's it -- there is nothing else in swiotlb-xen that I can think of. > > I don't have any good ideas, so I would only suggest to add more printks > and report the results, for instance: As suggested I added the more printks but only difference I see is the size apart from that everything looks same . Please find the attached logs for xen and native linux boot. Regards Rahul Attachment:
xen_boot_with_debug.log Attachment:
native_linux_boot_debug.log
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |