Re: xen-swiotlb issue when NVMe driver is enabled in Dom0 on ARM

[Rahul Singh <Rahul.Singh@xxxxxxx>]

Hi Stefano,

> On 15 Apr 2022, at 6:40 pm, Stefano Stabellini <sstabellini@xxxxxxxxxx> wrote:
> 
> On Fri, 15 Apr 2022, Christoph Hellwig wrote:
>> On Thu, Apr 14, 2022 at 01:39:23PM -0700, Stefano Stabellini wrote:
>>> OK, now we know that the code path with Xen is correct and it is the
>>> same code path taken (dma_alloc_direct) as when !CONFIG_XEN and !SMMU.
>>> That is how it should be.
>>> 
>>> I cannot explain why dma_alloc_direct() would fail when called from
>>> xen_swiotlb_alloc_coherent(), but it would succeed when called from
>>> dma_alloc_attrs() without Xen.
>>> 
>>> I am not aware of any restrictions that xen or swiotlb-xen would
>>> introduce in that regard. Unless you are just running out of memory
>>> because dom0_mem too low.
>> 
>> The crash is deep down in the page allocator.  Even if memory was low
>> it should no crash.  So there is some odd interaction between Xen
>> and the page allocator going on.  I think nvme and dma-direct really
>> are only the messenger here.
> 
> 
> I cannot think of anything but if that is the case I guess it is more
> likely related to reserved-memory not properly advertised or ACPI tables
> not properly populated.

I am not sure if it is true as we are able to boot with the same reserved 
memory or
the same ACPI table populated if we boot without swiotlb-xen dma ops.

> 
> 
> Rahul,
> 
> What happens if you boot Linux on Xen with swiotlb-xen disabled?

Linux boots fine without any issue if we disable swiotlb-xen as mentioned below.
> 
> 
> diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
> index 93e87b287556..26d9da58f2c6 100644
> --- a/arch/arm64/mm/dma-mapping.c
> +++ b/arch/arm64/mm/dma-mapping.c
> @@ -51,9 +51,4 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, 
> u64 size,
>       dev->dma_coherent = coherent;
>       if (iommu)
>               iommu_setup_dma_ops(dev, dma_base, size);
> -
> -#ifdef CONFIG_XEN
> -     if (xen_initial_domain())
> -             dev->dma_ops = &xen_swiotlb_dma_ops;
> -#endif
> }
> 
> 

Regards,
Rahul