[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: xen-swiotlb issue when NVMe driver is enabled in Dom0 on ARM


  • To: Stefano Stabellini <sstabellini@xxxxxxxxxx>
  • From: Rahul Singh <Rahul.Singh@xxxxxxx>
  • Date: Tue, 19 Apr 2022 13:36:49 +0000
  • Accept-language: en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Bnf9tjioEtVi95BBr9uz8icPnVlahpHkPLvXzsAuXXA=; b=d3lyquhdKfE0M17xzVSZ60Ff2COawT7c3tPUR49i3Oj7jV//AUxVaE08P8djRxskDDJllySAX/Mn0zMSbN8jaz3tz314M13rPsG9CMcRRnF2dyJWoc7ifOsKdaVnCJhRQvZ4E+dT/gt9jLilGD32nv18z6DjaimWj95v4iZke4cS8d200yFXXYGp7RccmuRBlKlYiOvih6RC65SdquKon/2k+J32e/fvsm2R18WrmZ7JkirBmnTwtfP9CIlbUWvnWb6ZTDbB7gsnd4fiF8INMsAoxbdcPj0D9jbt0j+RWfOj08JEBN2wjU8HnJYPk/NzcLIyxmuk4IoTsLvrkf60RQ==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UKik9FUDU4RMjRg07aikyhrOavfibutm/6BAtNljoSRcJAs0NC2vmWBakOrdzrHlS4AQOTMsiPGwvDjh7EhzYTlIapqVqwcZI8Fc4Cd9++cFrW3dICT37cXU0gyHps2h++ZIlCXv0w73ykvnC4MWqIjZ4RPI9bNwzeKKfCwoE/6Xg9bkItiEBDUU4CNoRo2Ts61sekyHFTV0ulE9KIWyQmmFbOMpgY7C9tsqwHUT5d4jWZC/fGfRlVTpK/OBay4owhIas9Y0UYCk365WrkomARuvsWEfvkT9rm2bpRkkkLR8650jXauSbh5t0QqDD+JOzvVzGUfbkkbyKalvcmbvzg==
  • Authentication-results-original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com;
  • Cc: Christoph Hellwig <hch@xxxxxx>, xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>, Bertrand Marquis <Bertrand.Marquis@xxxxxxx>, Julien Grall <julien@xxxxxxx>, Jan Beulich <jbeulich@xxxxxxxx>, "jgross@xxxxxxxx" <jgross@xxxxxxxx>, "boris.ostrovsky@xxxxxxxxxx" <boris.ostrovsky@xxxxxxxxxx>
  • Delivery-date: Tue, 19 Apr 2022 14:07:27 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Nodisclaimer: true
  • Original-authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com;
  • Thread-index: AQHYTzcUggEUWxM2006iot1qnYxlIKzuW0oAgAFU1ACAADDmgIAApv+AgAC5dQCAAp7ugIACQDIAgAEmAYA=
  • Thread-topic: xen-swiotlb issue when NVMe driver is enabled in Dom0 on ARM

Hi Stefano,

> On 18 Apr 2022, at 9:04 pm, Stefano Stabellini <sstabellini@xxxxxxxxxx> wrote:
>
> On Sun, 17 Apr 2022, Rahul Singh wrote:
>>> On 15 Apr 2022, at 6:40 pm, Stefano Stabellini <sstabellini@xxxxxxxxxx> wrote:
>>> On Fri, 15 Apr 2022, Christoph Hellwig wrote:
>>>> On Thu, Apr 14, 2022 at 01:39:23PM -0700, Stefano Stabellini wrote:
>>>>> OK, now we know that the code path with Xen is correct and it is the
>>>>> same code path taken (dma_alloc_direct) as when !CONFIG_XEN and !SMMU.
>>>>> That is how it should be.
>>>>>
>>>>> I cannot explain why dma_alloc_direct() would fail when called from
>>>>> xen_swiotlb_alloc_coherent(), but it would succeed when called from
>>>>> dma_alloc_attrs() without Xen.
>>>>>
>>>>> I am not aware of any restrictions that xen or swiotlb-xen would
>>>>> introduce in that regard. Unless you are just running out of memory
>>>>> because dom0_mem too low.
>>>>
>>>> The crash is deep down in the page allocator.  Even if memory was low
>>>> it should no crash.  So there is some odd interaction between Xen
>>>> and the page allocator going on.  I think nvme and dma-direct really
>>>> are only the messenger here.
>>>
>>>
>>> I cannot think of anything but if that is the case I guess it is more
>>> likely related to reserved-memory not properly advertised or ACPI tables
>>> not properly populated.
>>
>> I am not sure if it is true as we are able to boot with the same reserved memory or
>> the same ACPI table populated if we boot without swiotlb-xen dma ops.
>>
>>>
>>>
>>> Rahul,
>>>
>>> What happens if you boot Linux on Xen with swiotlb-xen disabled?
>>
>> Linux boots fine without any issue if we disable swiotlb-xen as mentioned below.
>
> The plot thinkens.
>
> Without swiotlb-xen, Linux boots fine. With swiotlb-xen it crashes.
> However, in both cases, the very same memory allocation function is
> used: dma_direct_alloc. In one case it works, in the other case it
> crashes.  Everything else is the same.
>
> There are a couple of questionable things with dma masks in
> xen_swiotlb_alloc_coherent, but they are *after* the call to
> xen_alloc_coherent_pages, which is the one that crashes. So they cannot
> be the cause of the crash.
>
> Before the call to xen_alloc_coherent_pages, there is only:
>
>  1) flags &= ~(__GFP_DMA | __GFP_HIGHMEM);
>  2) size = 1UL << (order + XEN_PAGE_SHIFT);
>
>
> 1) is already done by dma_alloc_attrs, so it is superfluous. I couldn't
> explain how 2) could possibly trigger the crash.  XEN_PAGE_SHIFT is
> always 12 even on 64K pages kernels. You can try removing 2) from
> xen_swiotlb_alloc_coherent, but we are really wandering in the dark
> here.

I tried removing the 2) but after that also issue remains.

>
> Then there is xen_swiotlb_init() which allocates some memory for
> swiotlb-xen at boot. It could lower the total amount of memory
> available, but if you disabled swiotlb-xen like I suggested,
> xen_swiotlb_init() still should get called and executed anyway at boot
> (it is called from arch/arm/xen/mm.c:xen_mm_init). So xen_swiotlb_init()
> shouldn't be the one causing problems.
>
> That's it -- there is nothing else in swiotlb-xen that I can think of.
>
> I don't have any good ideas, so I would only suggest to add more printks
> and report the results, for instance:

As suggested I added the more printks but only difference I see is the size apart
from that everything looks same .

Please find the attached logs for xen and native linux boot.



Regards
Rahul

Attachment: xen_boot_with_debug.log
Description: xen_boot_with_debug.log

Attachment: native_linux_boot_debug.log
Description: native_linux_boot_debug.log


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.