[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver

  • To: Stefano Stabellini <sstabellini@xxxxxxxxxx>
  • From: Rahul Singh <Rahul.Singh@xxxxxxx>
  • Date: Fri, 23 Oct 2020 11:35:00 +0000
  • Accept-language: en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AGEHtB9rNId9OdJGuNVRoSMtOawmeS4J9+jlHDhH3iU=; b=dzZsgL4X8O5/QJJ9L5j5+CxfcKTxwt3OjtVXZcF9tGviQKgcEA/xdgKCQJxvQ9haQcyajrpaIVyEPirfOiP7tJWqfzmj6F+dITl08/JEhJ/r3jXqR5+JbJlJmOj6M+6lGoizgE5qdkMdPJq4g7AI8h0ktG3SLjypGcorh8KjmZoP3YC5fjS3xCshk7Yul6ILAId7DjEkQ4BFo0vkWIzelFfcWo5jT9I8uzsLWgMxajHcawh7t8xLmlXRYKxxRvgX5adz7XPRf1M8RwxMiov1qqRtnrVDAMBry7nNo11SXpb/NAAX80cx0nMJlnOVhK30PYuVDYN9TsU8mFkXa90LSg==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aOsPaO781USbVjFuCN3O882eDeSVE73gr3L6BheiN8xoM8RdTABCmbVoZPepjG0JtfEXuK2Rl8PEasdbRAh1CJRqVzkwDAG0FjBN+7sM4XXaRXH5EUpITBvcjdzCL5gYKiy8h9lzKZOg/S9zeT1ckDjJcWi2d0UvaLZpQ2D+cUrYr89HSOmTKNmNEHcHppVR72sWF5t8MfJ0YrfDpawCr1d2AkycTT4EPtx9QSWwHmk1ksBj4shulM2ZgwCOGi4FuQXlVgBk8muM3+ZCAP+Ng33BjAu4ZFHT3PQYtQAAxx9HuOZEx+2HeMCgIAhJhWveeIXObrkny1oSXu6KAxlx4g==
  • Authentication-results-original: kernel.org; dkim=none (message not signed) header.d=none;kernel.org; dmarc=none action=none header.from=arm.com;
  • Cc: Julien Grall <julien@xxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxxx>, Bertrand Marquis <Bertrand.Marquis@xxxxxxx>, Jan Beulich <jbeulich@xxxxxxxx>, Paul Durrant <paul@xxxxxxx>, Volodymyr Babchuk <Volodymyr_Babchuk@xxxxxxxx>
  • Delivery-date: Fri, 23 Oct 2020 11:35:39 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Nodisclaimer: true
  • Original-authentication-results: kernel.org; dkim=none (message not signed) header.d=none;kernel.org; dmarc=none action=none header.from=arm.com;
  • Thread-index: AQHWoVvrkmJwOYERdUOadvid1OghFamgw0AAgAEzsQCAAWIugIABA+CAgADBXIA=
  • Thread-topic: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver


> On 23 Oct 2020, at 1:02 am, Stefano Stabellini <sstabellini@xxxxxxxxxx> wrote:
> On Thu, 22 Oct 2020, Julien Grall wrote:
>>>> On 20/10/2020 16:25, Rahul Singh wrote:
>>>>> Add support for ARM architected SMMUv3 implementations. It is based on
>>>>> the Linux SMMUv3 driver.
>>>>> Major differences between the Linux driver are as follows:
>>>>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>>>>>    that supports both Stage-1 and Stage-2 translations.
>>>>> 2. Use P2M  page table instead of creating one as SMMUv3 has the
>>>>>    capability to share the page tables with the CPU.
>>>>> 3. Tasklets is used in place of threaded IRQ's in Linux for event queue
>>>>>    and priority queue IRQ handling.
>>>> Tasklets are not a replacement for threaded IRQ. In particular, they will
>>>> have priority over anything else (IOW nothing will run on the pCPU until
>>>> they are done).
>>>> Do you know why Linux is using thread. Is it because of long running
>>>> operations?
>>> Yes you are right because of long running operations Linux is using the
>>> threaded IRQs.
>>> SMMUv3 reports fault/events bases on memory-based circular buffer queues not
>>> based on the register. As per my understanding, it is time-consuming to
>>> process the memory based queues in interrupt context because of that Linux
>>> is using threaded IRQ to process the faults/events from SMMU.
>>> I didn’t find any other solution in XEN in place of tasklet to defer the
>>> work, that’s why I used tasklet in XEN in replacement of threaded IRQs. If
>>> we do all work in interrupt context we will make XEN less responsive.
>> So we need to make sure that Xen continue to receives interrupts, but we also
>> need to make sure that a vCPU bound to the pCPU is also responsive.
>>> If you know another solution in XEN that will be used to defer the work in
>>> the interrupt please let me know I will try to use that.
>> One of my work colleague encountered a similar problem recently. He had a 
>> long
>> running tasklet and wanted to be broken down in smaller chunk.
>> We decided to use a timer to reschedule the taslket in the future. This 
>> allows
>> the scheduler to run other loads (e.g. vCPU) for some time.
>> This is pretty hackish but I couldn't find a better solution as tasklet have
>> high priority.
>> Maybe the other will have a better idea.
> Julien's suggestion is a good one.
> But I think tasklets can be configured to be called from the idle_loop,
> in which case they are not run in interrupt context?

 Yes you are right tasklet will be scheduled from the idle_loop that is not 
interrupt conext.

> Still, tasklets run until completion in Xen, which could take too long.
> The code has to voluntarily release control of the execution flow once
> it realizes it has been running for too long. The rescheduling via a
> timer works.
> Now, to brainstorm other possible alternatives, for hypercalls we have
> been using hypercall continuations.  Continuations is a way to break a
> hypercall implementation that takes too long into multiple execution
> chunks. It works by calling into itself again: making the same hypercall
> again with updated arguments, so that the scheduler has a chance to do
> other operations in between, including running other tasklets and
> softirqs.
> That works well because  the source of the work is a guest request,
> specifically a hypercall. However, in the case of the SMMU driver, there
> is no hypercall. The Xen driver has to do work in response to an
> interrupt and the work is not tied to one particular domain.
> So I don't think the hypercall continuation model could work here. The
> timer seems to be the best option.

Yes, I agree with you as the source of the work is not a guest request in the 
case of SMMU I think we can not use the hyper call continuation.

As suggested I will try to use the timer to schedule the work and will share 
the findings.
>>>>> 4. Latest version of the Linux SMMUv3 code implements the commands queue
>>>>>    access functions based on atomic operations implemented in Linux.
>>>> Can you provide more details?
>>> I tried to port the latest version of the SMMUv3 code than I observed that
>>> in order to port that code I have to also port atomic operation implemented
>>> in Linux to XEN. As latest Linux code uses atomic operation to process the
>>> command queues (atomic_cond_read_relaxed(),atomic_long_cond_read_relaxed() ,
>>> atomic_fetch_andnot_relaxed()) .
>> Thank you for the explanation. I think it would be best to import the atomic
>> helpers and use the latest code.
>> This will ensure that we don't re-introduce bugs and also buy us some time
>> before the Linux and Xen driver diverge again too much.
>> Stefano, what do you think?
> I think you are right.

Yes, I agree with you to have XEN code in sync with Linux code that's why I 
started with to port the Linux atomic operations to XEN  then I realised that 
it is not straightforward to port atomic operations and it requires lots of 
effort and testing. Therefore I decided to port the code before the atomic 
operation is introduced in Linux.




Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.