Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Thu, 22 Oct 2020, Julien Grall wrote:
> > > On 20/10/2020 16:25, Rahul Singh wrote:
> > > > Add support for ARM architected SMMUv3 implementations. It is based on
> > > > the Linux SMMUv3 driver.
> > > > Major differences between the Linux driver are as follows:
> > > > 1. Only Stage-2 translation is supported as compared to the Linux driver
> > > >     that supports both Stage-1 and Stage-2 translations.
> > > > 2. Use P2M  page table instead of creating one as SMMUv3 has the
> > > >     capability to share the page tables with the CPU.
> > > > 3. Tasklets is used in place of threaded IRQ's in Linux for event queue
> > > >     and priority queue IRQ handling.
> > > 
> > > Tasklets are not a replacement for threaded IRQ. In particular, they will
> > > have priority over anything else (IOW nothing will run on the pCPU until
> > > they are done).
> > > 
> > > Do you know why Linux is using thread. Is it because of long running
> > > operations?
> > 
> > Yes you are right because of long running operations Linux is using the
> > threaded IRQs.
> > 
> > SMMUv3 reports fault/events bases on memory-based circular buffer queues not
> > based on the register. As per my understanding, it is time-consuming to
> > process the memory based queues in interrupt context because of that Linux
> > is using threaded IRQ to process the faults/events from SMMU.
> > 
> > I didn’t find any other solution in XEN in place of tasklet to defer the
> > work, that’s why I used tasklet in XEN in replacement of threaded IRQs. If
> > we do all work in interrupt context we will make XEN less responsive.
> 
> So we need to make sure that Xen continue to receives interrupts, but we also
> need to make sure that a vCPU bound to the pCPU is also responsive.
> 
> > 
> > If you know another solution in XEN that will be used to defer the work in
> > the interrupt please let me know I will try to use that.
> 
> One of my work colleague encountered a similar problem recently. He had a long
> running tasklet and wanted to be broken down in smaller chunk.
> 
> We decided to use a timer to reschedule the taslket in the future. This allows
> the scheduler to run other loads (e.g. vCPU) for some time.
> 
> This is pretty hackish but I couldn't find a better solution as tasklet have
> high priority.
> 
> Maybe the other will have a better idea.

Julien's suggestion is a good one.

But I think tasklets can be configured to be called from the idle_loop,
in which case they are not run in interrupt context?

Still, tasklets run until completion in Xen, which could take too long.
The code has to voluntarily release control of the execution flow once
it realizes it has been running for too long. The rescheduling via a
timer works.


Now, to brainstorm other possible alternatives, for hypercalls we have
been using hypercall continuations.  Continuations is a way to break a
hypercall implementation that takes too long into multiple execution
chunks. It works by calling into itself again: making the same hypercall
again with updated arguments, so that the scheduler has a chance to do
other operations in between, including running other tasklets and
softirqs.

That works well because  the source of the work is a guest request,
specifically a hypercall. However, in the case of the SMMU driver, there
is no hypercall. The Xen driver has to do work in response to an
interrupt and the work is not tied to one particular domain.

So I don't think the hypercall continuation model could work here. The
timer seems to be the best option.


> > > > 4. Latest version of the Linux SMMUv3 code implements the commands queue
> > > >     access functions based on atomic operations implemented in Linux.
> > > 
> > > Can you provide more details?
> > 
> > I tried to port the latest version of the SMMUv3 code than I observed that
> > in order to port that code I have to also port atomic operation implemented
> > in Linux to XEN. As latest Linux code uses atomic operation to process the
> > command queues (atomic_cond_read_relaxed(),atomic_long_cond_read_relaxed() ,
> > atomic_fetch_andnot_relaxed()) .
> 
> Thank you for the explanation. I think it would be best to import the atomic
> helpers and use the latest code.
> 
> This will ensure that we don't re-introduce bugs and also buy us some time
> before the Linux and Xen driver diverge again too much.
> 
> Stefano, what do you think?

I think you are right.