Re: Troubles running Xen on Raspberry Pi 4 with 5.6.1 DomU

[Roman Shaposhnik <roman@xxxxxxxxxx>]

Hi Stefano!

On Fri, May 1, 2020 at 5:05 PM Stefano Stabellini
<sstabellini@xxxxxxxxxx> wrote:
>
> Hi Roman,
>
>
> In regards to the attached stack trace, nothing rings a bell
> unfortunately. I don't know why quirk_usb_early_handoff causes a crash.
> It would be useful to add a few printk in quirk_usb_early_handoff to
> know where the crash is happening exactly.

That definitely seems to be a cascading error from our DMA issues.
Basically when I completely disable DMA (as I showed before) this
issue completely goes away.

> In regards to Dornerworks's third patch, it doesn't look like it is
> related to the quirk_usb_early_handoff crash. The third patch is
> probably not useful anymore because dev->archdata.dev_dma_ops is gone
> completely.

Yup. That seems to be correct. Applying your WARN_ON(dev->dma_ops != NULL);
patch didn't show anything on the console -- so at least that is not
an issue anymore.

I think the focus really should be on:
    
https://github.com/dornerworks/xen-rpi4-builder/blob/master/patches/linux/0002-Disable-DMA-in-sdhci-driver.patch

But even this patch, I think, is a cascading error from something that is still
somehow broken about DMA handling in Xen. IOW, it may very well be the
case that the reason Dornerworks folks had to add:
    SDHCI_QUIRK_BROKEN_DMA | SDHCI_QUIRK_BROKEN_ADMA
is exactly the same reason we're now seeing DMA issues much earlier.

Do you think this is something that we could tackle? Like I mentioned,
having an upstream, 5.x kernel capable of being a Dom0 on RPi 4
would be really a huge deal to the Xen on ARM development community.

It seems that we're within reach of that -- I just wish I knew in
which direction
to take it.

Thanks,
Roman.

> However, just in case, something like the following would
> help recognize if the original bug still persists in newer kernels
> somehow:
>
>
> diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
> index 6c45350e33aa..61af12d79add 100644
> --- a/arch/arm64/mm/dma-mapping.c
> +++ b/arch/arm64/mm/dma-mapping.c
> @@ -53,7 +53,9 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, 
> u64 size,
>                 iommu_setup_dma_ops(dev, dma_base, size);
>
>  #ifdef CONFIG_XEN
> -       if (xen_initial_domain())
> +       if (xen_initial_domain()) {
> +               WARN_ON(dev->dma_ops != NULL);
>                 dev->dma_ops = &xen_swiotlb_dma_ops;
> +       }
>  #endif
>  }
>
>
> On Thu, 30 Apr 2020, Roman Shaposhnik wrote:
> > Hi!
> >
> > I'm trying to run Xen on Raspberry Pi 4 with 5.6.1 stock,
> > upstream kernel. The kernel itself works perfectly well
> > on the board. When I try booting it as Dom0 under Xen,
> > it goes into a stacktrace (attached).
> >
> > Looking at what nice folks over at Dornerworks have previously
> > done to make RPi kernels boot as Dom0 I've come across these
> > 3 patches:
> >     
> > https://github.com/dornerworks/xen-rpi4-builder/tree/master/patches/linux
> >
> > The first patch seems irrelevant (unless I'm missing something
> > really basic here). The 2nd patch applied with no issue (but
> > I don't think it is related) but the 3d patch failed to apply on
> > account of 5.6.1 kernel no longer having:
> >     dev->archdata.dev_dma_ops
> > E.g.
> >     
> > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/mm/dma-mapping.c?h=v5.6.1#n55
> >
> > I've tried to emulate the effect of the patch by simply introducing
> > a static variable that would signal that we already initialized
> > dev->dma_ops -- but that didn't help at all.
> >
> > I'm CCing Jeff Kubascik to see if the original authors of that Corey Minyard
> > to see if may be they have any suggestions on how this may be dealt
> > with.
> >
> > Any advice would be greatly appreciated!
> >
> > Thanks,
> > Roman.
> >