Re: Troubles running Xen on Raspberry Pi 4 with 5.6.1 DomU

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Wed, 13 May 2020, Julien Grall wrote:
> On 13/05/2020 19:19, Julien Grall wrote:
> > Hi,
> > 
> > On 13/05/2020 16:11, Stefano Stabellini wrote:
> > > On Wed, 13 May 2020, Julien Grall wrote:
> > > > Hi,
> > > > 
> > > > On 13/05/2020 01:33, Stefano Stabellini wrote:
> > > > > I worked with Roman to do several more tests and here is an update on
> > > > > the situation. We don't know why my patch didn't work when Boris'
> > > > > patch
> > > > > [1] worked.  Both of them should have worked the same way.
> > > > > 
> > > > > Anyway, we continued with Boris patch to debug the new mmc error which
> > > > > looks like this:
> > > > > 
> > > > > [    3.084464] mmc0: unrecognised SCR structure version 15
> > > > > [    3.089176] mmc0: error -22 whilst initialising SD card
> > > > > 
> > > > > I asked to add a lot of trancing, see attached diff.
> > > > 
> > > > Please avoid attachment on mailing list and use pastebin for diff.
> > > > 
> > > > > We discovered a bug
> > > > > in xen_swiotlb_init: if io_tlb_start != 0 at the beginning of
> > > > > xen_swiotlb_init, start_dma_addr is not set correctly. This oneline
> > > > > patch fixes it:
> > > > > 
> > > > > diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
> > > > > index 0a40ac332a4c..b75c43356eba 100644
> > > > > --- a/drivers/xen/swiotlb-xen.c
> > > > > +++ b/drivers/xen/swiotlb-xen.c
> > > > > @@ -191,6 +191,7 @@ int __ref xen_swiotlb_init(int verbose, bool
> > > > > early)
> > > > >            * IO TLB memory already allocated. Just use it.
> > > > >            */
> > > > >           if (io_tlb_start != 0) {
> > > > > +               start_dma_addr = io_tlb_start;
> > > > >                   xen_io_tlb_start = phys_to_virt(io_tlb_start);
> > > > >                   goto end;
> > > > >           }
> > > > > 
> > > > > Unfortunately it doesn't solve the mmc0 error.
> > > > > 
> > > > > 
> > > > > As you might notice from the logs, none of the other interesting
> > > > > printks
> > > > > printed anything, which seems to mean that the memory allocated by
> > > > > xen_swiotlb_alloc_coherent and mapped by xen_swiotlb_map_page should
> > > > > be
> > > > > just fine.
> > > > > 
> > > > > I am starting to be out of ideas. Do you guys have any suggestions on
> > > > > what could be the issue or what could be done to discover it?
> > > > 
> > > > Sorry if my suggestions are going to be obvious, but I can't confirm
> > > > whether
> > > > this was already done:
> > > >      1) Does the kernel boot on baremetal (i.e without Xen)? This should
> > > > help
> > > > to confirm whether the bug is Xen is related.
> > > 
> > > Yes it boots
> > > 
> > > >      2) Swiotlb should not be necessary for basic dom0 boot on Arm. Did
> > > > you try
> > > > to disable it? This should help to confirm whether swiotlb is the
> > > > problem or
> > > > not.
> > > 
> > > It boots disabling swiotlb-xen
> > 
> > Thank you for the answer! swiotlb-xen should basically be a NOP for dom0. So
> > this suggests swiotlb is doing some transformation on the DMA address.
> > 
> > I have an idea what may have gone wrong. AFAICT, xen-swiotlb seems to assume
> > the DMA address space and CPU address space is the same. Is RPI using the
> > same address space?
> 
> Another question, is the DMA request bounced? If so, are we sure the bounce
> buffer is in the first GB?

Yes, it is. This is actually where we spent the last few days, and I
found another little related bug in the initialization of the
swiotlb-xen but now I am sure the memory is under 1GB (0x34000000-0x38000000)