Re: Troubles running Xen on Raspberry Pi 4 with 5.6.1 DomU

[Corey Minyard <minyard@xxxxxxx>]

On Sat, May 02, 2020 at 12:46:14PM +0100, Julien Grall wrote:
> Hi,
> 
> On 02/05/2020 03:16, Corey Minyard wrote:
> > On Fri, May 01, 2020 at 06:06:11PM -0700, Roman Shaposhnik wrote:
> > > On Fri, May 1, 2020 at 4:42 AM Corey Minyard <minyard@xxxxxxx> wrote:
> > > > 
> > > > On Thu, Apr 30, 2020 at 07:20:05PM -0700, Roman Shaposhnik wrote:
> > > > > Hi!
> > > > > 
> > > > > I'm trying to run Xen on Raspberry Pi 4 with 5.6.1 stock,
> > > > > upstream kernel. The kernel itself works perfectly well
> > > > > on the board. When I try booting it as Dom0 under Xen,
> > > > > it goes into a stacktrace (attached).
> > > > 
> > > > Getting Xen working on the Pi4 requires a lot of moving parts, and they
> > > > all have to be right.
> > > 
> > > Tell me about it! It is a pretty frustrating journey to align
> > > everything just right.
> > > On the other hand -- it seems worth to enable RPi as an ARM development
> > > platform for Xen given how ubiquitous it is.
> > > 
> > > Hence me trying to combine pristine upstream kernel (5.6.1) with
> > > pristine upstream
> > > Xen to enable 100% upstream developer workflow on RPi.
> > > 
> > > > > Looking at what nice folks over at Dornerworks have previously
> > > > > done to make RPi kernels boot as Dom0 I've come across these
> > > > > 3 patches:
> > > > >      
> > > > > https://github.com/dornerworks/xen-rpi4-builder/tree/master/patches/linux
> > > > > 
> > > > > The first patch seems irrelevant (unless I'm missing something
> > > > > really basic here).
> > > > 
> > > > It might be irrelevant for your configuration, assuming that Xen gets
> > > > the right information from EFI.  I haven't tried EFI booting.
> > > 
> > > I'd doing a bit of belt-and-suspenders strategy really -- I'm actually
> > > using UEFI u-boot implementation that pre-populates device trees
> > > and then I'm also forcing an extra copy of it to be load explicitly
> > > via the GRUB devicetree command (GRUB runs as a UEFI payload).
> > > 
> > > I also have access to the semi-official TianoCore RPi4 port that seems
> > > to be working pretty well: https://github.com/pftf/RPi4/releases/tag/v1.5
> > > for booting all sort of UEFI payloads on RPi4.
> > > 
> > > > > The 2nd patch applied with no issue (but
> > > > > I don't think it is related) but the 3d patch failed to apply on
> > > > > account of 5.6.1 kernel no longer having:
> > > > >      dev->archdata.dev_dma_ops
> > > > > E.g.
> > > > >      
> > > > > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/mm/dma-mapping.c?h=v5.6.1#n55
> > > > > 
> > > > > I've tried to emulate the effect of the patch by simply introducing
> > > > > a static variable that would signal that we already initialized
> > > > > dev->dma_ops -- but that didn't help at all.
> > > > > 
> > > > > I'm CCing Jeff Kubascik to see if the original authors of that Corey 
> > > > > Minyard
> > > > > to see if may be they have any suggestions on how this may be dealt
> > > > > with.
> > > > > 
> > > > > Any advice would be greatly appreciated!
> > > > 
> > > > What's your Pi4 config.txt file look like? The GIC is not being handled
> > > > correctly, and I'm guessing that configuration is wrong.  You can't use
> > > > the stock config.txt file with Xen, you have to modify the 
> > > > configuration a
> > > > bit.
> > > 
> > > Understood. I'm actually using a custom one:
> > >      https://github.com/lf-edge/eve/blob/master/pkg/u-boot/rpi/config.txt
> > > 
> > > I could swear that I had a GIC setting in it -- but apparently no -- so
> > > I added the following at the top of what you could see at the URL above:
> > > 
> > > total_mem=4096
> > > enable_gic=1
> > > 
> > > > I think just adding:
> > > > 
> > > > enable_gic=1
> > > > total_mem=1024
> > > 
> > > Right -- but my board has 4G memory -- so I think what I did above should 
> > > work.
> > 
> > Nope.  If you say 4096M of RAM, your issue is almost certainly DMA, but
> > it's not (just) the Linux code.  On the Raspberry Pi 4, several devices
> > cannot DMA to above 1024M of RAM, but Xen does not handle this.  The
> > 1024M of RAM is a limitation you will have to live with until Xen has a
> > fix.
> 
> IIUC, dom0 would need to have some memory below 1GB for this to work, am I
> correct?

No.  If I am understanding this correctly, all the memory in dom0 below
1GB would have to be physically below 1GB.

The Linux patch set starts at:

https://lore.kernel.org/linux-iommu/20191015174616.GO13874@xxxxxxxxxxxxxxxxxxxx/t/

There is also an interesting reference at:

https://www.raspberrypi.org/forums/viewtopic.php?t=264975

-corey

> 
> If so could you try the following patch?
> 
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index 430708753642..002f49dba74b 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -282,7 +282,7 @@ static void __init allocate_memory_11(struct domain *d,
>       */
>      while ( order >= min_low_order )
>      {
> -        for ( bits = order ; bits <= (lowmem ? 32 : PADDR_BITS); bits++ )
> +        for ( bits = order ; bits <= (lowmem ? 30 : PADDR_BITS); bits++ )
>          {
>              pg = alloc_domheap_pages(d, order, MEMF_bits(bits));
>              if ( pg != NULL )
> @@ -313,7 +313,7 @@ static void __init allocate_memory_11(struct domain *d,
>      order = get_allocation_size(kinfo->unassigned_mem);
>      while ( kinfo->unassigned_mem && kinfo->mem.nr_banks < NR_MEM_BANKS )
>      {
> -        pg = alloc_domheap_pages(d, order, lowmem ? MEMF_bits(32) : 0);
> +        pg = alloc_domheap_pages(d, order, lowmem ? MEMF_bits(30) : 0);
>          if ( !pg )
>          {
>              order --;
> 
> 
> Cheers,
> 
> -- 
> Julien Grall