Re: Xen on RP4

[Roman Shaposhnik <roman@xxxxxxxxxx>]

On Tue, Nov 24, 2020 at 9:17 PM Elliott Mitchell <ehem+xen@xxxxxxx> wrote:
>
> On Tue, Nov 24, 2020 at 08:45:32PM -0800, Roman Shaposhnik wrote:
> > On Tue, Nov 24, 2020 at 8:41 PM Elliott Mitchell <ehem+xen@xxxxxxx> wrote:
> > >
> > > On Tue, Nov 24, 2020 at 08:01:32PM -0800, Roman Shaposhnik wrote:
> > > > On Tue, Nov 24, 2020 at 7:37 PM Elliott Mitchell <ehem+xen@xxxxxxx> 
> > > > wrote:
> > > > > Presently I'm using a 5.8 kernel with your patches and haven't seen
> > > > > graphical output under Xen with either boot stack.  I've confirmed 
> > > > > fully
> > > > > operational graphics without Xen on Tianocore, I've confirmed 
> > > > > operational
> > > > > virtual terminals with U-Boot and not Xen.
> > > > >
> > > > > I had been planning to wait a bit before moving to 5.9, but if that is
> > > > > the crucial ingredient I'll move early.
> > > >
> > > > We're still using 5.4 -- but it seems that the next LTS 5.10 is also 
> > > > functional.
> > > >
> > > > I can bet $10 whatever it is -- it is DT related ;-)
> > >
> > > Given how many of the pieces I'm assembling are alpha or beta level, I
> > > estimate a 50:50 chance on that.  Good odds it is device-tree, but good
> > > odds I grabbed a bad version of $something.
> > >
> > > I mostly wanted to know whether I was in completely uncharted territory
> > > and needed to wait for others to catch up, versus merely working in a
> > > situation where support is funky and I'm at an unknown location in
> > > charted territory.
> > >
> > > I'll be keeping the Tianocore setup available since Xen on ARM really
> > > /should/ allow ACPI...
> >
> > I don't think you're in the uncharted -- so perhaps a bit of debugging left.
> >
> > And, of course, alway feel free to compare what we do -- the image is
> > docker pull away.
>
> Actually, since device-tree is very much on my list of concerns, what is
> your Xen boot process setup like?
>
> Presently as previously mentioned I'm trying for
> U-Boot -> GRUB/EFI -> Xen.

Exactly the same. Here's what we put on a vfat partition:
     https://github.com/lf-edge/eve/tree/master/pkg/u-boot/rpi
and here's how u-boot is built:
     https://github.com/lf-edge/eve/blob/master/pkg/u-boot/Dockerfile

> According to the information I currently have
> the device-trees are often tied pretty closely to the kernel.  I'm also
> using GRUB 2.04 since that has proper support for loading Xen on ARM.

Yes. Our DT here:
    
https://github.com/lf-edge/eve/blob/master/pkg/u-boot/rpi/bcm2711-rpi-4-b.dtb
came from an honest build of our kernel (our build is still in flux -- hence
a quick hack of keeping a blob):
    https://github.com/lf-edge/eve/blob/master/pkg/kernel/Dockerfile#L154

> The section of grub.cfg for Linux is roughly:
>     linux /boot/vmlinuz-5.8.10-2rp4-6.1.7 
> root=UUID=01234567-dead-beef-d13d-456789abcdef ro
>     devicetree /boot/dtb-5.8.10-2rp4-6.1.7
>     initrd /boot/initrd.img-5.8.10-2rp4-6.1.7
>
> My testing section for Xen is:
>     xen_hypervisor /boot/xen-4.14-arm64.efi
>     xen_module /boot/vmlinuz-5.8.10-2rp4-6.1.7 
> root=UUID=01234567-dead-beef-d13d-456789abcdef ro
>     devicetree /boot/dtb-5.8.10-2rp4-6.1.7
>     xen_module --nounzip /boot/initrd.img-5.8.10-2rp4-6.1.7

Roughly the same -- but see Stefano's comment. More here:
    https://github.com/lf-edge/eve/blob/master/pkg/grub/rootfs.cfg

> I've frankly got no idea how to ensure the correct device-tree is passed
> to Xen.  Is GRUB's `devicetree` command correct when loading Xen?  Is a
> device-tree matched to the Linux kernel appropriate for Xen?
>
> (I'm guessing the second is "yes", but the first I don't have a clue)

While you can use `devicetree`  in GRUB. E.g.:
    https://github.com/lf-edge/eve/blob/master/pkg/grub/rootfs.cfg#L207
on EVE side we've only ever used it as an emergency override.

The happy path boot sequence preserves the DT that RPi bootloader
makes available to u-boot and it gets passed down the chain without
anybody doing anything.

Hope this helps.

Thanks,
Roman.