Re: Xen data from meta-virtualization layer

[Julien Grall <julien.grall.oss@xxxxxxxxx>]

Hi Michael,

On Fri, 4 Feb 2022 at 22:42, Michael Walle <michael@xxxxxxxx> wrote:
> Am 2022-02-04 22:11, schrieb Stefano Stabellini:
> > On Fri, 4 Feb 2022, Michael Walle wrote:
> >> > In regards to the reserved-memory regions, maybe we are not seeing them
> >> > because Leo posted the host device tree, not the one passed at runtime
> >> > from u-boot to Linux?
> >> >
> >> > If so, Leo, could you please boot Linux on native (no Xen) and get the
> >> > device tree from there at runtime using dtc -I fs -O dts
> >> > /proc/device-tree ?
> >> >
> >> >
> >> > However, the name of the reserved-memory region created by u-boot seems
> >> > to be "lpi_rd_table". I cannot find any mentions of lpi_rd_table in the
> >> > Linux kernel tree either.
> >> >
> >> > Zhiqiang, Leo is trying to boot Xen on sAL28. Linux booting on Xen
> >> > throws errors in regards to GIC/ITS initialization. On other hardware
> >> > Xen can use and virtualize GICv3 and ITS just fine. Could you please
> >> > explain what is different about sAL28 and how Xen/Linux is expected to
> >> > use the lpi_rd_table reserved-memory region?
> >>
> >> I actually stumbled across this thread after trying out Xen myself.
> >> I'm
> >> using lastest vanilla u-boot (with pending PSCI patches), vanilla
> >> kernel
> >> and vanilla Xen.
> >>
> >> So far I've discovered, that xen complains that it cannot route IRQ64
> >> to
> >> dom0. That is because on the LS1028A there is a dual UART (two 16550
> >> with one shared interrupt) and xen takes the first UART and then tries
> >> to map the interrupt of the second UART to linux. For now, I don't
> >> know
> >> how this is solved correctly. As a quick hack, I removed the second
> >> uart
> >> node from the device tree.
> >
> > This is an interesting problem. Removing the second UART is a good
> > workaround for now as there is no obvious solution I think.
>
> But not a very user friendly one, though. I guess the first UART
> is disabled/removed by Xen? I haven't looked at how it is handled.
>
> Can't we search for other uarts with the same interrupt and disable
> these, too? Maybe conditionally by the SoC compatible?

The problem sounds quite similar to the one we had on sunxi. Although
the UART was on the same page rather than sharing interrupts.

Xen has per-platform hook to prevent a device been assigned
to dom0. For an example, you could look at:

https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/arm/platforms/sunxi.c

> >> This is the first developer experience with Xen, so please bear with
> >> me
> >> :) It seems that Xen doesn't add the master to the IOMMU. To me it
> >> seems
> >> that only devices with a 'iommus' dt property are added. But in case
> >> of
> >> PCI devices the parent only has a iommu-map property.
> >>
> >> And it makes me wonder why Leo has an almost working setup. Maybe I'm
> >> missing some patches though.
> >
> > Xen 4.16 is able to parse StreamID in the "iommus" property and also
> > "mmu-masters" property. But It is not able to parse the "iommu-map"
> > property yet. So if 0x42a is described in device tree using "iommu-map"
> > then the error makes sense.
> >
> > A simple solution is to replace iommu-map with iommus in device tree.
>
> I'm not sure this is so easy, because they are dynamically assigned
> by the bootloader. Sure for now I could do that I guess, but iommu=0
> works as well ;)
>
> I now got Xen and Linux booting and see the same problems with the
> GIC ITS, that is that the enetc interrupts aren't delivered to the
> dom0 linux. I've also applied the patch in this thread and I'm
> seeing the same as Leo. Full boot log is here [1].
>
> I noticed the following.
> [    0.168544] pci 0000:00:00.0: Failed to add - passthrough or
> MSI/MSI-X might fail!

This message is harmless. This is printed because Xen on Arm
doesn't hypercall the hypercall to add a PCI device. On Arm,
we don't need it yet (it might be necessary for PCI passthrough) and
MSI/MSI-X are handled/registered the same way as on bare-metal
(for your case through the ITS)

>
> Not sure if it should work nonetheless.

I looked through the log and couldn't spot anything obvious. However,
skimming through Linux, I noticed there are some changes in the
ITS for freescale (now NXP) such as:

drivers/irqchip/irq-gic-v3-its-fsl-mc-msi.c

Is it something that might be used on the board you are using?

If the answer is yes, then my suggestion would be to look
how this is meant to interact with the ITS. It might be possible
that we are missing some pieces in Xen to properly support it.

>
> > It is possible that someone in CC to this email might already have a
> > patch to introduce parsing of iommu-map in Xen.

Pass. I don't have any and couldn't find any submission on the ML.


>
> I guess they've used the old mmu-masters property.
>
> Btw. I don't know if it matters, but the SMARC-sAL28 normally doesn't
> use TF-A and runs without it. Nonetheless, I've booted the board with
> the bl31 from NXP and it doesn't help either. There is still a
> difference between the NXP bootflow which uses bl1/bl2/bl31/u-boot
> and this board which uses u-boot-spl/u-boot or u-boot-spl/bl31/u-boot.
>
> I just found GIC setup code in the bl31. I'll also have a look there.
>
> -michael
>
> [1] https://pastebin.com/raw/XMjE3BvG