Re: Xen on RP4

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Thu, 29 Oct 2020, Elliott Mitchell wrote:
> On Wed, Oct 28, 2020 at 05:37:02PM -0700, Stefano Stabellini wrote:
> > On Tue, 27 Oct 2020, Elliott Mitchell wrote:
> > > On Mon, Oct 26, 2020 at 06:44:27PM +0000, Julien Grall wrote:
> > > > On 26/10/2020 16:03, Elliott Mitchell wrote:
> > > > > On Mon, Oct 26, 2020 at 01:31:42PM +0000, Julien Grall wrote:
> > > > >> On 24/10/2020 06:35, Elliott Mitchell wrote:
> > > > >>> ACPI has a distinct
> > > > >>> means of specifying a limited DMA-width; the above fails, because it
> > > > >>> assumes a *device-tree*.
> > > > >>
> > > > >> Do you know if it would be possible to infer from the ACPI static 
> > > > >> table
> > > > >> the DMA-width?
> > > > > 
> > > > > Yes, and it is.  Due to not knowing much about ACPI tables I don't 
> > > > > know
> > > > > what the C code would look like though (problem is which documentation
> > > > > should I be looking at first?).
> > > > 
> > > > What you provided below is an excerpt of the DSDT. AFAIK, DSDT content 
> > > > is written in AML. So far the shortest implementation I have seen for 
> > > > the AML parser is around 5000 lines (see [1]). It might be possible to 
> > > > strip some the code, although I think this will still probably too big 
> > > > for a single workaround.
> > > > 
> > > > What I meant by "static table" is a table that looks like a structure 
> > > > and can be parsed in a few lines. If we can't find on contain the DMA 
> > > > window, then the next best solution is to find a way to identity the 
> > > > platform.
> > > > 
> > > > I don't know enough ACPI to know if this solution is possible. A good 
> > > > starter would probably be the ACPI spec [2].
> > > 
> > > Okay, that is worse than I had thought (okay, ACPI is impressively
> > > complex for something nominally firmware-level).
> > >
> > > There are strings in the present Tianocore implementation for Raspberry
> > > PI 4B which could be targeted.  Notably included in the output during
> > > boot listing the tables, "RPIFDN", "RPIFDN RPI" and "RPIFDN RPI4" (I'm
> > > unsure how kosher these are as this wsn't implemented nor blessed by the
> > > Raspberry PI Foundation).
> > > 
> > > I strongly dislike this approach as you soon turn the Xen project into a
> > > database of hardware.  This is already occurring with
> > > xen/arch/arm/platforms and I would love to do something about this.  On
> > > that thought, how about utilizing Xen's command-line for this purpose?
> > 
> > I don't think that a command line option is a good idea: basically it is
> > punting to users the task of platform detection. Also, it means that
> > users will be necessarily forced to edit the uboot script or grub
> > configuration file to boot.
> 
> -EINVAL
> 
> Many Linux installations (near universal on desktop/server, but may be
> uncommon on ARM servers) Xen's command-line comes from grub.cfg.
> grub.cfg is in turn created by a series of scripts with several places
> for users to modify configuration without breaking things.
> 
> The scripts which create grub.cfg could add a "dma_mem=" option to Xen's
> command-line based upon what the running kernel reports.  If the kernel
> is running on top of Xen, it will still be able to retrieve this
> information out of ACPI.
> 
> This does mean distributions would need to modify scripts, but that is
> doable.  This also means a dumb user could potentially jump in, modify
> the value and thus cause unrecoverable breakage.  Yet on the flip side
> this also allows for the short-term stop-gap of smart users modifying
> the option as appropriate for new hardware.
> 
> Certainly it may not be the greatest approach, but it isn't as bad as
> you're claiming.

>From what I have seen in previous years getting all the distros to
update their scripts is actually a lot of work. It would take a long
time to see it through. A few years. I think it is likely we would solve
the problem more quickly with a different solution.

 
> > Note that even if we introduced a new command line, we wouldn't take
> > away the need for xen/arch/arm/platforms anyway.
> 
> Perhaps, but it could allow for this setting at least to be moved to
> somewhere outside of Xen.

I was trying to say that it is not just about dma_mem=. There are other
things under xen/arch/arm/platforms that are sometimes required.  In the
specific case of the RPi4, if we added a general dma_mem= parameter, we
would still need to detect the platform to be able to do a platform
reboot, unless it is guaranteed that when ACPI is present ATF is also
present?

More on this below.


> I'm inclined to agree with Juergen Gro??, this reads kind of like having
> an extra domain run during Xen's initialization which can talk ACPI.
> 
> > > Have a procedure of during installation/updates retrieve DMA limitation
> > > information from the running OS and the following boot Xen will follow
> > > the appropriate setup.  By its nature, Domain 0 will have the information
> > > needed, just becomes an issue of how hard that is to retrieve...
> > 
> > Historically that is what we used to do for many things related to ACPI,
> > but unfortunately it leads to a pretty bad architecture where Xen
> > depends on Dom0 for booting rather than the other way around. (Dom0
> > should be the one requiring Xen for booting, given that Xen is higher
> > privilege and boots first.)
> > 
> > 
> > I think the best compromise is still to use an ACPI string to detect the
> > platform. For instance, would it be possible to use the OEMID fields in
> > RSDT, XSDT, FADT?  Possibly even a combination of them?
> > 
> > Another option might be to get the platform name from UEFI somehow. 
> 
> I included appropriate strings in e-mail.  Suitable strings do appear in
> `dmesg`.

Do you mean the ones mentioned in the previous email or did you forget
an attachment here? I don't think you added the value of OEMID anywhere?


> Problem is this feels like you're hard-coding a fixed list of platforms
> Xen can run on.  Instead values like these should be provided by
> firmware.  ACPI includes a method for encoding DMA limitations,

I see where you are coming from: DMA limitations is a general problem,
so it looks like there should be a general solution.

Yes, I agree that it would be good to have a general solution to
automatically detect DMA limitations, no matter if we do ACPI or device
tree.

However, xen/arch/arm/platforms is not just for DMA limitations. There
are per-platform quirks and workarounds that sometimes are required for
booting. Even if we solved the DMA limitations problem generically
(which would be good if we find a way), we are still likely to have to
find another way to detect the platform name.


> device-tree really should have one added.  Only challenge for
> device-tree is getting everyone to agree on names and parameters.

Device tree has ways to describe dma limitations, but they are not
really taken into account when allocating dom0 memory today
unfortunately :-/


> Looking at it, there are really issues with the allocate_memory_11()
> function in xen/arch/arm/domain_build.c.  Two tasks have been merged and
> I'm unsure they were merged correctly.
> 
> I'm unaware of any Xen-capable platforms with such, but DMA can have
> distinct restrictions outside of what allocate_memory_11() provides for.
> ACPI allows for explicit address ranges and in the past many devices have
> used addresses that didn't start at zero.
> 
> Additionally a device might have several devices with restricted DMA and
> these could have differing non-overlapping ranges.  Domain 0 might need
> memory in several DMA ranges.  Luckily In the past few years I haven't
> read about any potentially Xen-capable devices where DMA-capable memory
> had differing capabilities/performance from non-DMA-capable memory.

What you described is possible, at least theoretically. Any patches to
improve allocate_memory_11() in that respect would be great to have.


> Meanwhile for a platform which does have DMA limitations, the kernel and
> boot information for domain 0 shouldn't be placed in DMA-capable memory
> if domain 0 has any memory which isn't DMA-capable.  Yet it appears
> allocate_memory_11() will cause kernel/initrd/boot information to be
> placed in DMA-capable memory.

The dom0 kernel (if zImage) needs to be loaded in the first 128Mb of
RAM. In any case, the kernel will relocate those modules in high memory
and free the original memory during boot, so it shouldn't make a lot of
difference?


> If my understanding is correct (this is a BIG IF), as a last step
> allocate_memory_11() should reverse the order of memory banks.  Another
> trick is DMA-capable banks need to be subject to ballooning AFTER
> non-DMA-capable banks (I'm unsure how often ballooning is used on ARM).

Why reverse the order?