Re: [Xen-devel] [PATCH] xen/arm64: Use __flush_dcache_area instead of __flush_dcache_all

[Mark Rutland <mark.rutland@xxxxxxx>]

On Tue, Oct 14, 2014 at 11:39:37AM +0100, Ian Campbell wrote:
> On Tue, 2014-10-14 at 11:32 +0100, Mark Rutland wrote:
> > On Tue, Oct 14, 2014 at 10:35:23AM +0100, Ian Campbell wrote:
> > > On Tue, 2014-10-14 at 10:21 +0100, Mark Rutland wrote:
> > > > Hi Roy,
> > > > 
> > > > [...]
> > > > 
> > > > > It seems that for Xen we do need to flush the FDT as well - I get a
> > > > > variety of crashes
> > > > > with a corrupt FDT when cache state is modeled on the FVP model, and
> > > > > Suravee sees similar
> > > > > behavior on Seattle. I was not expecting this, as I looked at the code
> > > > > in Xen and the caches/TLB
> > > > > are enabled quite early on, before the FDT is accessed by Xen.  I then
> > > > > looked at the mappings
> > > > > used by  edk2 and Xen, and found some differences.  Even after
> > > > > modifying edk2 to use the same
> > > > > configuration as Xen, the flushing of the FDT is still required. Xen
> > > > > and edk2 use the same memory
> > > > > attributes  in the MAIR_EL2 register (0xFF), but had different
> > > > > sharing, access perm, and nG settings.
> > > > 
> > > > I don't think the access perm or nG settings should have any effect, but
> > > > the shareability forms part of the memory attributes (along with the
> > > > memory type and cacheability), and there are several rules that apply
> > > > when accessing a memory location with mismatched attributes. See the
> > > > ARMv8 ARM - The AArch64 Application Level Memory Model - Mismatched
> > > > memory attributes.
> > > > 
> > > > In Linux we're likely getting lucky, and the shareability we use varies
> > > > for an SMP or UP kernel. So we need maintenance in at least one of those
> > > > cases. This would also apply to any initrd or other image.
> > > > 
> > > > Do you happen to know the shareability used by EDK2 and Xen?
> > > 
> > > Xen maps everything inner-shareable. Dunno about EDK2.
> > 
> > Ok. That matches what an SMP Linux kernel will do, so it looks like
> > we're just getting lucky with Linux. I'lll have a play and see if I can
> > trigger similar issues.
> > 
> > > Is the real issue here not a lack of specification for some corner cases
> > > of the boot protocol? Can we get that fixed somehow?
> > 
> > To an extent, yes. We can try to fix up the Linux side with patche to
> > Documentation/arm64/booting.txt. As far as I am aware, for UEFI that
> > will require membership of the UEFI forum.
> > 
> 
> Is Documentation/arm64/booting.txt relevant here since the kernel is
> being launched as an EFI app, which already has a standardised calling
> convention of its own. I suppose booting.txt is in addition to the UEFI
> convention. It probably would be best to formalise that (what if a
> second OS comes along with contradictory requirements?)

If we're trying to fix up UEFI, that needs to happen at the UEFI forum
level. I believe there are some additional reqwuirements in SBSA/SBBR,
but I haven't studied them in detail.

If there are requirements that Linux needs to have met regardless of
UEFI, we should ensure we mention that in booting.txt.

It would be nice to have cross-OS agreement on boot protocols, but at
the moment the table is somewhat empty beyond Linux and Xen. I had a
conversation with the FreeBSD guys working on 64-bit ARM stuff, but
they're still at an early stage, and I can't recall the specifics of
their boot process.

> > > Part of me wants to suggest that UEFI (and bootloaders generally) ought
> > > to be cleaning caches for anything they have loaded into RAM before
> > > launching an OS as a matter of good hygiene.
> > 
> > In general, yes.
> > 
> > Unfortunately, UEFI can't perform the maintenance in this case, because
> > the stub modifies things. I was under the impression it copied and
> > modified the FDT to embed the command line -- UEFI has no visibiltiy of
> > this and therefore cannot be in charge of flushing it. So in this case,
> > the stub needs to be thought of as the bootloader, and needs to be in
> > charge of any required maintenance.
> 
> Right, that's what I was thinking. UEFI enters bootloader with
> everything it has done all nice and clean and consistent. Anything the
> stub then does it is responsible for maintaining the cleanliness.

There are two horrible parts here:

 * EFI has no idea what a boot loader is. As far as it's aware, the
   kernel + efi stub is just another UEFI application until it calls
   ExitBootServices. For all UEFI knows, it may as well be a calculator
   until that point, and flushing the entire cache hierarchy for a
   calculator seems a little extreme.

 * Defining "nice and clean and consistent".
  
   As far as I am aware, UEFI may have an arbitrary set of mappings
   present during boot services time, with arbitrary drivers active. 
   That means that UEFI can create dirty cache entries concurrently with
   the bootloader, in addition to the usual clean entries that can be
   allocated at any time thanks to speculative fetches.
   
   So while we're in the bootloader, any system level caches can have
   entries allocated to it, and as those aren't architected the only
   thing we can do is flush those by VA for the portions we care about.
   
So we can have "initially consistent", but that might not be useful.

> > There are a tonne of subtleties here, and certain properties we would
> > like (e.g. a completely clean cache hierarchy upon entry to the OS)
> > aren't necessarily possible to provide in general (thanks to the wonders
> > of non-architected system level caches, interaction with bootloaders,
> > etc).
> 
> I suppose it is easier for the UEFI implementation, since it knows the
> platform it runs on and there knows about the caches. Harder for the
> stub though :-/

Yeah. System-level caches interact badly with pretty much any scenario
where ownership of the MMU is transferred (UEFI boot, kexec), and there
doesn't seem to be a single agent that can be charged with ownership of
maintenance.

This is something I've been meaning to revisit, but it takes a while to
get back up to speed on the minutiae of the cache architecture and the
rules for memory attributes, and I haven't had the time recently.

We do have a very heavy hammer that we know will work: flushing the
memory by PA in the stub once the MMU and caches are disabled. A
back-of-the-envelope calculation shows that could take minutes to issue
on a server machine (say 2GHz, with 16GB of RAM), so that's very much a
last resort.

We could try to manage the system caches explicitly, but then we need
code to do so very early, we need to have them described in the
appropriate firmware tables, and they need to be manageable from the
non-secure side (which I believe is not always the case). That somewhat
defeat the portability aspect of booting as an EFI application.

So yes, it's harder for the stub :/

Mark.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel