Re: [Xen-devel] [PATCH] xen/arm64: Use __flush_dcache_area instead of __flush_dcache_all

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxxxxx>]

On Tue, 14 Oct 2014, Mark Rutland wrote:
> [...]
> 
> > >> >    As far as I am aware, UEFI may have an arbitrary set of mappings
> > >> >    present during boot services time, with arbitrary drivers active.
> > >> >    That means that UEFI can create dirty cache entries concurrently 
> > >> > with
> > >> >    the bootloader, in addition to the usual clean entries that can be
> > >> >    allocated at any time thanks to speculative fetches.
> > 
> > UEFI specifies that memory in the EFI memory map is flat mapped, but
> > I'd have to look to see if
> > it prohibits other mappings in addition to that.  Other mappings are
> > implementation
> > dependent (devices, etc. or memory not in the EFI memory map.)
> 
> Regardless of the set of mapping that may exist, the key point is that
> we don't know what may have been allocated into a cache. Any portion of
> memory could have entries in the cache hierarchy, which could be clean
> or dirty.
> 
> > In reviewing the Aarch64 specific portion of the spec (section 2.3.6
> > Aarch64 Platforms)
> > it says in part:
> > 
> > Â Implementations of boot services will enable architecturally
> > manageable caches and TLBs i.e.
> >   those that can be managed directly using implementation independent
> > registers using
> >   mechanisms and procedures defined in the ARM Architecture Reference
> > Manual. They should
> >   not enable caches requiring platform information to manage or invoke
> > non-architectural cache/
> >   TLB lockdown mechanisms.
> > 
> > Does this imply that system level caches should not be enabled?
> 
> Arguably yes, but on a technicality no, because it's possible to flush
> them by VA (albeit extremely slowly).

I think that this point should really be clearer at the spec level.


> >> >    So while we're in the bootloader, any system level caches can have
> > >> >    entries allocated to it, and as those aren't architected the only
> > >> >    thing we can do is flush those by VA for the portions we care about.
> > 
> > Maybe the firmware is 'wrong' to enable these caches?
> 
> It is certainly arguable.
> 
> > Are we guaranteed that these caches can be disabled on all
> > implementations?
> 
> I believe on some implementations the non-secure side will not have
> access to the control registers. Beyond that I don't know.
> 
> > Updating/clarifying the spec to have these disabled could simplify the
> > problem a bit.
> 
> Possibly, yes. I'm not sure what we'd clarify it to say, however.

We should start a discussion about this with the relevant parties.


> > >> > So we can have "initially consistent", but that might not be useful.
> > >>
> > >> Hrm, yes, rather unfortunate.
> > >>
> > >> >
> > >> > > > There are a tonne of subtleties here, and certain properties we 
> > >> > > > would
> > >> > > > like (e.g. a completely clean cache hierarchy upon entry to the OS)
> > >> > > > aren't necessarily possible to provide in general (thanks to the 
> > >> > > > wonders
> > >> > > > of non-architected system level caches, interaction with 
> > >> > > > bootloaders,
> > >> > > > etc).
> > >> > >
> > >> > > I suppose it is easier for the UEFI implementation, since it knows 
> > >> > > the
> > >> > > platform it runs on and there knows about the caches. Harder for the
> > >> > > stub though :-/
> > >> >
> > >> > Yeah. System-level caches interact badly with pretty much any scenario
> > >> > where ownership of the MMU is transferred (UEFI boot, kexec), and there
> > >> > doesn't seem to be a single agent that can be charged with ownership of
> > >> > maintenance.
> > >> >
> > >> > This is something I've been meaning to revisit, but it takes a while to
> > >> > get back up to speed on the minutiae of the cache architecture and the
> > >> > rules for memory attributes, and I haven't had the time recently.
> > >> >
> > >> > We do have a very heavy hammer that we know will work: flushing the
> > >> > memory by PA in the stub once the MMU and caches are disabled. A
> > >> > back-of-the-envelope calculation shows that could take minutes to issue
> > >> > on a server machine (say 2GHz, with 16GB of RAM), so that's very much a
> > >> > last resort.
> > >>
> > >> Ouch...
> > >
> > > Looking at that again, I was off by an order of 1000, and that actually
> > > comes to about 0.13 seconds (though solely for CMO issue). So that might
> > > not be as blunt as I made it out to be, but it's still not great as
> > > platforms get larger.
> > 
> > I think we should be able to limit the memory we need to flush, as
> > there should be no
> > need to flush the free memory, just what is in use.  I think that good
> > portions, if not all of that
> > could be flushed from the C code with caches enabled, as we know they won't 
> > be
> > modified after that point (FDT, initrd, etc.)  We can do this in C
> > code after calling
> > ExitBootServices(), and immediately before calling the Xen entry point
> > efi_xen_start().
> > There are no EFI calls in this path between the last bit of C code and
> > the disabling
> > of caches and MMU, so I think we should be able to identify if
> > anything would need
> > to be flushed in the ASM code with caches off.
> 
> I agree the vast majority of this maintenance could be done by C code.
> 
> There might be a need to flush that free memory, depending on how it is
> mapped, unless you are proposing a lazy flush-before-use strategy.

Is it actually safe to only flush what we use (DTB, Xen, initrd, Linux)?
What if the firmware wrote something else (ACPI tables?) that we might
have to access?
What if the firmware wrote something else that we don't care about? Xen
scrubs all ram early at boot, so this last point might not be an issue.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel