Re: [Xen-devel] [RFC] xen/arm: Handling cache maintenance instructions by set/way

[Julien Grall <julien.grall@xxxxxxxxxx>]

Hi George,

On 12/06/2017 12:28 PM, George Dunlap wrote:
> On 12/05/2017 06:39 PM, Julien Grall wrote:
> > Hi all,
> >
> > Even though it is an Arm failure, I have CCed x86 folks to get feedback
> > on the approach. I have a WIP branch I could share if that interest people.
> >
> > Few months ago, we noticed an heisenbug on jobs run by osstest on the
> > cubietrucks (see [1]). From the log, we figured out that the guest vCPU
> > 0 is in data/prefetch abort state at early boot. I have been able to
> > reproduce it reliably, although from the little information I have I
> > think it is related to a cache issue because we don't trap cache
> > maintenance instructions by set/way.
> >
> > This is a set of 3 instructions (clean, clean & invalidate, invalidate)
> > working on a given cache level by S/W. Because the OS is not allowed to
> > infer the S/W to PA mapping, it can only use S/W to nuke the whole
> > cache. "The expected usage of the cache maintenance that operate by
> > set/way is associated with powerdown and powerup of caches, if this is
> > required by the implementation" (see D3-2020 ARM DDI 0487B.b).
> >
> > Those instructions will target a local processor and usually working in
> > batch for nuking the cache. This means if the vCPU is migrated to
> > another pCPU in the middle of the process, the cache may not be cleaned.
> > This would result to data corruption and potential crash of the OS.
> I don't quite understand the failure mode here: Why does vCPU migration
> cause cache inconsistency in the middle of one of these "cleans", but
> not under normal operation?
Because they target a specific S/W cache level whereas other cache 
operations are working with VA.
To make it short, the other VA cache instructions will work to Poinut of 
Coherency/Point of Unification and guarantee that the caches will be 
consistent. For more details see B2.2.6 in ARM DDI 046C.c.
> > For those been worry about the performance impact, I have looked at the
> > current use of S/W instructions:
> >      - Linux Arm64: The last used in the kernel was beginning of 2015
> >      - Linux Arm32: Still use S/W for boot and secondary CPU bring-up. No
> > plan to change.
> >      - UEFI: A couple of use in UEFI, but I have heard they plan to
> > remove them (need confirmation).
> >
> > I haven't looked at all the OSes. However, given the Arm Arm clearly
> > state S/W instructions are not easily virtualizable, I would expect
> > guest OSes developers to try there best to limit the use of the
> > instructions.
> >
> > To limit the performance impact, we could introduce a guest option to
> > tell whether the guest will use S/W. If it does plan to use S/W, PoD
> > will be disabled.
> >
> > Now regarding the hardware domain. At the moment, it has its RAM direct
> > mapped. Supporting direct mapping in PoD will be quite a pain for a
> > limited benefits (see why above). In that case I would suggest to impose
> > vCPU pinning for the hardware domain if the S/W are expected to be used.
> > Again, a command line option could be introduced here.
> >
> > Any feedbacks on the approach will be welcomed.
> I still don't entirely understand the underlying failure mode, but there
> are a couple of things we could consider:
>
> 1. Automatically disabling 'vcpu migration' when caching is turned off.
> This wouldn't prevent a vcpu from being preempted, just from being run
> somewhere else.
This suggest the guest will directly perform S/W, right? So you leave 
the possibility to the guest to flush all caches the vCPU can access. 
This an easy way for the guest to affect the cache entry of other guests.
I think this would help some potential data attack.

> 2. It sounds like rather than using PoD, you could use the
> "misconfigured p2m table" technique that x86 uses: set bits in the p2m
> entry which cause a specific kind of HAP fault when accessed.  The fault
> handler then looks in the p2m entry, and if it finds an otherwise valid
> entry, it just fixes the "misconfigured" bits and continues.
I thought about this. But when do you set the entry to misconfigured?

If you take the example of Linux 32-bit. There are a couple of full 
cache clean during the boot of uni-processor. So you would need to go 
through the p2m multiple time and reset the access bits.
Cheers,

--
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel