Re: U-Boot on XEN - dcache off

[Ayan Kumar Halder <ayankuma@xxxxxxx>]

+ xen-devel (Since the question is related to Xen on Arm in general).

Hi Michal,

Julien/Stefano/Bertrand - Please feel free to add in.

On 14/07/2023 15:14, Michal Simek wrote:
> +xen
>
> On 7/14/23 16:11, Michal Simek wrote:
> > Hi Stefano,
> >
> > I am playing with running u-boot on the top of xen 
> > (xenguest_arm64_defconfig) based on ask from Kris to run QNX on the 
> > top of it.
> > In past we found that QNX requires to have DCACHE off before it's start.
> > Chris found in past that dcache off is not working and he passed this 
> > to us to take look at it more.
> > That's why I enabled cache commands to be able to run dcache off and 
> > see the hang as Kris.
> > I found one interesting commit created by Marc for Android which were 
> > taken back to u-boot upstream. It is talking about KVM and I barely 
> > understand what he is talking about.
> > I of course enabled this option to try it and behavior is the same.
> > That's why my question is what Xen is doing in connection to dcache 
> > operations.
WRT cache operations, Refer 
https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=53b705d02cec03861044e673536586bd1b2443bd 

When the data abort is caused due to cache maintenance for an address,
there are three scenarios:-

1. Address belonging to a non emulated region - For this, Xen should
set the corresponding bit in the translation table entry to valid and
return to the guest to retry the instruction. This can happen sometimes
as Xen need to set the translation table entry to invalid. (for eg
'Break-Before-Make' sequence). Xen returns to the guest to retry the
instruction.

2. Address belongs to an emulated region - Xen should ignore the
instruction (ie increment the PC) and return to the guest.

3. Address is invalid - Xen should forward the data abort to the guest.

> > Do you know what by-VA cache maintenance operations are?
These are cache operations by virtual address. Refer 
https://developer.arm.com/documentation/den0024/a/Caches/Cache-maintenance
The cache operations (invalidate, clean, zero) can be performed in any 
of the three ways :-
1. all - The entire cache is affected

2. VA - Cache line that contains a certain virtual address is affected.

3. Set/way :- Cache line selected by its position in the cache structure.

> > commit 46dc54287031759c03c68902283d92076938305c
> > Author:     Marc Zyngier <maz@xxxxxxxxxx>
> > AuthorDate: Thu Feb 9 04:54:27 2023 +0800
> > Commit:     Tom Rini <trini@xxxxxxxxxxxx>
> > CommitDate: Mon Mar 6 17:03:55 2023 -0500
> >
> >      arm: cpu: Add optional CMOs by VA
> >
> >      Exposing set/way cache maintenance to a virtual machine is 
> > unsafe, not
> >      least because the instructions are not permission-checked but also
> >      because they are not broadcast between CPUs. Consequently, KVM 
> > traps and
> >      emulates such maintenance in the host kernel using by-VA operations 
I am not sure if Xen behave in same way (ie traps and emulates *all* 
cache maintenance instructions).
@Julien, Stefano, Bertrand - Do you know more about this ?

> > and
> >      looping over the stage-2 page-tables. However, when running under
> >      protected KVM, these instructions are not able to be emulated 
> > and will
> >      instead result in an exception being delivered to the guest.
> >
> >      Introduce CONFIG_CMO_BY_VA_ONLY so that virtual platforms can 
> > select
> >      this option and perform by-VA cache maintenance instead of using 
> > the
> >      set/way instructions.
> >
> >      Signed-off-by: Marc Zyngier <maz@xxxxxxxxxx>
> >      Signed-off-by: Will Deacon <willdeacon@xxxxxxxxxx>
> >      Signed-off-by: Pierre-Clément Tosi <ptosi@xxxxxxxxxx>
> >      [ Paul: pick from the Android tree. Fixup Pierre's commit. And 
> > fix some
> >        checkpatch warnings. Rebased to upstream. ]
> >      Signed-off-by: Ying-Chun Liu (PaulLiu) <paul.liu@xxxxxxxxxx>
> >      Cc: Tom Rini <trini@xxxxxxxxxxxx>
> >      Link: 
> > https://android.googlesource.com/platform/external/u-boot/+/db5507f47f4f57f766d52f753ff2cc761afc213b
> >      Link: 
> > https://android.googlesource.com/platform/external/u-boot/+/2baf54e743380a1e4a6bc2dbdde020a2e783ff67
> >
> > Code ends when it tries to call
> > set_sctlr(sctlr & ~(CR_C|CR_M));
> >
> > Here is implementation. I confirm EL level which is obviously EL1.
> >
> > 180 static inline void set_sctlr(unsigned long val)
> > 181 {
> > 182         unsigned int el;
> > 183
> > 184         el = current_el();
> > 185         if (el == 1)
> > 186                 asm volatile("msr sctlr_el1, %0" : : "r" (val) : 
> > "cc");
> > 187         else if (el == 2)
> > 188                 asm volatile("msr sctlr_el2, %0" : : "r" (val) : 
> > "cc");
> > 189         else
> > 190                 asm volatile("msr sctlr_el3, %0" : : "r" (val) : 
> > "cc");
> > 191
> > 192         asm volatile("isb");
> > 193 }
> >
> > Is there any reason why it is not possible to clear to sctlr_el1 
> > CR_C/CR_M - cache disabling, MMU disabling?
Not sure IIUC,  sctlr_el1 CR_C/CR_M are being cleared to disable D 
cache/MMU. Did I miss something ?
- Ayan

> > I checked already that u-boot starts with cache off/mmu off and when 
> > tables are setup bits are setup.
> > Thanks,
> > Michal