Re: Smoke test failure on Arm (was Re: [PATCH v4 0/8] xen/arm: Emulate ID registers)

[André Przywara <andre.przywara@xxxxxxx>]

On 05/01/2021 16:06, Julien Grall wrote:
> (+ Ian and Andre)
> 
> Hi Bertrand,
> 
> On IRC, Ian pointed out that the smoke test was failing on Cubietruck.
> The only patches because the last success and the failure are your series.
> 
> This seems to be a very early failure as there is no output from Xen [1].
> 
> I originally thought the problem was because some of the ID registers
> (such as ID_PFR2) introduced in patch #2 doesn't exist in Armv7.
> 
> But per B7.2.1 in ARM DDI 0406C.d, unallocated ID registers should be
> RAZ. So it would result to a crash. Andre confirmed that PFR2 can be
> accessed by writing a small baremetal code and booted on Cortex-A7 and
> Cortex-A20.
> 
> So I am not entirely sure what's the problem. Andre kindly accepted to
> try to boot Xen on his board. Hopefully, this will give us a clue on the
> problem.
> 
> If not, I will borrow a Cubietruck in OssTest and see if I can reproduce
> it and debug it.


So I just compiled master and staging and ran just that on an Allwinner
H3 (Cortex-A7 r0p5). Master boots fine (till it complains about the
missing Dom0, as expected). However staging indeed fails:

(XEN) Xen version 4.15-unstable (andprz01@xxxxxxxxxxxx)
(arm-slackware-linux-gnueabihf-gcc (GCC) 8.2.0) debug=y  Tue Jan  5
16:09:40 GMT 2021
(XEN) Latest ChangeSet: Sun Nov 8 15:59:42 2020 +0100 git:c992efd06a
(XEN) build-id: 85d361b8565b90d4e0defe2beb2419e191fd76b4
(XEN) CPU0: Unexpected Trap: Undefined Instruction
(XEN) ----[ Xen-4.15-unstable  arm32  debug=y   Not tainted ]----
(XEN) CPU:    0
(XEN) PC:     0026b8c8 identify_cpu+0xc0/0xd4
(XEN) CPSR:   600001da MODE:Hypervisor
(XEN)      R0: 002acb20 R1: 00000000 R2: 00000000 R3: 11111111
(XEN)      R4: 002acb1c R5: 002acb20 R6: 4e000000 R7: 00000000
(XEN)      R8: 00000002 R9: 002d8200 R10:00008000 R11:002f7e6c R12:00000080
(XEN) HYP: SP: 002f7e68 LR: 002c419c
(XEN)
(XEN)   VTCR_EL2: 80002646
(XEN)  VTTBR_EL2: 00000018e628bb80
(XEN)
(XEN)  SCTLR_EL2: 30cd187f
(XEN)    HCR_EL2: 00000038
(XEN)  TTBR0_EL2: 000000004013a000
(XEN)
(XEN)    ESR_EL2: 00000000
(XEN)  HPFAR_EL2: 0003fff0
(XEN)      HDFAR: 9d110000
(XEN)      HIFAR: 0000a04a
(XEN)
(XEN) Xen stack trace from sp=002f7e68:
(XEN)    00000000 002f7f54 00008000 00000000 00002000 002a4584 00000000
00000000
(XEN)    00000000 00008000 49ff5000 002d81f0 40000000 00000000 00002000
00000001
(XEN)    00000000 50000000 49ffd000 00000000 50000000 00000000 00000000
50000000
(XEN)    4c000000 00000000 4e000000 00000000 ffffffff ffffffff 50000000
00000000
(XEN)    50000000 00000000 50000000 00000000 00000000 00000000 00000000
00000000
(XEN)    00000000 00000003 00000000 00000000 ffffffff 00000040 ffffffff
00000000
(XEN)    00000000 00000000 00000000 002a7000 40008050 0000001a 00000000
49ff5000
(XEN)    40008000 3fe08000 00000004 0020006c 00000000 00000000 00000000
00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000
(XEN) Xen call trace:
(XEN)    [<0026b8c8>] identify_cpu+0xc0/0xd4 (PC)
(XEN)    [<002c419c>] start_xen+0x778/0xe50 (LR)
(XEN)    [<002f7f54>] 002f7f54
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) CPU0: Unexpected Trap: Undefined Instruction
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...


The code in question:
  26b8c0:       eef63a10        vmrs    r3, mvfr1
  26b8c4:       e5803058        str     r3, [r0, #88]   ; 0x58
> 26b8c8:       eef53a10        vmrs    r3, mvfr2
  26b8cc:       e580305c        str     r3, [r0, #92]   ; 0x5c
  26b8d0:       e28bd000        add     sp, fp, #0
  26b8d4:       e49db004        pop     {fp}       ; (ldr fp, [sp], #4)
  26b8d8:       e12fff1e        bx      lr

And indeed MVFR2 is not mentioned in the ARMv7 ARM, and in contrast to
the CP15 CPUID registers this is using the VMRS instruction, so it's not
protected by future-proof CPUID register scheme.

Not sure what to do about this, maybe #ifdef'ing this register access
with arm64?
I guess this comes from the slightly too optimistic code-sharing between
arm32 and arm64?

Cheers,
Andre

> On 17/12/2020 15:38, Bertrand Marquis wrote:
>> The goal of this serie is to emulate coprocessor ID registers so that
>> Xen only publish to guest features that are supported by Xen and can
>> actually be used by guests.
>> One practical example where this is required are SVE support which is
>> forbidden by Xen as it is not supported, but if Linux is compiled with
>> it, it will crash on boot. An other one is AMU which is also forbidden
>> by Xen but one Linux compiled with it would crash if the platform
>> supports it.
>>
>> To be able to emulate the coprocessor registers defining what features
>> are supported by the hardware, the TID3 bit of HCR must be disabled and
>> Xen must emulated the values of those registers when an exception is
>> catched when a guest is accessing it.
>>
>> This serie is first creating a guest cpuinfo structure which will
>> contain the values that we want to publish to the guests and then
>> provides the proper emulationg for those registers when Xen is getting
>> an exception due to an access to any of those registers.
>>
>> This is a first simple implementation to solve the problem and the way
>> to define the values that we provide to guests and which features are
>> disabled will be in a future patchset enhance so that we could decide
>> per guest what can be used or not and depending on this deduce the bits
>> to activate in HCR and the values that we must publish on ID registers.
>>
>> ---
>> Changes in V2:
>>    Fix First patch to properly handle DFR1 register and increase dbg32
>>    size. Other patches have just been rebased.
>>
>> Changes in V3:
>>    Add handling of reserved registers as RAZ
>>    Minor fixes described in each patch
>>
>> Changes in V4:
>>    Add a patch to switch implementation to use READ_SYSREG instead of the
>>    32/64 bit version of it.
>>    Move cases for reserved register handling from macros to the code
>>    itself.
>>    Various typos fixes.
>>
>> Bertrand Marquis (8):
>>    xen/arm: Use READ_SYSREG instead of 32/64 versions
>>    xen/arm: Add ID registers and complete cpuinfo
>>    xen/arm: Add arm64 ID registers definitions
>>    xen/arm: create a cpuinfo structure for guest
>>    xen/arm: Add handler for ID registers on arm64
>>    xen/arm: Add handler for cp15 ID registers
>>    xen/arm: Add CP10 exception support to handle MVFR
>>    xen/arm: Activate TID3 in HCR_EL2
>>
>>   xen/arch/arm/arm64/vsysreg.c        |  82 ++++++++++++++++++++
>>   xen/arch/arm/cpufeature.c           | 113 ++++++++++++++++++++++------
>>   xen/arch/arm/traps.c                |   7 +-
>>   xen/arch/arm/vcpreg.c               | 102 +++++++++++++++++++++++++
>>   xen/include/asm-arm/arm64/hsr.h     |  37 +++++++++
>>   xen/include/asm-arm/arm64/sysregs.h |  28 +++++++
>>   xen/include/asm-arm/cpregs.h        |  15 ++++
>>   xen/include/asm-arm/cpufeature.h    |  58 +++++++++++---
>>   xen/include/asm-arm/perfc_defn.h    |   1 +
>>   xen/include/asm-arm/traps.h         |   1 +
>>   10 files changed, 409 insertions(+), 35 deletions(-)
>>
> 
> [1]
> http://logs.test-lab.xenproject.org/osstest/logs/158152/test-armhf-armhf-xl/info.html
> 
>