Re: Xen/ARM - Query about a data abort seen while reading GICD registers

[Bertrand Marquis <Bertrand.Marquis@xxxxxxx>]

Hi Ayan,

> On 17 Nov 2021, at 16:21, Ayan Kumar Halder <ayan.kumar.halder@xxxxxxxxxx> 
> wrote:
> 
> Hi Bertrand,
> 
> Many thanks for your response.
> 
> On 17/11/2021 15:48, Bertrand Marquis wrote:
>> Hi Ayan,
>>> On 16 Nov 2021, at 16:24, Ayan Kumar Halder <ayan.kumar.halder@xxxxxxxxxx> 
>>> wrote:
>>> 
>>> Hi Bertrand,
>>> 
>>> Thanks for looking into it.
>>> 
>>> On 16/11/2021 15:36, Bertrand Marquis wrote:
>>>> Hi Ayan,
>>>>> On 16 Nov 2021, at 15:27, Ayan Kumar Halder 
>>>>> <ayan.kumar.halder@xxxxxxxxxx> wrote:
>>>>> 
>>>>> Hi Xen/Arm experts,
>>>>> 
>>>>> I am facing a very strange issue while running a baremetal application as 
>>>>> a DomU guest on arm64 platform.
>>>>> 
>>>>> The baremetal app tries to read the GICD register with post indexing as 
>>>>> follows :-
>>>>> ldr x1, =0x3001000
>>>>> ldr w2, [x1], #4 <<<------ PC = 0x40000ca8
>>>> Increment on on load is not supported by the emulation layer.
>>> 
>>> That is surprising. The reason being if I try to read the GICC register 
>>> (0x3002000) with post indexing then it works fine.
>> When the ISV bit is not set, Xen would have to decode the instruction to 
>> actually emulate the access and properly modify the registers values as long 
>> as providing the “emulated” access value.
> 
> This is very interesting. Is this being done for any of the other 
> instructions in Xen ?

No Xen is not trying to decode any instructions.
If you look at Linux source code, this is the kind of stuff that the kernel is 
delegating to user application (qemu) to do as it can be quite complex.

>> There is not such a decoder in Xen right now which means those kind of 
>> accesses are not supported for emulated mmio accesses.
> 
> I am actually trying to understand where I will need to make the changes if I 
> have to add support for the decoder. The reason being this issue is being 
> faced by one of our customer application.
> Besides changing the instruction to prevent post increment, is there any 
> other mitigation ?

Not that I know of.

> 
> The reason being I don't see a way to instruct the compiler to not generate 
> the post indexing instructions.

You can define io access functions instead of letting the compiler generate the 
read/write functions.
Look at arch/arm64/include/asm-arm/io.h in linux for example.

>>> 
>>> Can you point me in source where this limitation is set ?
>> In this case the isv field is set to 0 when the exception is received by Xen 
>> which means the decoding part of HSR is invalid so Xen is not handling the 
>> emulation of the access.
>> @julien and Stefano: I have been going around and around the code but I 
>> cannot find where the valid bit of hsr_dabt is checked, could you help me 
>> here ?
> 
> By any chance, are you referring to 
> https://github.com/Xilinx/xen/blob/4bd2da58b5b008f77429007a307b658db9c0f636/xen/arch/arm/io.c#L129
>  ?

Yes thanks :-) I was stuck in traps.c but it is actually done later.

> 
> Also, post indexing works for some other registers (eg GICC). So, I am 
> guessing that the behaviour is specific for GICD or random ?

Distributor is emulated by Xen.
Cpu part of the GIC is using the hardware so Xen is not emulating io access to 
this area, the processor is which is why it works in that case.

Bertrand

> 
> - Ayan
>>>> Could you try with:
>>>> add x1, x1, #4
>>>> ldr w2, [x1]
>>> 
>>> This worked fine.
>> Good :-)
>> Bertrand
>>> 
>>> -Ayan
>>>> Regards
>>>> Bertrand
>>>>> 
>>>>> And then I get :-
>>>>> HSR=0x92000005 pc=0x00000040000ca8 gva=0x3001000 gpa=0x00000003001000
>>>>> 
>>>>> This problem occurs only while reading the GICD registers with post 
>>>>> indexing. If I read the register with pre-indexing, then I do not see any 
>>>>> abort. Alternatively, if I read GICC register with post indexing, I don't 
>>>>> see the abort either.
>>>>> 
>>>>> From the HSR value, I interpret it as
>>>>> EC = 100100b # Data abort from lower exception
>>>>> IL = 1b # 32 bit instruction trapped
>>>>> DFSC = 101 # Translation fault level 1
>>>>> 
>>>>> On debugging, I found that the abort is triggered from
>>>>> 
>>>>> try_handle_mmio()
>>>>> { ...
>>>>>    /* All the instructions used on emulated MMIO region should be valid */
>>>>>    if ( !dabt.valid ) {
>>>>> 
>>>>>        return IO_ABORT;
>>>>>    }
>>>>> ...
>>>>> }
>>>>> 
>>>>> From the Arm V8 Arm specs, I understand that dabt.valid is ISV, bit[24] 
>>>>> in "ISS encoding for an exception from a Data Abort".
>>>>> 
>>>>> 
>>>>> I saw that the caller is
>>>>> 
>>>>> do_trap_guest_sync() "case HSR_EC_DATA_ABORT_LOWER_EL"
>>>>> where dabt.valid is false.
>>>>> In the success scenario, dabt.valid is true.
>>>>> 
>>>>> I could not find the caller for do_trap_guest_sync()
>>>>> 
>>>>> So, can anyone help me here
>>>>> 1. Who is the caller for do_trap_guest_sync() ?
>>>>> 2. Any idea on what the issue is and how I can debug it further ?
>>>>> 
>>>>> Kind regards,
>>>>> Ayan