Re: changing Dom0 data during smc call inside of xen during cache coloring

[Oleg Nikitenko <oleshiiwood@xxxxxxxxx>]

Hello Julien,

I am still fighting with xen Cache Coloring with aes.

When I sent a request to hardware aes after xen with CC started I got the mistake in CSU.

When I dumped structure contents on both sides I got the different data.

Xilinx related contact wrote to me.

When cache coloring is enabled, Dom0 is not 1:1 mapped (guest physical addresses in Dom0 != physical addresses). If the Xilinx drivers in Linux (xcsudma.c) issue EEMI calls with a guest physical address (for instance the address of a memory buffer allocated by Linux), then this address is no longer a physical address and would need to be translated. EEMI calls always get forwarded to Xen first, then Xen issues a corresponding EEMI call to the firmware (see xen/arch/arm/platforms/xilinx-eemi.c:xilinx_eemi). But Xen is probably passing the EEMI calls parameters unmodified. Then PMU tries to read the address but since this is not a physical address, it fails. Basically we need to add code to Xen xen/arch/arm/platforms/xilinx-eemi.c:xilinx_eemi to translate any guest physical addresses passed as EEMI calls arguments into physical addresses before making the EEMI call to firmware.

This is an example patch, which is translating the parameter on register x2 for the EEMI call 0xC200002F. I haven't checked the EEMI protocol for this call but this just an example to show you how to translate parameters.

diff --git a/xen/arch/arm/platforms/xilinx-eemi.c b/xen/arch/arm/platforms/xilinx-eemi.c index 500c86dc69..bff1b71196 100644 --- a/xen/arch/arm/platforms/xilinx-eemi.c +++ b/xen/arch/arm/platforms/xilinx-eemi.c @@ -409,6 +409,30 @@ bool xilinx_eemi(struct cpu_user_regs *regs, const uint32_t fid, } goto forward_to_fw;

• case 0xC200002F:
• {
• uint64_t example_possible_address_param = get_user_reg(regs, 2);
• uint64_t translated_address = mfn_to_maddr(gfn_to_mfn(current->domain,
• gaddr_to_gfn(example_possible_address_param)));
• translated_address += example_possible_address_param & ~PAGE_MASK; +
• arm_smccc_1_1_smc(get_user_reg(regs, 0),
• get_user_reg(regs, 1),
• translated_address,
• get_user_reg(regs, 3),
• get_user_reg(regs, 4),
• get_user_reg(regs, 5),
• get_user_reg(regs, 6),
• get_user_reg(regs, 7),
• &res); +
• set_user_reg(regs, 0, res.a0);
• set_user_reg(regs, 1, res.a1);
• set_user_reg(regs, 2, res.a2);
• set_user_reg(regs, 3, res.a3);
• return true;
• }

+ default: if ( is_hardware_domain(current->domain) ) goto forward_to_fw;

The aes request structure contains physical addresses of the source and destination.

These addresses are obtained via two calls dma_alloc_coherent.

The address of this structure is kept at x2 register.

I applied the suggested scheme in xen for xilinx_eemi(...) function.

case 0xC200002F:
{
uint64_t paramaddr = get_user_reg(regs, 2);
uint64_t phyaddr = mfn_to_maddr(gfn_to_mfn(current->domain, gaddr_to_gfn(paramaddr)));
phyaddr += (paramaddr & ~PAGE_MASK);
gprintk(XENLOG_DEBUG, "Forwarding AES operation: %u r2 %lx -> %lx\n", fid, paramaddr, phyaddr);
set_user_reg(regs, 2, phyaddr);
}
goto forward_to_fw;

As a result I got the same issue as earlier.

[   17.350086] zynq_aes_gcm                                                                                                                        user log

[   17.350202] @ dma_alloc firmware:zynqmp-firmware:zynqmp-aes @                                                       kernel log from Dom0
[   17.353015] @@@ firmware:zynqmp-firmware:zynqmp-aes 0 @@@
[   17.358515] zynqmp_aes [0] ffffffc00910d000 2806000 firmware:zynqmp-firmware:zynqmp-aes
[   17.366546] @ dma_alloc firmware:zynqmp-firmware:zynqmp-aes @
[   17.372347] @@@ firmware:zynqmp-firmware:zynqmp-aes 0 @@@
[   17.377775] zynqmp_aes [1] ffffffc009115000 42a14000 keytype 1
[   17.383660] zynqmp_aes [2] dump request align 1 ++
[   17.388501] 00 60 80 02 00 00 00 00
[   17.392032] 50 60 80 02 00 00 00 00
[   17.395583] 00 00 00 00 00 00 00 00
[   17.399117] 00 60 80 02 00 00 00 00
[   17.402664] 40 00 00 00 00 00 00 00
[   17.406226] 00 00 00 00 00 00 00 00
[   17.409755] 01 00 00 00 00 00 00 00
[   17.413311] zynqmp_aes [3] dump request --

(XEN) d0v1 Forwarding AES operation: 3254779951 r2 0 -> 11432000                                                        log from xen

@ 000042A14000 @                                                                                                                                      csu log from aes
04 E4 00 6F 05 E4 00 6F
06 E4 00 6F 07 E4 00 6F
10 E4 00 6F 11 E4 00 6F
12 E4 00 6F 13 E4 00 6F
14 E4 00 6F 15 E4 00 6F
16 E4 00 6F 17 E4 00 6F
18 E4 00 6F 19 E4 00 6F

ERROR:   pm_aes_engine ### args 6 ret 0 addr 0 42a14000 ###                                                                ATF log

So the address of the structure was not changed.

This is the question. 

How can I map this address to xen and change physical addresses there ?

Regards,

Oleg Nikitenko

чт, 28 сент. 2023 г. в 11:15, Julien Grall <julien@xxxxxxx>:

On 27/09/2023 11:07, Oleg Nikitenko wrote:
> Hello,

Hi,

> It is necessary to change some structure contents from xen.
> I have access to the registers.
> One of them keeps the physical address of the structure.
> But this physical address is valid for Dom0 only.
> Dom0 kernel gets it by the call dma_alloc_coherent
> A lower mentioned scheme does not work.

It is not clear to me what you mean by does not work. Are you getting
the wrong address?

>
>              uint64_t paramaddr = (uint64_t)get_user_reg(regs, 2);
>              uint64_t phyaddr = mfn_to_maddr(gfn_to_mfn(current->domain,
>                  gaddr_to_gfn(paramaddr)));
>              phyaddr += (paramaddr & ~PAGE_MASK);

Can you provide a bit more context of what are you trying to do with
phyaddr afterwards? Are you trying to map it in Xen?

Cheers,

--
Julien Grall