Re: [Xen-devel] Problems building and running Xen on Hikey960

[Julien Grall <julien.grall@xxxxxxx>]

On 08/11/2018 07:07, Matthew Daley wrote:
> Hi all,
Hi Matthew,

> I'm trying to build and run Linux (Debian) on top of Xen on a Hikey960
> ARM board but am having trouble.
>
> I've been following Konrad's guide on the Wiki (at
> https://wiki.xenproject.org/wiki/HiKey960), namely:
> * I have upgraded the board's base firmware to support UEFI
> * I am building the hikey960-upstream-rebase branch of
> https://github.com/96boards-hikey/linux.git (with a patch from Linus's
> tree to fix a Hynix UFS bug)
> * I am building the master branch of Xen unstable
> * I am building the master branch of GRUB (with a patch to fix a DTB
> padding bug)
>
> However, I see two problems when booting.
>
> Firstly, Xen fails to bring up any other CPUs but the one it is booting on:
>
> (XEN) Bringing up CPU1
> (XEN) Failed to bring up CPU1
> (XEN) Failed to bring up CPU 1 (error -9)
> (XEN) Bringing up CPU2
> (XEN) Failed to bring up CPU2
> (XEN) Failed to bring up CPU 2 (error -9)
> (XEN) Bringing up CPU3
> (XEN) Failed to bring up CPU3
> (XEN) Failed to bring up CPU 3 (error -9)
> (XEN) Bringing up CPU4
> (XEN) Failed to bring up CPU4
> (XEN) Failed to bring up CPU 4 (error -9)
> (XEN) Bringing up CPU5
> (XEN) Failed to bring up CPU5
> (XEN) Failed to bring up CPU 5 (error -9)
> (XEN) Bringing up CPU6
> (XEN) Failed to bring up CPU6
> (XEN) Failed to bring up CPU 6 (error -9)
> (XEN) Bringing up CPU7
> (XEN) Failed to bring up CPU7
> (XEN) Failed to bring up CPU 7 (error -9)
> (XEN) Brought up 1 CPUs
>
> I have traced this error code -9 being returned by call_psci_cpu_on.
A similar error was reported a couple of months on the mailing list. From the 
report, a regression was introduced between Xen 4.8 and unstable.
Unfortunately, I don't have an hikey board to bisect it. May I ask if you can 
bisect it? If you can point the offending commit, I should be able to provide 
ideas why it breaks.
> Secondly, Linux fails when it tries to initialise AMBA devices:
>
> [    0.941352] Synchronous External Abort: synchronous external abort
> (0x96000210) at 0xffff0000093fdfe
> 0
> [    0.950601] Internal error: : 96000210 [#1] PREEMPT SMP
> [    0.955866] Modules linked in:
> [    0.958990] CPU: 0 PID: 1 Comm: swapper/0 Not tainted
> 4.14.0-rc7-linaro-hikey960+ #8
> [    0.966791] Hardware name: HiKey960 (DT)
> [    0.970777] task: ffff80001d900000 task.stack: ffff000008058000
> [    0.976778] PC is at amba_device_try_add+0x108/0x260
> [    0.981791] LR is at amba_device_try_add+0xf0/0x260
> [    0.986735] pc : [<ffff0000084eada8>] lr : [<ffff0000084ead90>]
> pstate: 60000045
> [    0.994192] sp : ffff00000805bbf0
> [    0.997572] x29: ffff00000805bbf0 x28: 0000000000000000
> [    1.002953] x27: ffff0000090203a8 x26: 0000000000000000
> [    1.008326] x25: ffff80001dbf9810 x24: 0000000000000000
> [    1.013702] x23: ffff0000093fd000 x22: 0000000000001000
> [    1.019079] x21: ffff80001cc1b6f8 x20: 0000000000000000
> [    1.024455] x19: ffff80001cc1b400 x18: 0000000000000010
> [    1.029832] x17: 0000000000000001 x16: 00000000deadbeef
> [    1.035209] x15: 0000000000000006 x14: ffffffffffffffff
> [    1.040585] x13: 0000000000000020 x12: 0101010101010101
> [    1.045962] x11: 0000000000000020 x10: 0101010101010101
> [    1.051338] x9 : 0000000000000000 x8 : ffff80001cc0cf00
> [    1.056717] x7 : 0000000000000000 x6 : 000000000000003f
> [    1.062092] x5 : 0000000000000000 x4 : 0000000000000000
> [    1.067468] x3 : 0000000000000000 x2 : 0000000000000000
> [    1.072845] x1 : ffff80001d900000 x0 : ffff0000093fdfe0
> [    1.078223] Process swapper/0 (pid: 1, stack limit = 0xffff000008058000)
> [    1.084989] Call trace:
> [    1.087504] Exception stack(0xffff00000805bab0 to 0xffff00000805bbf0)
> [    1.094008] baa0:
> ffff0000093fdfe0 ffff80001d900000
> [    1.101902] bac0: 0000000000000000 0000000000000000
> 0000000000000000 0000000000000000
> [    1.109790] bae0: 000000000000003f 0000000000000000
> ffff80001cc0cf00 0000000000000000
> [    1.117684] bb00: 0101010101010101 0000000000000020
> 0101010101010101 0000000000000020
> [    1.125576] bb20: ffffffffffffffff 0000000000000006
> 00000000deadbeef 0000000000000001
> [    1.133468] bb40: 0000000000000010 ffff80001cc1b400
> 0000000000000000 ffff80001cc1b6f8
> [    1.141356] bb60: 0000000000001000 ffff0000093fd000
> 0000000000000000 ffff80001dbf9810
> [    1.149248] bb80: 0000000000000000 ffff0000090203a8
> 0000000000000000 ffff00000805bbf0
> [    1.157139] bba0: ffff0000084ead90 ffff00000805bbf0
> ffff0000084eada8 0000000060000045
> [    1.165034] bbc0: ffff00000805bbf0 ffff0000084ead90
> ffffffffffffffff 00000000fffffffe
> [    1.172921] bbe0: ffff00000805bbf0 ffff0000084eada8
> [    1.177865] [<ffff0000084eada8>] amba_device_try_add+0x108/0x260
> [    1.183935] [<ffff0000084eafec>] amba_device_add+0x1c/0xd8
> [    1.189493] [<ffff00000890fbbc>] of_platform_bus_create+0x26c/0x300
> [    1.195814] [<ffff00000890fa74>] of_platform_bus_create+0x124/0x300
> [    1.202145] [<ffff00000890fd7c>] of_platform_populate+0x4c/0xb0
> [    1.208135] [<ffff000008f6e2ac>] of_platform_default_populate_init+0x64/0x78
> [    1.215247] [<ffff000008083978>] do_one_initcall+0x38/0x120
> [    1.220882] [<ffff000008f20d18>] kernel_init_freeable+0x184/0x224
> [    1.227038] [<ffff000008a6a360>] kernel_init+0x10/0x100
> [    1.232323] [<ffff000008084b60>] ret_from_fork+0x10/0x18
> [    1.237703] Code: d10082c0 52800002 8b0002e0 52800018 (b9400001)
> [    1.243880] ---[ end trace dcbf70aa30c979a8 ]---
> [    1.248573] Kernel panic - not syncing: Attempted to kill init!
> exitcode=0x0000000b
>
> This does not occur when booting the same Linux kernel without Xen.
>
> I have traced this to amba_device_try_add trying to access the pid and
> cid through ioremap'd addresses of this node in the device tree (in
> arch/arm64/boot/dts/hisilicon/hi3660-coresight.dtsi):
Thank you for confirming it works on Linux baremetal and tracked down the 
problem.

It is not entirely clear why you receive an external abort here. This may be due 
to misconfiguration of the stage-2 page-tables.
Could you try Xen with this small changes? The patch should print a message is 
the virtual abort was received by Xen but forwarded to the guest.
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 51d2e42c77..f95135d030 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -660,6 +660,8 @@ static void inject_vabt_exception(struct cpu_user_regs 
*regs)
 {
     const union hsr hsr = { .bits = regs->hsr };

+    printk("%pv: Inject Virtual Abort\n", current);
+    dump_execution_state();
+
     /*
      * SVC/HVC/SMC already have an adjusted PC (See ARM ARM DDI 0487A.j
      * D1.10.1 for more details), which we need to correct in order to

> /* A73 cluster internal coresight */
> etm@4,ed440000 {
>          compatible = "arm,coresight-etm4x","arm,primecell";
>          reg = <0 0xed440000 0 0x1000>;
>          clocks = <&pclk>;
>          clock-names = "apb_pclk";
>          cpu = <&cpu4>;
>          port {
>                  etm4_out_port: endpoint {
>                  remote-endpoint = <&funnel1_in_port0>;
>                  };
>          };
> };
>
> ARM is still relatively new to me and I'm stuck what I should be
> attempting next. I would simply not compile Linux AMBA support (by not
> setting CONFIG_AMBA) but it appears that this is selected as a reverse
> dependency by CONFIG_ARM on Linux, so am unsure if this is wise or
> even possible.
I would just drop the node you copied above from the Device-Tree and see if you 
can go further in the boot.
Cheers,

--
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel