[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] HiKey 960 (ARM 64) rcu_preempt detected stalls



Xen Developers,

Attempting to run Xen on a second generation (has switches instead of jumpers) 
HiKey 960, following the guidance on the Xen wiki. Everything builds, flashes, 
and runs as expected up to the point shown below. Xen hangs in a loop detecting 
rcu_preempt stalls. The specific CPU is not significant, as it varies from boot 
to boot. I have let this run for an hour and Xen never gets out of this loop. 
In this state, Xen is responsive to an initial serial attention sequence of 
ctrl-As. It successfully drops into the keyhandler mode and the help keyhandler 
works fine, but if you try any other keyhandler, Xen crashes.

I asked a colleague to try this on a second HiKey 960, using a different 
cross-compile host, and she gets exactly the same result.

Sincerely,

John

…………………..

Booting `Xen'


Loading driver at 0x000B87F3000 EntryPoint=0x000B88990C8
Loading driver at 0x000B87F3000 EntryPoint=0x000B88990C8 
Using modules provided by bootloader in FDT
Xen 4.10.0-rc (c/s Mon Oct 16 15:14:16 2017 +0100 git:24fb44e971) EFI loader
 Xen 4.10.0-rc
(XEN) Xen version 4.10.0-rc (mc@xxxxxxxxxx) (aarch64-linux-gnu-gcc (Linaro GCC 
7.1-2017.05) 7.1.1 20170510) debug=y  Fri Oct 27 08:58:09 EDT 2017
(XEN) Latest ChangeSet: Mon Oct 16 15:14:16 2017 +0100 git:24fb44e971
(XEN) Processor: 410fd034: "ARM Limited", variant: 0x0, part 0xd03, rev 0x4
(XEN) 64-bit Execution:
(XEN)   Processor Features: 0000000000002222 0000000000000000
(XEN)     Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
(XEN)     Extensions: FloatingPoint AdvancedSIMD
(XEN)   Debug Features: 0000000010305106 0000000000000000
(XEN)   Auxiliary Features: 0000000000000000 0000000000000000
(XEN)   Memory Model Features: 0000000000001122 0000000000000000
(XEN)   ISA Features:  0000000000011120 0000000000000000
(XEN) 32-bit Execution:
(XEN)   Processor Features: 00000131:00011011
(XEN)     Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle
(XEN)     Extensions: GenericTimer Security
(XEN)   Debug Features: 03010066
(XEN)   Auxiliary Features: 00000000
(XEN)   Memory Model Features: 10201105 40000000 01260000 02102211
(XEN)  ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121
(XEN) Using PSCI-1.0 for SMP bringup
(XEN) SMP: Allowing 8 CPUs
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 1920 KHz
(XEN) GICv2 initialization:
(XEN)         gic_dist_addr=00000000e82b1000
(XEN)         gic_cpu_addr=00000000e82b2000
(XEN)         gic_hyp_addr=00000000e82b4000
(XEN)         gic_vcpu_addr=00000000e82b6000
(XEN)         gic_maintenance_irq=25
(XEN) GICv2: 384 lines, 8 cpus, secure (IID 0200143b).
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Allocated console ring of 64 KiB.
(XEN) Bringing up CPU1
(XEN) CPU 1 booted.
(XEN) Bringing up CPU2
(XEN) CPU 2 booted.
(XEN) Bringing up CPU3
(XEN) CPU 3 booted.
(XEN) Bringing up CPU4
(XEN) CPU 4 booted.
(XEN) Bringing up CPU5
(XEN) CPU 5 booted.
(XEN) Bringing up CPU6
(XEN) CPU 6 booted.
(XEN) Bringing up CPU7
(XEN) CPU 7 booted.
(XEN) Brought up 8 CPUs
(XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID
(XEN) P2M: 3 levels with order-1 root, VTCR 0x80023558
(XEN) I/O virtualisation disabled
(XEN) build-id: d34b11f07b634270d6ce4d49a1110fb777f4a16e
(XEN) alternatives: Patching with alt table 00000000400bced0 -> 00000000400bd3a4
(XEN) grant_table.c:1688:IDLEv0 Expanding d0 grant table from 0 to 1 frames
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading kernel from boot module @ 00000000b8917000
(XEN) Allocating 1:1 mappings totalling 512MB for dom0:
(XEN) BANK[0] 0x00000020000000-0x00000030000000 (256MB)
(XEN) BANK[1] 0x000000a0000000-0x000000b0000000 (256MB)
(XEN) Grant table range: 0x000000bfe00000-0x000000bfe40000
(XEN) Loading zImage from 00000000b8917000 to 0000000020080000-0000000021133a00
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Loading dom0 DTB to 0x0000000028000000-0x0000000028008f9d
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM on 1 nodes using 8 CPUs
(XEN) ...done.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) ***************************************************
(XEN) PLEASE SPECIFY dom0_mem PARAMETER - USING 512M FOR NOW
(XEN) ***************************************************
(XEN) 3... 2... 1... 
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to 
Xen)
(XEN) Freed 272kB init memory.
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER4
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER8
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER12
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER16
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER20
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER24
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER28
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER32
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER36
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER40
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER44
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER0
[    0.000000] Booting Linux on physical CPU 0x0

[    0.000000] Linux version 4.13.0-linaro-hikey960+ (mc@xenpro3) (gcc version 
7.1.1 20170510 (Linaro GCC 7.1-2017.05)) #1 SMP PREEMPT Wed Sep 13 10:16:12 EDT 
2017

[    0.000000] Boot CPU: AArch64 Processor [410fd034]

[    0.000000] Machine model: HiKey960

[    0.000000] Xen 4.10 support found

[    0.000000] efi: Getting EFI parameters from FDT:

[    0.000000] efi: UEFI not found.

[    0.000000] cma: Reserved 16 MiB at 0x00000000af000000

[    0.000000] NUMA: No NUMA configuration found

[    0.000000] NUMA: Faking a node at [mem 
0x0000000000000000-0x00000000afffffff]

[    0.000000] NUMA: NODE_DATA [mem 0xaefc0980-0xaefc247f]

[    0.000000] Zone ranges:

[    0.000000]   DMA      [mem 0x0000000020000000-0x00000000afffffff]

[    0.000000]   Normal   empty

[    0.000000] Movable zone start for each node

[    0.000000] Early memory node ranges

[    0.000000]   node   0: [mem 0x0000000020000000-0x000000002fffffff]

[    0.000000]   node   0: [mem 0x00000000a0000000-0x00000000afffffff]

[    0.000000] Initmem setup node 0 [mem 0x0000000020000000-0x00000000afffffff]

[    0.000000] psci: probing for conduit method from DT.

[    0.000000] psci: PSCIv0.2 detected in firmware.

[    0.000000] psci: Using standard PSCI v0.2 function IDs

[    0.000000] psci: Trusted OS migration not required

[    0.000000] percpu: Embedded 26 pages/cpu @ffff8000aeeed000 s67352 r8192 
d30952 u106496

[    0.000000] Detected VIPT I-cache on CPU0

[    0.000000] CPU features: enabling workaround for ARM erratum 845719

[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total 
pages: 129024

[    0.000000] Policy zone: DMA

[    0.000000] Kernel command line: console=tty0 console=hvc0 root=/dev/sdd10 
rw efi=noruntime

[    0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)

[    0.000000] Memory: 456460K/524288K available (10236K kernel code, 1166K 
rwdata, 4532K rodata, 1152K init, 402K bss, 51444K reserved, 16384K 
cma-reserved)

[    0.000000] Virtual kernel memory layout:

[    0.000000]     modules : 0xffff000000000000 - 0xffff000008000000   (   128 
MB)

[    0.000000]     vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000   (129022 
GB)

[    0.000000]       .text : 0xffff000008080000 - 0xffff000008a80000   ( 10240 
KB)

[    0.000000]     .rodata : 0xffff000008a80000 - 0xffff000008ef0000   (  4544 
KB)

[    0.000000]       .init : 0xffff000008ef0000 - 0xffff000009010000   (  1152 
KB)

[    0.000000]       .data : 0xffff000009010000 - 0xffff000009133a00   (  1167 
KB)

[    0.000000]        .bss : 0xffff000009133a00 - 0xffff0000091982e0   (   403 
KB)

[    0.000000]     fixed   : 0xffff7dfffe7fd000 - 0xffff7dfffec00000   (  4108 
KB)

[    0.000000]     PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000   (    16 
MB)

[    0.000000]     vmemmap : 0xffff7e0000000000 - 0xffff800000000000   (  2048 
GB maximum)

[    0.000000]               0xffff7e0000800000 - 0xffff7e0002c00000   (    36 
MB actual)

[    0.000000]     memory  : 0xffff800020000000 - 0xffff8000b0000000   (  2304 
MB)

[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1

[    0.000000] Preemptible hierarchical RCU implementation.

[    0.000000]  RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=8.

[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=8

[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0

[    0.000000] arch_timer: cp15 timer(s) running at 1.92MHz (virt).

[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff 
max_cycles: 0x1c570327e, max_idle_ns: 1763180809505 ns

[    0.000003] sched_clock: 56 bits at 1920kHz, resolution 520ns, wraps every 
4398046510977ns

[    0.000084] clocksource: arm,sp804: mask: 0xffffffff max_cycles: 0xffffffff, 
max_idle_ns: 58334160866034 ns

[    0.000445] Console: colour dummy device 80x25

[    0.000869] console [tty0] enabled

[    0.346638] console [hvc0] enabled

[    0.350171] Calibrating delay loop (skipped), value calculated using timer 
frequency.. 3.84 BogoMIPS (lpj=7680)

[    0.360275] pid_max: default: 32768 minimum: 301

[    0.364997] Security Framework initialized

[    0.369253] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)

[    0.376201] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)

[    0.383079] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes)

[    0.389752] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes)

[    0.412958] ASID allocator initialised with 65536 entries

[    0.418758] xen:grant_table: Grant tables using version 1 layout

[    0.424697] Grant table initialized

[    0.428255] xen:events: Using FIFO-based ABI

[    0.432614] Xen: initializing cpu0

[    0.440590] Hierarchical SRCU implementation.

[    0.453546] EFI services will not be available.

[    0.466011] smp: Bringing up secondary CPUs ...

(XEN) d0v1: vGICD: unhandled word write 0xffffffff to ICACTIVER0
(XEN) d0v2: vGICD: unhandled word write 0xffffffff to ICACTIVER0
(XEN) d0v3: vGICD: unhandled word write 0xffffffff to ICACTIVER0
(XEN) d0v4: vGICD: unhandled word write 0xffffffff to ICACTIVER0
(XEN) d0v5: vGICD: unhandled word write 0xffffffff to ICACTIVER0
(XEN) d0v6: vGICD: unhandled word write 0xffffffff to ICACTIVER0
(XEN) d0v7: vGICD: unhandled word write 0xffffffff to ICACTIVER0
[    0.498615] Detected VIPT I-cache on CPU1

[    0.504266] Xen: initializing cpu1

[    0.504287] CPU1: Booted secondary processor [410fd034]

[    0.530658] Detected VIPT I-cache on CPU2

[    0.536254] Xen: initializing cpu2

[    0.536269] CPU2: Booted secondary processor [410fd034]

[    0.562733] Detected VIPT I-cache on CPU3

[    0.568319] Xen: initializing cpu3

[    0.568332] CPU3: Booted secondary processor [410fd034]

[    0.594821] Detected VIPT I-cache on CPU4

[    0.594830] CPU features: SANITY CHECK: Unexpected variation in 
SYS_ID_AA64MMFR0_EL1. Boot CPU: 0x00000000001122, CPU4: 0x00000000101122

[    0.594836] CPU features: Unsupported CPU feature variation detected.

[    0.600427] Xen: initializing cpu4

[    0.600443] CPU4: Booted secondary processor [410fd034]

[    0.626893] Detected VIPT I-cache on CPU5

[    0.626898] CPU features: SANITY CHECK: Unexpected variation in 
SYS_ID_AA64MMFR0_EL1. Boot CPU: 0x00000000001122, CPU5: 0x00000000101122

[    0.632479] Xen: initializing cpu5

[    0.632496] CPU5: Booted secondary processor [410fd034]

[    0.658973] Detected VIPT I-cache on CPU6

[    0.658978] CPU features: SANITY CHECK: Unexpected variation in 
SYS_ID_AA64MMFR0_EL1. Boot CPU: 0x00000000001122, CPU6: 0x00000000101122

[    0.664564] Xen: initializing cpu6

[    0.664578] CPU6: Booted secondary processor [410fd034]

[    0.691060] Detected VIPT I-cache on CPU7

[    0.691065] CPU features: SANITY CHECK: Unexpected variation in 
SYS_ID_AA64MMFR0_EL1. Boot CPU: 0x00000000001122, CPU7: 0x00000000101122

[    0.696651] Xen: initializing cpu7

[    0.696665] CPU7: Booted secondary processor [410fd034]

[    0.696741] smp: Brought up 1 node, 8 CPUs

[    0.846402] SMP: Total of 8 processors activated.

[    0.851172] CPU features: detected feature: 32-bit EL0 Support

[    0.857889] CPU: All CPU(s) started at EL1

[    0.861949] alternatives: patching kernel code

[    0.867120] devtmpfs: initialized

[    0.874142] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, 
max_idle_ns: 7645041785100000 ns

[    0.883880] futex hash table entries: 2048 (order: 6, 262144 bytes)

[    0.890666] pinctrl core: initialized pinctrl subsystem

[    0.896177] random: fast init done

[    0.896816] DMI not present or invalid.

[    0.897101] NET: Registered protocol family 16

[    0.910104] vdso: 2 pages (1 code @ ffff000008a86000, 1 data @ 
ffff000009015000)

[    0.917428] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.

[    0.931098] DMA: preallocated 256 KiB pool for atomic allocations

[    0.937167] xen:swiotlb_xen: Warning: only able to allocate 4 MB for 
software IO TLB

[    0.945347] software IO TLB [mem 0xabc00000-0xac000000] (4MB) mapped at 
[ffff8000abc00000-ffff8000abffffff]

[    0.955365] Serial: AMBA PL011 UART driver

[    0.978400] hi3660-mbox e896b000.mailbox: Mailbox enabled

[   22.029322] INFO: rcu_preempt detected stalls on CPUs/tasks:

[   22.034891]  3-...: (1 GPs behind) idle=34e/140000000000000/0 softirq=48/49 
fqs=2426 

[   22.042778]  (detected by 0, t=5252 jiffies, g=-291, c=-292, q=111)

[   22.049106] Task dump for CPU 3:

[   22.052399] swapper/0       R  running task        0     1      0 0x00000002

[   22.059514] Call trace:

[   22.062037] [<ffff0000080858c0>] __switch_to+0x98/0xb0

[   22.067232] [<0000000000000005>] 0x5

[   85.222065] INFO: rcu_preempt detected stalls on CPUs/tasks:

[   85.227626]  3-...: (1 GPs behind) idle=34e/140000000000000/0 softirq=48/49 
fqs=10304 

[   85.235605]  (detected by 0, t=21007 jiffies, g=-291, c=-292, q=111)

[   85.242018] Task dump for CPU 3:

[   85.245312] swapper/0       R  running task        0     1      0 0x00000002

[   85.252424] Call trace:

[   85.254944] [<ffff0000080858c0>] __switch_to+0x98/0xb0

[   85.260143] [<0000000000000005>] 0x5

[  148.407057] INFO: rcu_preempt detected stalls on CPUs/tasks:

[  148.412620]  3-...: (1 GPs behind) idle=34e/140000000000000/0 softirq=48/49 
fqs=18182 

[  148.420596]  (detected by 0, t=36762 jiffies, g=-291, c=-292, q=111)

[  148.427010] Task dump for CPU 3:

[  148.430303] swapper/0       R  running task        0     1      0 0x00000002

[  148.437417] Call trace:

[  148.439931] [<ffff0000080858c0>] __switch_to+0x98/0xb0

[  148.445138] [<0000000000000005>] 0x5



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.