xen with colors issue again

[Oleg Nikitenko <oleshiiwood@xxxxxxxxx>]

Hello,

I built the xlnx_rebase_4.17 branch and ran it in our environment with colors.

I ran into the next issue. Looks like some device was stucking.

It may come up immediately on start or after 20-30 minutes later even with no DomUs.

A xen command line is

"console=dtuart dtuart=serial0 dom0_mem=1800M dom0_max_vcpus=2 dom0_vcpus_pin bootscrub=0 vwfi=native sched=null timer_slop=0 llc-coloring=on llc-way-size=64K xen-llc-colors=0-1 dom0-llc-colors=2-8";

This is a first one coming up

(XEN) d0v0 Forwarding AES operation: 3254779951
Assert occurred from file xcsudma.c at line 143

I found out this is inside some DMA handler in FSBL code

void XCsuDma_Transfer(XCsuDma *InstancePtr, XCsuDma_Channel Channel,
                                        u64 Addr, u32 Size, u8 EnDataLast)

        Xil_AssertVoid(Size <= (u32)(XCSUDMA_SIZE_MAX));

This is a second one coming up.

[  188.737910] zynqmp_aes firmware:zynqmp-firmware:zynqmp-aes: ERROR: Gcm Tag mismatch
(XEN) d0v0 Forwarding AES operation: 3254779951
[  188.748496] zynqmp_aes firmware:zynqmp-firmware:zynqmp-aes: ERROR : Non word aligned data
(XEN) d0v0 Forwarding AES operation: 3254779951
[  198.826279] zynqmp_aes firmware:zynqmp-firmware:zynqmp-aes: ERROR : Non word aligned data
(XEN) d0v0 Forwarding AES operation: 3254779951
[  198.837363] zynqmp_aes firmware:zynqmp-firmware:zynqmp-aes: ERROR: Invalid
(XEN) d0v0 Forwarding AES operation: 3254779951
Received exception
MSR: 0x200, EAR: 0x181, EDR: 0x0, ESR: 0x861
[  229.916284] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[  229.916667] (detected by 1, t=5252 jiffies, g=1953, q=101)
[  229.922286] rcu: All QSes seen, last rcu_sched kthread activity 5252 (4294949715-4294944463), jiffies_till_next_fqs=1, root ->qsmask 0x0
[  229.934569] rcu: rcu_sched kthread timer wakeup didn't happen for 5251 jiffies! g1953 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200
[  229.945727] rcu: Possible timer handling issue on cpu=0 timer-softirq=3481
[  229.952734] rcu: rcu_sched kthread starved for 5252 jiffies! g1953 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=0
[  229.962940] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
[  229.971936] rcu: RCU grace-period kthread stack dump:
[  229.977041] task:rcu_sched       state:R stack:    0 pid:   12 ppid:     2 flags:0x00000008
[  229.985433] Call trace:
[  229.987939]  __switch_to+0xf4/0x13c
[  229.991486]  __schedule+0x2f0/0x690
[  229.995032]  schedule+0x5c/0xc4
[  229.998232]  schedule_timeout+0x80/0xf0
[  230.002125]  rcu_gp_fqs_loop+0xf0/0x2b4
[  230.006017]  rcu_gp_kthread+0xe8/0x100
[  230.009824]  kthread+0x120/0x130
[  230.013111]  ret_from_fork+0x10/0x20
[  230.016744] rcu: Stack dump where RCU GP kthread last ran:
[  230.022279] Task dump for CPU 0:
[  230.025567] task:tokio-runtime-w state:R  running task     stack:    0 pid:  795 ppid:   408 flags:0x00000000
[  230.035514] Call trace:
[  230.038022]  __switch_to+0xf4/0x13c
[  230.041569]  0xffffffae44320a00
[  292.936283] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[  292.936646] (detected by 1, t=21007 jiffies, g=1953, q=113)
[  292.942354] rcu: All QSes seen, last rcu_sched kthread activity 21007 (4294965470-4294944463), jiffies_till_next_fqs=1, root ->qsmask 0x0
[  292.954723] rcu: rcu_sched kthread timer wakeup didn't happen for 21006 jiffies! g1953 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200
[  292.965968] rcu: Possible timer handling issue on cpu=0 timer-softirq=3481
[  292.972975] rcu: rcu_sched kthread starved for 21007 jiffies! g1953 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=0
[  292.983268] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
[  292.992264] rcu: RCU grace-period kthread stack dump:
[  292.997368] task:rcu_sched       state:R stack:    0 pid:   12 ppid:     2 flags:0x00000008
[  293.005759] Call trace:
[  293.008266]  __switch_to+0xf4/0x13c
[  293.011814]  __schedule+0x2f0/0x690
[  293.015359]  schedule+0x5c/0xc4
[  293.018560]  schedule_timeout+0x80/0xf0
[  293.022453]  rcu_gp_fqs_loop+0xf0/0x2b4
[  293.026345]  rcu_gp_kthread+0xe8/0x100
[  293.030151]  kthread+0x120/0x130
[  293.033438]  ret_from_fork+0x10/0x20
[  293.037071] rcu: Stack dump where RCU GP kthread last ran:
[  293.042607] Task dump for CPU 0:
[  293.045895] task:tokio-runtime-w state:R  running task     stack:    0 pid:  795 ppid:   408 flags:0x00000000
[  293.055842] Call trace:
[  293.058350]  __switch_to+0xf4/0x13c
[  293.061897]  0xffffffae44320a00

Maybe someone will be able to give me some hints where I should make sense ?

Regards,

Oleg