[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: xen cache colors in ARM



Hi Oleg, 

Here is the issue from your logs:

SError Interrupt on CPU0, code 0xbe000000 -- SError

SErrors are special signals to notify software of serious hardware
errors.  Something is going very wrong. Defective hardware is a
possibility.  Another possibility if software accessing address ranges
that it is not supposed to, sometimes it causes SErrors.

Cheers,

Stefano



On Mon, 24 Apr 2023, Oleg Nikitenko wrote:

> Hello,
> 
> Thanks guys.
> I found out where the problem was.
> Now dom0 booted more. But I have a new one.
> This is a kernel panic during Dom0 loading.
> Maybe someone is able to suggest something ?
> 
> Regards,
> O.
> 
> [    3.771362] sfp_register_bus: upstream ops attach
> [    3.776119] sfp_register_bus: Bus registered
> [    3.780459] sfp_register_socket: register sfp_bus succeeded
> [    3.789399] of_cfs_init
> [    3.789499] of_cfs_init: OK
> [    3.791685] clk: Not disabling unused clocks
> [   11.010355] SError Interrupt on CPU0, code 0xbe000000 -- SError
> [   11.010380] CPU: 0 PID: 9 Comm: kworker/u4:0 Not tainted 
> 5.15.72-xilinx-v2022.1 #1
> [   11.010393] Workqueue: events_unbound async_run_entry_fn
> [   11.010414] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> [   11.010422] pc : simple_write_end+0xd0/0x130
> [   11.010431] lr : generic_perform_write+0x118/0x1e0
> [   11.010438] sp : ffffffc00809b910
> [   11.010441] x29: ffffffc00809b910 x28: 0000000000000000 x27: 
> ffffffef69ba88c0
> [   11.010451] x26: 0000000000003eec x25: ffffff807515db00 x24: 
> 0000000000000000
> [   11.010459] x23: ffffffc00809ba90 x22: 0000000002aac000 x21: 
> ffffff807315a260
> [   11.010472] x20: 0000000000001000 x19: fffffffe02000000 x18: 
> 0000000000000000
> [   11.010481] x17: 00000000ffffffff x16: 0000000000008000 x15: 
> 0000000000000000
> [   11.010490] x14: 0000000000000000 x13: 0000000000000000 x12: 
> 0000000000000000
> [   11.010498] x11: 0000000000000000 x10: 0000000000000000 x9 : 
> 0000000000000000
> [   11.010507] x8 : 0000000000000000 x7 : ffffffef693ba680 x6 : 
> 000000002d89b700
> [   11.010515] x5 : fffffffe02000000 x4 : ffffff807315a3c8 x3 : 
> 0000000000001000
> [   11.010524] x2 : 0000000002aab000 x1 : 0000000000000001 x0 : 
> 0000000000000005
> [   11.010534] Kernel panic - not syncing: Asynchronous SError Interrupt
> [   11.010539] CPU: 0 PID: 9 Comm: kworker/u4:0 Not tainted 
> 5.15.72-xilinx-v2022.1 #1
> [   11.010545] Hardware name: D14 Viper Board - White Unit (DT)
> [   11.010548] Workqueue: events_unbound async_run_entry_fn
> [   11.010556] Call trace:
> [   11.010558]  dump_backtrace+0x0/0x1c4
> [   11.010567]  show_stack+0x18/0x2c
> [   11.010574]  dump_stack_lvl+0x7c/0xa0
> [   11.010583]  dump_stack+0x18/0x34
> [   11.010588]  panic+0x14c/0x2f8
> [   11.010597]  print_tainted+0x0/0xb0
> [   11.010606]  arm64_serror_panic+0x6c/0x7c
> [   11.010614]  do_serror+0x28/0x60
> [   11.010621]  el1h_64_error_handler+0x30/0x50
> [   11.010628]  el1h_64_error+0x78/0x7c
> [   11.010633]  simple_write_end+0xd0/0x130
> [   11.010639]  generic_perform_write+0x118/0x1e0
> [   11.010644]  __generic_file_write_iter+0x138/0x1c4
> [   11.010650]  generic_file_write_iter+0x78/0xd0
> [   11.010656]  __kernel_write+0xfc/0x2ac
> [   11.010665]  kernel_write+0x88/0x160
> [   11.010673]  xwrite+0x44/0x94
> [   11.010680]  do_copy+0xa8/0x104
> [   11.010686]  write_buffer+0x38/0x58
> [   11.010692]  flush_buffer+0x4c/0xbc
> [   11.010698]  __gunzip+0x280/0x310
> [   11.010704]  gunzip+0x1c/0x28
> [   11.010709]  unpack_to_rootfs+0x170/0x2b0
> [   11.010715]  do_populate_rootfs+0x80/0x164
> [   11.010722]  async_run_entry_fn+0x48/0x164
> [   11.010728]  process_one_work+0x1e4/0x3a0
> [   11.010736]  worker_thread+0x7c/0x4c0
> [   11.010743]  kthread+0x120/0x130
> [   11.010750]  ret_from_fork+0x10/0x20
> [   11.010757] SMP: stopping secondary CPUs
> [   11.010784] Kernel Offset: 0x2f61200000 from 0xffffffc008000000
> [   11.010788] PHYS_OFFSET: 0x0
> [   11.010790] CPU features: 0x00000401,00000842
> [   11.010795] Memory Limit: none
> [   11.277509] ---[ end Kernel panic - not syncing: Asynchronous SError 
> Interrupt ]---
> 
> пт, 21 апр. 2023 г. в 15:52, Michal Orzel <michal.orzel@xxxxxxx>:
>       Hi Oleg,
> 
>       On 21/04/2023 14:49, Oleg Nikitenko wrote:
>       >       
>       >
>       >
>       > Hello Michal,
>       >
>       > I was not able to enable earlyprintk in the xen for now.
>       > I decided to choose another way.
>       > This is a xen's command line that I found out completely.
>       >
>       > (XEN) $$$$ console=dtuart dtuart=serial0 dom0_mem=1600M 
> dom0_max_vcpus=2 dom0_vcpus_pin bootscrub=0 vwfi=native sched=null
>       timer_slop=0
>       Yes, adding a printk() in Xen was also a good idea.
> 
>       >
>       > So you are absolutely right about a command line.
>       > Now I am going to find out why xen did not have the correct 
> parameters from the device tree.
>       Maybe you will find this document helpful:
>       
> https://github.com/Xilinx/xen/blob/xlnx_rebase_4.16/docs/misc/arm/device-tree/booting.txt
> 
>       ~Michal
> 
>       >
>       > Regards,
>       > Oleg
>       >
>       > пт, 21 апр. 2023 г. в 11:16, Michal Orzel <michal.orzel@xxxxxxx 
> <mailto:michal.orzel@xxxxxxx>>:
>       >
>       >
>       >     On 21/04/2023 10:04, Oleg Nikitenko wrote:
>       >     >       
>       >     >
>       >     >
>       >     > Hello Michal,
>       >     >
>       >     > Yes, I use yocto.
>       >     >
>       >     > Yesterday all day long I tried to follow your suggestions.
>       >     > I faced a problem.
>       >     > Manually in the xen config build file I pasted the strings:
>       >     In the .config file or in some Yocto file (listing additional 
> Kconfig options) added to SRC_URI?
>       >     You shouldn't really modify .config file but if you do, you 
> should execute "make olddefconfig" afterwards.
>       >
>       >     >
>       >     > CONFIG_EARLY_PRINTK
>       >     > CONFIG_EARLY_PRINTK_ZYNQMP
>       >     > CONFIG_EARLY_UART_CHOICE_CADENCE
>       >     I hope you added =y to them.
>       >
>       >     Anyway, you have at least the following solutions:
>       >     1) Run bitbake xen -c menuconfig to properly set early printk
>       >     2) Find out how you enable other Kconfig options in your project 
> (e.g. CONFIG_COLORING=y that is not enabled by default)
>       >     3) Append the following to "xen/arch/arm/configs/arm64_defconfig":
>       >     CONFIG_EARLY_PRINTK_ZYNQMP=y
>       >
>       >     ~Michal
>       >
>       >     >
>       >     > Host hangs in build time. 
>       >     > Maybe I did not set something in the config build file ?
>       >     >
>       >     > Regards,
>       >     > Oleg
>       >     >
>       >     > чт, 20 апр. 2023 г. в 11:57, Oleg Nikitenko 
> <oleshiiwood@xxxxxxxxx <mailto:oleshiiwood@xxxxxxxxx>
>       <mailto:oleshiiwood@xxxxxxxxx <mailto:oleshiiwood@xxxxxxxxx>>>:
>       >     >
>       >     >     Thanks Michal,
>       >     >
>       >     >     You gave me an idea.
>       >     >     I am going to try it today.
>       >     >
>       >     >     Regards,
>       >     >     O.
>       >     >
>       >     >     чт, 20 апр. 2023 г. в 11:56, Oleg Nikitenko 
> <oleshiiwood@xxxxxxxxx <mailto:oleshiiwood@xxxxxxxxx>
>       <mailto:oleshiiwood@xxxxxxxxx <mailto:oleshiiwood@xxxxxxxxx>>>:
>       >     >
>       >     >         Thanks Stefano.
>       >     >
>       >     >         I am going to do it today.
>       >     >
>       >     >         Regards,
>       >     >         O.
>       >     >
>       >     >         ср, 19 апр. 2023 г. в 23:05, Stefano Stabellini 
> <sstabellini@xxxxxxxxxx <mailto:sstabellini@xxxxxxxxxx>
>       <mailto:sstabellini@xxxxxxxxxx <mailto:sstabellini@xxxxxxxxxx>>>:
>       >     >
>       >     >             On Wed, 19 Apr 2023, Oleg Nikitenko wrote:
>       >     >             > Hi Michal,
>       >     >             >
>       >     >             > I corrected xen's command line.
>       >     >             > Now it is
>       >     >             > xen,xen-bootargs = "console=dtuart dtuart=serial0 
> dom0_mem=1600M dom0_max_vcpus=2 dom0_vcpus_pin
>       bootscrub=0 vwfi=native sched=null
>       >     >             > timer_slop=0 way_size=65536 xen_colors=0-3 
> dom0_colors=4-7";
>       >     >
>       >     >             4 colors is way too many for xen, just do 
> xen_colors=0-0. There is no
>       >     >             advantage in using more than 1 color for Xen.
>       >     >
>       >     >             4 colors is too few for dom0, if you are giving 
> 1600M of memory to Dom0.
>       >     >             Each color is 256M. For 1600M you should give at 
> least 7 colors. Try:
>       >     >
>       >     >             xen_colors=0-0 dom0_colors=1-8
>       >     >
>       >     >
>       >     >
>       >     >             > Unfortunately the result was the same.
>       >     >             >
>       >     >             > (XEN)  - Dom0 mode: Relaxed
>       >     >             > (XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit 
> VMID
>       >     >             > (XEN) P2M: 3 levels with order-1 root, VTCR 
> 0x0000000080023558
>       >     >             > (XEN) Scheduling granularity: cpu, 1 CPU per 
> sched-resource
>       >     >             > (XEN) Coloring general information
>       >     >             > (XEN) Way size: 64kB
>       >     >             > (XEN) Max. number of colors available: 16
>       >     >             > (XEN) Xen color(s): [ 0 ]
>       >     >             > (XEN) alternatives: Patching with alt table 
> 00000000002cc690 -> 00000000002ccc0c
>       >     >             > (XEN) Color array allocation failed for dom0
>       >     >             > (XEN)
>       >     >             > (XEN) ****************************************
>       >     >             > (XEN) Panic on CPU 0:
>       >     >             > (XEN) Error creating domain 0
>       >     >             > (XEN) ****************************************
>       >     >             > (XEN)
>       >     >             > (XEN) Reboot in five seconds...
>       >     >             >
>       >     >             > I am going to find out how command line arguments 
> passed and parsed.
>       >     >             >
>       >     >             > Regards,
>       >     >             > Oleg
>       >     >             >
>       >     >             > ср, 19 апр. 2023 г. в 11:25, Oleg Nikitenko 
> <oleshiiwood@xxxxxxxxx <mailto:oleshiiwood@xxxxxxxxx>
>       <mailto:oleshiiwood@xxxxxxxxx <mailto:oleshiiwood@xxxxxxxxx>>>:
>       >     >             >       Hi Michal,
>       >     >             >
>       >     >             > You put my nose into the problem. Thank you.
>       >     >             > I am going to use your point.
>       >     >             > Let's see what happens.
>       >     >             >
>       >     >             > Regards,
>       >     >             > Oleg
>       >     >             >
>       >     >             >
>       >     >             > ср, 19 апр. 2023 г. в 10:37, Michal Orzel 
> <michal.orzel@xxxxxxx <mailto:michal.orzel@xxxxxxx>
>       <mailto:michal.orzel@xxxxxxx <mailto:michal.orzel@xxxxxxx>>>:
>       >     >             >       Hi Oleg,
>       >     >             >
>       >     >             >       On 19/04/2023 09:03, Oleg Nikitenko wrote:
>       >     >             >       >       
>       >     >             >       >
>       >     >             >       >
>       >     >             >       > Hello Stefano,
>       >     >             >       >
>       >     >             >       > Thanks for the clarification.
>       >     >             >       > My company uses yocto for image 
> generation.
>       >     >             >       > What kind of information do you need to 
> consult me in this case ?
>       >     >             >       >
>       >     >             >       > Maybe modules sizes/addresses which were 
> mentioned by @Julien Grall <mailto:julien@xxxxxxx
>       <mailto:julien@xxxxxxx> <mailto:julien@xxxxxxx 
> <mailto:julien@xxxxxxx>>> ?
>       >     >             >
>       >     >             >       Sorry for jumping into discussion, but 
> FWICS the Xen command line you provided seems to be not the
>       one
>       >     >             >       Xen booted with. The error you are 
> observing most likely is due to dom0 colors configuration not
>       being
>       >     >             >       specified (i.e. lack of dom0_colors=<> 
> parameter). Although in the command line you provided, this
>       parameter
>       >     >             >       is set, I strongly doubt that this is the 
> actual command line in use.
>       >     >             >
>       >     >             >       You wrote:
>       >     >             >       xen,xen-bootargs = "console=dtuart 
> dtuart=serial0 dom0_mem=1600M dom0_max_vcpus=2 dom0_vcpus_pin
>       bootscrub=0 vwfi=native
>       >     >             >       sched=null timer_slop=0 way_szize=65536 
> xen_colors=0-3 dom0_colors=4-7";
>       >     >             >
>       >     >             >       but:
>       >     >             >       1) way_szize has a typo
>       >     >             >       2) you specified 4 colors (0-3) for Xen, 
> but the boot log says that Xen has only one:
>       >     >             >       (XEN) Xen color(s): [ 0 ]
>       >     >             >
>       >     >             >       This makes me believe that no colors 
> configuration actually end up in command line that Xen booted
>       with.
>       >     >             >       Single color for Xen is a "default if not 
> specified" and way size was probably calculated by asking
>       HW.
>       >     >             >
>       >     >             >       So I would suggest to first cross-check the 
> command line in use.
>       >     >             >
>       >     >             >       ~Michal
>       >     >             >
>       >     >             >
>       >     >             >       >
>       >     >             >       > Regards,
>       >     >             >       > Oleg
>       >     >             >       >
>       >     >             >       > вт, 18 апр. 2023 г. в 20:44, Stefano 
> Stabellini <sstabellini@xxxxxxxxxx
>       <mailto:sstabellini@xxxxxxxxxx> <mailto:sstabellini@xxxxxxxxxx 
> <mailto:sstabellini@xxxxxxxxxx>> <mailto:sstabellini@xxxxxxxxxx
>       <mailto:sstabellini@xxxxxxxxxx> <mailto:sstabellini@xxxxxxxxxx 
> <mailto:sstabellini@xxxxxxxxxx>>>>:
>       >     >             >       >
>       >     >             >       >     On Tue, 18 Apr 2023, Oleg Nikitenko 
> wrote:
>       >     >             >       >     > Hi Julien,
>       >     >             >       >     >
>       >     >             >       >     > >> This feature has not been merged 
> in Xen upstream yet
>       >     >             >       >     >
>       >     >             >       >     > > would assume that upstream + the 
> series on the ML [1] work
>       >     >             >       >     >
>       >     >             >       >     > Please clarify this point.
>       >     >             >       >     > Because the two thoughts are 
> controversial.
>       >     >             >       >
>       >     >             >       >     Hi Oleg,
>       >     >             >       >
>       >     >             >       >     As Julien wrote, there is nothing 
> controversial. As you are aware,
>       >     >             >       >     Xilinx maintains a separate Xen tree 
> specific for Xilinx here:
>       >     >             >       >     https://github.com/xilinx/xen 
> <https://github.com/xilinx/xen> <https://github.com/xilinx/xen
>       <https://github.com/xilinx/xen>> <https://github.com/xilinx/xen 
> <https://github.com/xilinx/xen> <https://github.com/xilinx/xen
>       <https://github.com/xilinx/xen>>>
>       >     >             >       >
>       >     >             >       >     and the branch you are using 
> (xlnx_rebase_4.16) comes from there.
>       >     >             >       >
>       >     >             >       >
>       >     >             >       >     Instead, the upstream Xen tree lives 
> here:
>       >     >             >       >     
> https://xenbits.xen.org/gitweb/?p=xen.git;a=summary
>       <https://xenbits.xen.org/gitweb/?p=xen.git;a=summary> 
> <https://xenbits.xen.org/gitweb/?p=xen.git;a=summary
>       <https://xenbits.xen.org/gitweb/?p=xen.git;a=summary>> 
> <https://xenbits.xen.org/gitweb/?p=xen.git;a=summary
>       <https://xenbits.xen.org/gitweb/?p=xen.git;a=summary> 
> <https://xenbits.xen.org/gitweb/?p=xen.git;a=summary
>       <https://xenbits.xen.org/gitweb/?p=xen.git;a=summary>>>
>       >     >             >       >
>       >     >             >       >
>       >     >             >       >     The Cache Coloring feature that you 
> are trying to configure is present
>       >     >             >       >     in xlnx_rebase_4.16, but not yet 
> present upstream (there is an
>       >     >             >       >     outstanding patch series to add cache 
> coloring to Xen upstream but it
>       >     >             >       >     hasn't been merged yet.)
>       >     >             >       >
>       >     >             >       >
>       >     >             >       >     Anyway, if you are using 
> xlnx_rebase_4.16 it doesn't matter too much for
>       >     >             >       >     you as you already have Cache 
> Coloring as a feature there.
>       >     >             >       >
>       >     >             >       >
>       >     >             >       >     I take you are using ImageBuilder to 
> generate the boot configuration? If
>       >     >             >       >     so, please post the ImageBuilder 
> config file that you are using.
>       >     >             >       >
>       >     >             >       >     But from the boot message, it looks 
> like the colors configuration for
>       >     >             >       >     Dom0 is incorrect.
>       >     >             >       >
>       >     >             >
>       >     >             >
>       >     >             >
>       >     >
>       >
> 
> 
> 

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.