[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re:[arm] Dom0 hangs after enable KROBE_EVENTS and/or UPROBE_EVENTS in kernel config

  • To: Julien Grall <julien@xxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • From: Oleksii Moisieiev <Oleksii_Moisieiev@xxxxxxxx>
  • Date: Wed, 21 Jul 2021 14:40:30 +0000
  • Accept-language: en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ysKhj7G1Qffb8zJBTxVn9L+4oC1dxhsRi9f5Q0kfAP8=; b=XIkvOBEYe6FAOc5DiEZAiWY9kBNNx8hEdd29oi4nLU+IzRNys1vW1d7xVkCj9Px+OVxleD57CV3Bfs+C7JIVULxapJ0/j6gwjLLjSkPWSVpZ5erZDHXhIsxxw9FzDgr4F8siqxQNbl3j1LBIm8d6ajcnp8aB/K6M5TItMqi4+sAlfGvm2bEocOJsQn56Iym0Jq9mpMGsiJtAXErq5wY+S0IuuerPOn2S1MhgNHWm97r9k5Sx7pktrdgjFrSyXmFYMgrpbtTynbJGBdh/TONYXkRK67TJrYXfcUFSY/3MX5eHZTA+fhlwlxv76RD1Csoy5/lfw/afWyLCTjW+G18k4Q==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oP796EEy9T/9JbIESt31MHWR19ftTD/AOyFXsA3xAMkRruuUONNsS0dCZ5119Yjq0lsThpoGf99Dd0PFjSEIkqB+fSjg2Wjr0fU8m8x131PYc5s3Vs4hhrLyG3HIavh9tu8+ZCtXRFXfxaq38Gcb86nx3O7LxurlozTHufb46t66JAAucp6g8EL3qB81z2gGxausjEFF8wWYDSmCRmJPSdjTywJ5BISeYTTes4/mWEEz7AtKaXXVZGqCaqAnwhwPoriYzXrpvV4SoPIBn8iTMi820IgXQd28jQzCKyV2Z/JbFjwAXsKKrJLwmcgRugTDm9pE9neIUCf8Ci/Yt9DSEA==
  • Authentication-results: xen.org; dkim=none (message not signed) header.d=none;xen.org; dmarc=none action=none header.from=epam.com;
  • Cc: Andrii Anisov <Andrii_Anisov@xxxxxxxx>, Stefano Stabellini <sstabellini@xxxxxxxxxx>
  • Delivery-date: Wed, 21 Jul 2021 14:40:46 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Thread-index: AQHXfga6kF2DIDySdk2Twa4+MGYJE6tNKWWUgAAlvH6AABguAIAAAJXm
  • Thread-topic: Re:[arm] Dom0 hangs after enable KROBE_EVENTS and/or UPROBE_EVENTS in kernel config

Hello Julien,

Thank you for the quick response.
Please answers below.

From: Julien Grall <julien@xxxxxxx>
Sent: Wednesday, July 21, 2021 4:09 PM
To: Oleksii Moisieiev <Oleksii_Moisieiev@xxxxxxxx>; xen-devel@xxxxxxxxxxxxxxxxxxxx <xen-devel@xxxxxxxxxxxxxxxxxxxx>
Cc: Andrii Anisov <Andrii_Anisov@xxxxxxxx>; Stefano Stabellini <sstabellini@xxxxxxxxxx>
Subject: Re: Dom0 hangs after enable KROBE_EVENTS and/or UPROBE_EVENTS in kernel config
>(+ Stefano)

>On 21/07/2021 12:44, Oleksii Moisieiev wrote:
>> Hello,


>Thanks for the report.

>I nearly miss this e-mail because the title doesn't suggest this is an
>Arm and I wasn't CCed. In future, would you be able to CC the Arm
>maintainers (you can find them in MAINTAINERS) and mention arm in the title?

I'm sorry for inconvenience, fixed topic and added arm maintainers to CC.

>> I've got a problem that Dom0 hangs without any output from kernel once I
>> enable CONFIG_KPROBE_EVENTS and/or CONFIG_UPROBE_EVENTS in dom0 kernel.
>> Everything works fine when kprobe/uprobe events are disabled.

>By disabled, do you mean compile out?

Yes. Just changed .config lines:
and recompiled kernel.

>> My setup:
>> Board: H3ULCB Kinfisher board
>> Xen: revision dba774896f7dd74773c14d537643b7d7477fefcd (stable-4.15)
>> https://urldefense.com/v3/__https://github.com/xen-project/xen.git__;!!GF_29dbcQIUBPA!m4NHC2XbbSHWWZjQ7CX1ZZhaET6l0bQhZo581jtCmpst8E8JBp8Qri3haIaks6cbo7Ri$ [github[.]com]
>> <https://urldefense.com/v3/__https://github.com/xen-project/xen.git__;!!GF_29dbcQIUBPA!m4NHC2XbbSHWWZjQ7CX1ZZhaET6l0bQhZo581jtCmpst8E8JBp8Qri3haIaks6cbo7Ri$ [github[.]com]>;
>> Kernel: revision 09162bc32c880a791c6c0668ce0745cf7958f576 (v5.10-rc4)

>Hmmm... 5.10 was released a few months ago and there are probably a few
>stable release for the version. Can you try the latest 5.10 stable?

Switched to tag v5.10 rev: 
2c85ebc57b3e of https://github.com/torvalds/linux.git
and got the same problem, that I see no output from kernel. All tests were done with earlycon parameter set in the kernel cmdline.

>> https://urldefense.com/v3/__https://github.com/torvalds/linux.git__;!!GF_29dbcQIUBPA!m4NHC2XbbSHWWZjQ7CX1ZZhaET6l0bQhZo581jtCmpst8E8JBp8Qri3haIaks29w69MC$ [github[.]com]
>> <https://urldefense.com/v3/__https://github.com/torvalds/linux.git__;!!GF_29dbcQIUBPA!m4NHC2XbbSHWWZjQ7CX1ZZhaET6l0bQhZo581jtCmpst8E8JBp8Qri3haIaks29w69MC$ [github[.]com]>;
>> kernel config: see attached;
>> dtb: see attached;

>Please avoid large attachment as they will be duplicated on every
>mailbox. Instead, in the future, please upload them somewhere (your own
>webserve, pastebin...) and provide a link in the e-mail.

I'm sorry for that.

>> If kprobe/uprobe events are enabled - I see no output after xen switched
>> input to Dom0, if disabled - system boots up successfully.
>The console subsystem tends to be enabled quite late in the boot
>process. So this may mean a panic during early boot.

>If you haven't done yet, I would suggest to add earlycon=xenboot on the
>dom0 command line. This will print some messages during early boot.

All tests were done with earlycon parameter set in the kernel command line (xen, dom0-bootargs).

>> Both configs work fine when I boot without xen.
>> Dom0 information from Xen console shows that only one CPU works, and PC
>> stays in "__arch_counter_get_cntvct" function on read_sysreg call. //
>> I did further investigation and found that kernel 5.4 doesn't have such
>> kind of issues.
>> After bisecting kernel,between 5.10 and 5.4, I found that output
>> disappeared on commit:
>> 76085aff29f585139a37a10ea0a7daa63f70872c

> From the information you provided so far, I am a bit confused how this
>could be the source of the problem. But given this is not the latest
>5.10, I will wait for you to confirm the bug is still present before
>providing more input.

I was confused with this commit either. As I mentioned above, I've checked with the latest stable 5.10 kernel and still got the same problem.

>> Another issue, which was revealed after I got kernel output was kernel
>> oops with message that CPU is in inconsistent state.
>> [0.415612] EFI services will not be available.
>> [0.420267] smp: Bringing up secondary CPUs ...
>> [0.425185] Detected PIPT I-cache on CPU1
>> [0.425267] Xen: initializing cpu1
>> [0.425292] CPU1: Booted secondary processor 0x0000000001 [0x411fd073]
>> [0.425815] Detected PIPT I-cache on CPU2
>> [0.425879] Xen: initializing cpu2
>> [0.425899] CPU2: Booted secondary processor 0x0000000002 [0x411fd073]
>> [0.426362] Detected PIPT I-cache on CPU3
>> [0.426425] Xen: initializing cpu3
>> [0.426444] CPU3: Booted secondary processor 0x0000000003 [0x411fd073]
>> [0.426537] smp: Brought up 1 node, 4 CPUs
>> [0.472807] SMP: Total of 4 processors activated.
>> [0.477551] CPU features: detected: 32-bit EL0 Support
>> [0.482745] CPU features: detected: CRC32 instructions
>> [0.499470] ------------[ cut here ]------------
>> [0.504034] CPU: CPUs started in inconsistent modes

>Looking at Linux 5.7 code, this is printed when not all the CPUs are
>booted in the same mode (i.e. EL1 or EL2).

>This is quite odd. So let me ask a question first, did you see this
>error during the bisection or on the latest 5.7?

Switched to kernel v5.7 tag, rev:3d77e6a8804.
On 5.7 kernel I can see printk output, but getting CPUs started in inconsistent modes error.
Also, I tried with hmp-unsafe=false ( in xen cmdline, so only 0-3 CPU were enabled. And still got the same issue.


>Julien Grall



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.