Re: [bug report] xen/arm64: singlestep doesn't work on Dom0

[Alex Bennée <alex.bennee@xxxxxxxxxx>]

Masami Hiramatsu <masami.hiramatsu@xxxxxxxxxx> writes:

> Hello,
>
> When I tested the kprobes on Dom0 kernel, it caused a kernel panic.
>
> Here is an example, to clarify the bug is in the single-stepping
> (software-step exception), I added a printk() where the kprobes setup
> single-stepping.
>
> root@develbox:~# cd /sys/kernel/debug/tracing/
> root@develbox:/sys/kernel/debug/tracing# echo p vfs_read > kprobe_events
> root@develbox:/sys/kernel/debug/tracing# echo 1 > events/kprobes/enable
> root@develbox:/sys/kernel/debug/tracing# [  112.282480] kprobes:
> singlestep ool insn at ffff800011785000    <--- This shows single
> stepping buffer (trampoline)
> [  112.288077] ------------[ cut here ]------------
> [  112.292745] kernel BUG at arch/arm64/kernel/traps.c:406!
> [  112.298129] Internal error: Oops - BUG: 0 [#1] SMP
> [  112.302987] Modules linked in: fuse bridge stp llc binfmt_misc
> nls_ascii nls_cp437 vfat fat ahci libahci libata hid_generic udlfb
> scsi_mod aes_ce_blk crypto_simd evdev cryptd aes_ce_cipher usbhid
> ghash_ce realtek gf128mul hid netsec sha2_ce mdio_devres i2c_algo_bit
> of_mdio sha256_arm64 fb_ddc sha1_ce fixed_phy gpio_keys leds_gpio
> libphy bpf_preload ip_tables x_tables autofs4 xhci_pci
> xhci_pci_renesas xhci_hcd usbcore gpio_mb86s7x
> [  112.341097] CPU: 13 PID: 1045 Comm: bash Not tainted 5.10.0-rc1+ #44
> [  112.347515] Hardware name: Socionext Developer Box (DT)
> [  112.352813] pstate: 00000085 (nzcv daIf -PAN -UAO -TCO BTYPE=--)
> [  112.358897] pc : do_undefinstr+0x354/0x378
> [  112.363053] lr : do_undefinstr+0x270/0x378
> [  112.367218] sp : ffff8000122fbc50
> [  112.370603] x29: ffff8000122fbc50 x28: ffff00084bc9e080
> [  112.375985] x27: 0000000000000000 x26: 0000000000000000
> [  112.381366] x25: 0000000000000000 x24: 0000000000000000
> [  112.386748] x23: 0000000080000085 x22: ffff800011785004
> [  112.392129] x21: ffff8000122fbe00 x20: ffff8000122fbcc0
> [  112.397511] x19: ffff800011249988 x18: 0000000000000000
> [  112.402892] x17: 0000000000000000 x16: 0000000000000000
> [  112.408274] x15: 0000000000000000 x14: 0000000000000000
> [  112.413655] x13: 0000000000000000 x12: 0000000000000000
> [  112.419037] x11: 0000000000000000 x10: 0000000000000000
> [  112.424426] x9 : ffff800010314614 x8 : 0000000000000000
> [  112.429801] x7 : 0000000000000000 x6 : ffff8000122fbca8
> [  112.435189] x5 : 0000000000000000 x4 : ffff800011400110
> [  112.440564] x3 : 00000000d5300000 x2 : ffff800011255f78
> [  112.445946] x1 : ffff800011400110 x0 : 0000000080000085
> [  112.451328] Call trace:
> [  112.453848]  do_undefinstr+0x354/0x378
> [  112.457669]  el1_sync_handler+0xa8/0x138
> [  112.461658]  el1_sync+0x7c/0x100
> [  112.464958]  0xffff800011785004     /// <- Undefined instruction
> error happens on the next instruction of single stepping buffer.
> [  112.468172]  __arm64_sys_read+0x24/0x30
> [  112.472078]  el0_svc_common.constprop.3+0x94/0x178
> [  112.476936]  do_el0_svc+0x2c/0x98
> [  112.480321]  el0_sync_handler+0x118/0x168
> [  112.484407]  el0_sync+0x158/0x180
> [  112.487789] Code: d2801400 17ffffbe a9025bf5 f9001bf7 (d4210000)
> [  112.493951] ---[ end trace 3564a3bf75d1618c ]---
>
> So, this seems that the Linux kernel couldn't catch the software-step 
> exception.
>
> I confirmed the same kernel doesn't cause this error without Xen. I
> guess Xen is not correctly setting the debug registers when the cpu
> goes EL1.

Having worked on the arm64 KVM debug logic I have a little familiarity
with how it works on KVM. 

> (Or, would we handle debug exceptions in Xen and transfer it to EL1
> OS? I'm not sure how it was designed)

Xen looks as though it should be trapping a chunk of debug accesses:

    /* Trap Debug and Performance Monitor accesses */
    WRITE_SYSREG(HDCR_TDRA|HDCR_TDOSA|HDCR_TDA|HDCR_TPM|HDCR_TPMCR,
                 MDCR_EL2);

but it doesn't set HDCR_TDE so it won't re-route debug exceptions to EL2
which should be the place that is dealing with them. Also I can't see
where the debug registers are saved/restored. In KVM we maintain a
shadow copy of the guest debug register state while guest debugging is
in effect as any breakpoints you want to trigger need to be copied
across.

I also can't see where CPSR_SS or DBG_MDSCR_SS is set in the mdscr.

FWIW most of the logic in KVM is contained in:

  arch/arm64/kvm/debug.c

with some smatterings of handling the traps and context swapping
elsewhere in the code.

>
> Thank you,


-- 
Alex Bennée