Re: [bug report] xen/arm64: singlestep doesn't work on Dom0

[Masami Hiramatsu <masami.hiramatsu@xxxxxxxxxx>]

Hi Alex,

2020年10月28日(水) 21:03 Alex Bennée <alex.bennee@xxxxxxxxxx>:
>
>
> Masami Hiramatsu <masami.hiramatsu@xxxxxxxxxx> writes:
>
> > Hello,
> >
> > When I tested the kprobes on Dom0 kernel, it caused a kernel panic.
> >
> > Here is an example, to clarify the bug is in the single-stepping
> > (software-step exception), I added a printk() where the kprobes setup
> > single-stepping.
> >
> > root@develbox:~# cd /sys/kernel/debug/tracing/
> > root@develbox:/sys/kernel/debug/tracing# echo p vfs_read > kprobe_events
> > root@develbox:/sys/kernel/debug/tracing# echo 1 > events/kprobes/enable
> > root@develbox:/sys/kernel/debug/tracing# [  112.282480] kprobes:
> > singlestep ool insn at ffff800011785000    <--- This shows single
> > stepping buffer (trampoline)
> > [  112.288077] ------------[ cut here ]------------
> > [  112.292745] kernel BUG at arch/arm64/kernel/traps.c:406!
> > [  112.298129] Internal error: Oops - BUG: 0 [#1] SMP
> > [  112.302987] Modules linked in: fuse bridge stp llc binfmt_misc
> > nls_ascii nls_cp437 vfat fat ahci libahci libata hid_generic udlfb
> > scsi_mod aes_ce_blk crypto_simd evdev cryptd aes_ce_cipher usbhid
> > ghash_ce realtek gf128mul hid netsec sha2_ce mdio_devres i2c_algo_bit
> > of_mdio sha256_arm64 fb_ddc sha1_ce fixed_phy gpio_keys leds_gpio
> > libphy bpf_preload ip_tables x_tables autofs4 xhci_pci
> > xhci_pci_renesas xhci_hcd usbcore gpio_mb86s7x
> > [  112.341097] CPU: 13 PID: 1045 Comm: bash Not tainted 5.10.0-rc1+ #44
> > [  112.347515] Hardware name: Socionext Developer Box (DT)
> > [  112.352813] pstate: 00000085 (nzcv daIf -PAN -UAO -TCO BTYPE=--)
> > [  112.358897] pc : do_undefinstr+0x354/0x378
> > [  112.363053] lr : do_undefinstr+0x270/0x378
> > [  112.367218] sp : ffff8000122fbc50
> > [  112.370603] x29: ffff8000122fbc50 x28: ffff00084bc9e080
> > [  112.375985] x27: 0000000000000000 x26: 0000000000000000
> > [  112.381366] x25: 0000000000000000 x24: 0000000000000000
> > [  112.386748] x23: 0000000080000085 x22: ffff800011785004
> > [  112.392129] x21: ffff8000122fbe00 x20: ffff8000122fbcc0
> > [  112.397511] x19: ffff800011249988 x18: 0000000000000000
> > [  112.402892] x17: 0000000000000000 x16: 0000000000000000
> > [  112.408274] x15: 0000000000000000 x14: 0000000000000000
> > [  112.413655] x13: 0000000000000000 x12: 0000000000000000
> > [  112.419037] x11: 0000000000000000 x10: 0000000000000000
> > [  112.424426] x9 : ffff800010314614 x8 : 0000000000000000
> > [  112.429801] x7 : 0000000000000000 x6 : ffff8000122fbca8
> > [  112.435189] x5 : 0000000000000000 x4 : ffff800011400110
> > [  112.440564] x3 : 00000000d5300000 x2 : ffff800011255f78
> > [  112.445946] x1 : ffff800011400110 x0 : 0000000080000085
> > [  112.451328] Call trace:
> > [  112.453848]  do_undefinstr+0x354/0x378
> > [  112.457669]  el1_sync_handler+0xa8/0x138
> > [  112.461658]  el1_sync+0x7c/0x100
> > [  112.464958]  0xffff800011785004     /// <- Undefined instruction
> > error happens on the next instruction of single stepping buffer.
> > [  112.468172]  __arm64_sys_read+0x24/0x30
> > [  112.472078]  el0_svc_common.constprop.3+0x94/0x178
> > [  112.476936]  do_el0_svc+0x2c/0x98
> > [  112.480321]  el0_sync_handler+0x118/0x168
> > [  112.484407]  el0_sync+0x158/0x180
> > [  112.487789] Code: d2801400 17ffffbe a9025bf5 f9001bf7 (d4210000)
> > [  112.493951] ---[ end trace 3564a3bf75d1618c ]---
> >
> > So, this seems that the Linux kernel couldn't catch the software-step 
> > exception.
> >
> > I confirmed the same kernel doesn't cause this error without Xen. I
> > guess Xen is not correctly setting the debug registers when the cpu
> > goes EL1.
>
> Having worked on the arm64 KVM debug logic I have a little familiarity
> with how it works on KVM.
>
> > (Or, would we handle debug exceptions in Xen and transfer it to EL1
> > OS? I'm not sure how it was designed)
>
> Xen looks as though it should be trapping a chunk of debug accesses:
>
>     /* Trap Debug and Performance Monitor accesses */
>     WRITE_SYSREG(HDCR_TDRA|HDCR_TDOSA|HDCR_TDA|HDCR_TPM|HDCR_TPMCR,
>                  MDCR_EL2);
>
> but it doesn't set HDCR_TDE so it won't re-route debug exceptions to EL2
> which should be the place that is dealing with them.

Would we need to re-route the debug exception to EL2? I thought it was
OK to send it to EL1 (Dom0 kernel).

> Also I can't see
> where the debug registers are saved/restored. In KVM we maintain a
> shadow copy of the guest debug register state while guest debugging is
> in effect as any breakpoints you want to trigger need to be copied
> across.

Hmm, so KVM will trap the access of the debug registers in EL1?


> I also can't see where CPSR_SS or DBG_MDSCR_SS is set in the mdscr.
>
> FWIW most of the logic in KVM is contained in:
>
>   arch/arm64/kvm/debug.c
>
> with some smatterings of handling the traps and context swapping
> elsewhere in the code.

Thank you,

-- 
Masami Hiramatsu