Re: [Xen-devel] Regression with commit "x86/pv: Drop int80_bounce from struct pv_vcpu" f75b1a5247b3b311d3aa50de4c0e5f2d68085cb1

[Juergen Gross <jgross@xxxxxxxx>]

On 13/03/18 12:34, Sander Eikelenboom wrote:
> On 12/03/18 21:04, Boris Ostrovsky wrote:
>> On 03/12/2018 03:05 PM, Andrew Cooper wrote:
>>> On 10/03/18 16:27, Andrew Cooper wrote:
>>>> On 10/03/2018 16:14, Sander Eikelenboom wrote:
>>>>> Hi Andrew,
>>>>>
>>>>> It seems commit "x86/pv: Drop int80_bounce from struct pv_vcpu" 
>>>>> (f75b1a5247b3b311d3aa50de4c0e5f2d68085cb1) causes an issue on my machine, 
>>>>> an AMD phenom X6.
>>>>>
>>>>> When trying to installing a new kernel package which runs the Debian
>>>>> update-initramfs tools with xen-unstable which happened to be at commit 
>>>>> c9bd8a73656d7435b1055ee8825823aee995993e as last commit the tool stalls
>>>>> and i get this kernel splat:
>>>>>
>>>>> [  284.910674] BUG: unable to handle kernel NULL pointer dereference at 
>>>>> 0000000000000000
>>>>> [  284.919696] IP:           (null)
>>>>> [  284.928315] PGD 0 P4D 0 
>>>>> [  284.943343] Oops: 0010 [#1] SMP NOPTI
>>>>> [  284.957008] Modules linked in:
>>>>> [  284.965521] CPU: 5 PID: 24729 Comm: ld-linux.so.2 Not tainted 
>>>>> 4.16.0-rc4-20180305-linus-pvhpatches-doflr+ #1
>>>>> [  284.974154] Hardware name: MSI MS-7640/890FXA-GD70 (MS-7640)  , BIOS 
>>>>> V1.8B1 09/13/2010
>>>>> [  284.983198] RIP: e030:          (null)
>>>>> [  284.992006] RSP: e02b:ffffc90001497ed8 EFLAGS: 00010286
>>>>> [  285.000612] RAX: 0000000000000000 RBX: ffff880074c64500 RCX: 
>>>>> ffffffff82f8d1c0
>>>>> [  285.009122] RDX: ffffffff82f8d1c0 RSI: 0000000020020002 RDI: 
>>>>> ffffffff82f8d1c0
>>>>> [  285.017598] RBP: ffff880074c64b7c R08: 0000000000000000 R09: 
>>>>> 0000000000000000
>>>>> [  285.025999] R10: 0000000000000000 R11: 0000000000000000 R12: 
>>>>> ffffffff82f8d1c0
>>>>> [  285.034400] R13: 0000000000000000 R14: 0000000000000000 R15: 
>>>>> ffff880074c64b50
>>>>> [  285.042718] FS:  00007f919fe2eb40(0000) GS:ffff88007d140000(0000) 
>>>>> knlGS:0000000000000000
>>>>> [  285.051001] CS:  e033 DS: 002b ES: 002b CR0: 0000000080050033
>>>>> [  285.059458] CR2: 0000000000000000 CR3: 0000000002824000 CR4: 
>>>>> 0000000000000660
>>>>> [  285.067813] Call Trace:
>>>>> [  285.075947]  ? task_work_run+0x85/0xa0
>>>>> [  285.084025]  ? exit_to_usermode_loop+0x72/0x80
>>>>> [  285.091980]  ? do_int80_syscall_32+0xfe/0x120
>>>>> [  285.099896]  ? entry_INT80_compat+0x7f/0x90
>>>>> [  285.107688]  ? fpu__drop+0x23/0x40
>>>>> [  285.115362] Code:  Bad RIP value.
>>>>> [  285.123072] RIP:           (null) RSP: ffffc90001497ed8
>>>>> [  285.130714] CR2: 0000000000000000
>>>>> [  285.138219] ---[ end trace 4d3317497f4ba022 ]---
>>>>> [  285.145671] Fixing recursive fault but reboot is needed!
>>>>>
>>>>> After updating xen-unstable to the latest available commit 
>>>>> 185413355fe331cbc926d48568838227234c9a20,
>>>>> the tool doesn't stall anymore but i still get a kernel splat:
>>>>>
>>>>> [  198.594638] ------------[ cut here ]------------
>>>>> [  198.594641] Invalid address limit on user-mode return
>>>>> [  198.594651] WARNING: CPU: 1 PID: 75 at ./include/linux/syscalls.h:236 
>>>>> do_int80_syscall_32+0xe5/0x120
>>>>> [  198.594652] Modules linked in:
>>>>> [  198.594655] CPU: 1 PID: 75 Comm: kworker/1:1 Not tainted 
>>>>> 4.16.0-rc4-20180305-linus-pvhpatches-doflr+ #1
>>>>> [  198.594656] Hardware name: MSI MS-7640/890FXA-GD70 (MS-7640)  , BIOS 
>>>>> V1.8B1 09/13/2010
>>>>> [  198.594658] Workqueue: events free_work
>>>>> [  198.594660] RIP: e030:do_int80_syscall_32+0xe5/0x120
>>>>> [  198.594661] RSP: e02b:ffffc90000b8ff40 EFLAGS: 00010086
>>>>> [  198.594662] RAX: 0000000000000029 RBX: ffffc90000b8ff58 RCX: 
>>>>> ffffffff82868e38
>>>>> [  198.594663] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 
>>>>> 0000000000000001
>>>>> [  198.594664] RBP: ffff880078623980 R08: 0000000000000dfa R09: 
>>>>> 000000000000063b
>>>>> [  198.594664] R10: 0000000000000000 R11: 000000000000063b R12: 
>>>>> 0000000000000000
>>>>> [  198.594665] R13: 0000000000000000 R14: 0000000000000000 R15: 
>>>>> 0000000000000000
>>>>> [  198.594672] FS:  00007fa252372b40(0000) GS:ffff88007d040000(0000) 
>>>>> knlGS:0000000000000000
>>>>> [  198.594673] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>> [  198.594674] CR2: 00000000f7f303e4 CR3: 0000000002824000 CR4: 
>>>>> 0000000000000660
>>>>> [  198.594676] Call Trace:
>>>>> [  198.594683]  entry_INT80_compat+0x7f/0x90
>>>>> [  198.594685]  ? vunmap_page_range+0x2a0/0x340
>>>>> [  198.594686] Code: 03 7f 48 8b 75 00 f7 c6 0e 38 00 00 75 2e 83 65 08 
>>>>> f9 5b 5d c3 e8 0c fb ff ff e9 53 ff ff ff 48 c7 c7 58 35 57 82 e8 ab 3e 
>>>>> 0c 00 <0f> 0b bf 09 00 00 00 48 89 ee e8 8c 00 0d 00 eb b8 48 89 df e8 
>>>>> [  198.594706] ---[ end trace 90bcd2147bc825ef ]---
>>>>>
>>>>> After reverting commit f75b1a5247b3b311d3aa50de4c0e5f2d68085cb1 the issue 
>>>>> is gone.
>>>> :(
>>>>
>>>> This will be the issue which OSSTest is probably bisecting to as well. 
>>>> It is quite odd to see a 64bit process using int80 as opposed to syscall.
>>>>
>>>> I'll see about double checking my assembly code, and will also try to
>>>> identify why my unit tests haven't noticed an issue.
>>> As a progress report, this is proving to be terrible bug to debug.
> 
> What triggers it for me reliable is: ldd -v /lib/x86_64-linux-gnu/libc.so.6
> 
> Sometimes it runs ok with as output (but i do get a kernel splat):
> 
>       /lib64/ld-linux-x86-64.so.2 (0x00007f1e1840f000)
>       linux-vdso.so.1 (0x00007ffe5875f000)
> 
>       Version information:
>       /lib/x86_64-linux-gnu/libc.so.6:
>               ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
>               ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => 
> /lib64/ld-linux-x86-64.so.2
> 
> sometimes it stalls/locks.

A 32-bit pv domain crashes reliably with said commit when booting,
without the commit it can boot.

Crash is:

[    1.332526] Write protecting the kernel text: 3404k
[    1.333162] Write protecting the kernel read-only data: 1928k
[    1.333167] NX-protecting the kernel data: 2732k
: : No such file or directory
[    1.335803] Kernel panic - not syncing: Attempted to kill init!
[    1.335811] Pid: 1, comm: init Not tainted 3.0.101-xen #25
[    1.335816] Call Trace:
[    1.335830]  [<c0009011>] try_stack_unwind+0x1a1/0x1b0
[    1.335848]  [<c00077f7>] dump_trace+0x47/0x110
[    1.335868]  [<c0008b9b>] show_trace_log_lvl+0x4b/0x60
[    1.335875]  [<c0008bc8>] show_trace+0x18/0x20
[    1.335884]  [<c03493e8>] dump_stack+0x6d/0x72
[    1.335905]  [<c034947b>] panic+0x57/0x170
[    1.335913]  [<c003880a>] forget_original_parent+0x31a/0x320
[    1.335929]  [<c003881b>] exit_notify+0xb/0x190
[    1.335935]  [<c0038b85>] do_exit+0x1e5/0x3d0
[    1.335945]  [<c0039082>] do_group_exit+0x32/0xc0
[    1.335951]  [<c0039121>] sys_exit_group+0x11/0x20
[    1.335957]  [<c034c89d>] syscall_call+0x7/0x7
[    1.335968]  [<f57fe424>] 0xf57fe423


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel