Re: [Xen-devel] Regression with commit "x86/pv: Drop int80_bounce from struct pv_vcpu" f75b1a5247b3b311d3aa50de4c0e5f2d68085cb1

[Boris Ostrovsky <boris.ostrovsky@xxxxxxxxxx>]

On 03/12/2018 03:05 PM, Andrew Cooper wrote:
> On 10/03/18 16:27, Andrew Cooper wrote:
>> On 10/03/2018 16:14, Sander Eikelenboom wrote:
>>> Hi Andrew,
>>>
>>> It seems commit "x86/pv: Drop int80_bounce from struct pv_vcpu" 
>>> (f75b1a5247b3b311d3aa50de4c0e5f2d68085cb1) causes an issue on my machine, 
>>> an AMD phenom X6.
>>>
>>> When trying to installing a new kernel package which runs the Debian
>>> update-initramfs tools with xen-unstable which happened to be at commit 
>>> c9bd8a73656d7435b1055ee8825823aee995993e as last commit the tool stalls
>>> and i get this kernel splat:
>>>
>>> [  284.910674] BUG: unable to handle kernel NULL pointer dereference at 
>>> 0000000000000000
>>> [  284.919696] IP:           (null)
>>> [  284.928315] PGD 0 P4D 0 
>>> [  284.943343] Oops: 0010 [#1] SMP NOPTI
>>> [  284.957008] Modules linked in:
>>> [  284.965521] CPU: 5 PID: 24729 Comm: ld-linux.so.2 Not tainted 
>>> 4.16.0-rc4-20180305-linus-pvhpatches-doflr+ #1
>>> [  284.974154] Hardware name: MSI MS-7640/890FXA-GD70 (MS-7640)  , BIOS 
>>> V1.8B1 09/13/2010
>>> [  284.983198] RIP: e030:          (null)
>>> [  284.992006] RSP: e02b:ffffc90001497ed8 EFLAGS: 00010286
>>> [  285.000612] RAX: 0000000000000000 RBX: ffff880074c64500 RCX: 
>>> ffffffff82f8d1c0
>>> [  285.009122] RDX: ffffffff82f8d1c0 RSI: 0000000020020002 RDI: 
>>> ffffffff82f8d1c0
>>> [  285.017598] RBP: ffff880074c64b7c R08: 0000000000000000 R09: 
>>> 0000000000000000
>>> [  285.025999] R10: 0000000000000000 R11: 0000000000000000 R12: 
>>> ffffffff82f8d1c0
>>> [  285.034400] R13: 0000000000000000 R14: 0000000000000000 R15: 
>>> ffff880074c64b50
>>> [  285.042718] FS:  00007f919fe2eb40(0000) GS:ffff88007d140000(0000) 
>>> knlGS:0000000000000000
>>> [  285.051001] CS:  e033 DS: 002b ES: 002b CR0: 0000000080050033
>>> [  285.059458] CR2: 0000000000000000 CR3: 0000000002824000 CR4: 
>>> 0000000000000660
>>> [  285.067813] Call Trace:
>>> [  285.075947]  ? task_work_run+0x85/0xa0
>>> [  285.084025]  ? exit_to_usermode_loop+0x72/0x80
>>> [  285.091980]  ? do_int80_syscall_32+0xfe/0x120
>>> [  285.099896]  ? entry_INT80_compat+0x7f/0x90
>>> [  285.107688]  ? fpu__drop+0x23/0x40
>>> [  285.115362] Code:  Bad RIP value.
>>> [  285.123072] RIP:           (null) RSP: ffffc90001497ed8
>>> [  285.130714] CR2: 0000000000000000
>>> [  285.138219] ---[ end trace 4d3317497f4ba022 ]---
>>> [  285.145671] Fixing recursive fault but reboot is needed!
>>>
>>> After updating xen-unstable to the latest available commit 
>>> 185413355fe331cbc926d48568838227234c9a20,
>>> the tool doesn't stall anymore but i still get a kernel splat:
>>>
>>> [  198.594638] ------------[ cut here ]------------
>>> [  198.594641] Invalid address limit on user-mode return
>>> [  198.594651] WARNING: CPU: 1 PID: 75 at ./include/linux/syscalls.h:236 
>>> do_int80_syscall_32+0xe5/0x120
>>> [  198.594652] Modules linked in:
>>> [  198.594655] CPU: 1 PID: 75 Comm: kworker/1:1 Not tainted 
>>> 4.16.0-rc4-20180305-linus-pvhpatches-doflr+ #1
>>> [  198.594656] Hardware name: MSI MS-7640/890FXA-GD70 (MS-7640)  , BIOS 
>>> V1.8B1 09/13/2010
>>> [  198.594658] Workqueue: events free_work
>>> [  198.594660] RIP: e030:do_int80_syscall_32+0xe5/0x120
>>> [  198.594661] RSP: e02b:ffffc90000b8ff40 EFLAGS: 00010086
>>> [  198.594662] RAX: 0000000000000029 RBX: ffffc90000b8ff58 RCX: 
>>> ffffffff82868e38
>>> [  198.594663] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 
>>> 0000000000000001
>>> [  198.594664] RBP: ffff880078623980 R08: 0000000000000dfa R09: 
>>> 000000000000063b
>>> [  198.594664] R10: 0000000000000000 R11: 000000000000063b R12: 
>>> 0000000000000000
>>> [  198.594665] R13: 0000000000000000 R14: 0000000000000000 R15: 
>>> 0000000000000000
>>> [  198.594672] FS:  00007fa252372b40(0000) GS:ffff88007d040000(0000) 
>>> knlGS:0000000000000000
>>> [  198.594673] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [  198.594674] CR2: 00000000f7f303e4 CR3: 0000000002824000 CR4: 
>>> 0000000000000660
>>> [  198.594676] Call Trace:
>>> [  198.594683]  entry_INT80_compat+0x7f/0x90
>>> [  198.594685]  ? vunmap_page_range+0x2a0/0x340
>>> [  198.594686] Code: 03 7f 48 8b 75 00 f7 c6 0e 38 00 00 75 2e 83 65 08 f9 
>>> 5b 5d c3 e8 0c fb ff ff e9 53 ff ff ff 48 c7 c7 58 35 57 82 e8 ab 3e 0c 00 
>>> <0f> 0b bf 09 00 00 00 48 89 ee e8 8c 00 0d 00 eb b8 48 89 df e8 
>>> [  198.594706] ---[ end trace 90bcd2147bc825ef ]---
>>>
>>> After reverting commit f75b1a5247b3b311d3aa50de4c0e5f2d68085cb1 the issue 
>>> is gone.
>> :(
>>
>> This will be the issue which OSSTest is probably bisecting to as well. 
>> It is quite odd to see a 64bit process using int80 as opposed to syscall.
>>
>> I'll see about double checking my assembly code, and will also try to
>> identify why my unit tests haven't noticed an issue.
> As a progress report, this is proving to be terrible bug to debug.
>
> I've confirmed your findings.  However, my repro takes 10 minutes, and
> I've failed to make it any faster.  It is more complicated than just
> using 32bit userspace in a 64bit VM, and putting debugging in the
> hypervisor makes the problem go away.


I've bisected my failures yesterday to this commit as well. I fail right
away during boot, a couple of different kernels. Haven't had a chance to
look at this yet (and probably won't be able to at least for the next
couple of days). But if you need something quick I can do that.

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel