|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] Xen-unstable 4.8: Host crash when shutting down guest with pci device passed through using MSI-X interrupts.
Monday, July 18, 2016, 7:48:20 PM, you wrote:
> On 18/07/16 11:21, linux@xxxxxxxxxxxxxx wrote:
>> Hi Jan,
>>
>> It seems that since your patch series starting with commit:
>> 2016-06-22 x86/vMSI-X: defer intercept handler registration
>> 74c6dc2d0ac4dcab0c6243cdf6ed550c1532b798
>>
>> The shutdown of a guest which has a PCI device passed through which
>> uses MSI-X interrupts causes
>> a host crash, see the splat below. Somehow it also doesn't reboot in 5
>> seconds as it is supposed to (i don't have no-reboot on the command
>> line).
>>
>> --
>> Sander
>>
>>
>> (XEN) [2016-07-16 16:03:17.069] ----[ Xen-4.8-unstable x86_64
>> debug=y Not tainted ]----
>> (XEN) [2016-07-16 16:03:17.069] CPU: 0
>> (XEN) [2016-07-16 16:03:17.069] RIP: e008:[<ffff82d0801e39de>]
>> msixtbl_pt_unregister+0x7b/0xd9
>> (XEN) [2016-07-16 16:03:17.069] RFLAGS: 0000000000010082 CONTEXT:
>> hypervisor (d0v0)
>> (XEN) [2016-07-16 16:03:17.069] rax: ffff83055c678e40 rbx:
>> ffff83055c685500 rcx: 0000000000000001
>> (XEN) [2016-07-16 16:03:17.069] rdx: 0000000000000000 rsi:
>> 0000000000001ab0 rdi: ffff8305313b85a0
>> (XEN) [2016-07-16 16:03:17.069] rbp: ffff83009fd07c78 rsp:
>> ffff83009fd07c68 r8: ffff8305356dfff0
>> (XEN) [2016-07-16 16:03:17.069] r9: ffff8305356df480 r10:
>> ffff830503420c50 r11: 0000000000000282
>> (XEN) [2016-07-16 16:03:17.069] r12: ffff8305313b8000 r13:
>> ffff83009fd07e48 r14: ffff8305313b8000
>> (XEN) [2016-07-16 16:03:17.069] r15: ffff8305356df4a8 cr0:
>> 0000000080050033 cr4: 00000000000006e0
>> (XEN) [2016-07-16 16:03:17.069] cr3: 000000053639f000 cr2:
>> 0000000000000000
>> (XEN) [2016-07-16 16:03:17.069] ds: 0000 es: 0000 fs: 0000 gs:
>> 0000 ss: e010 cs: e008
>> (XEN) [2016-07-16 16:03:17.069] Xen code around <ffff82d0801e39de>
>> (msixtbl_pt_unregister+0x7b/0xd9):
>> (XEN) [2016-07-16 16:03:17.069] 39 42 18 74 19 48 89 ca <48> 8b 0a 0f
>> 18 09 48 39 fa 75 ec 48 8d 7b 24 e8
>> (XEN) [2016-07-16 16:03:17.069] Xen stack trace from
>> rsp=ffff83009fd07c68:
>> (XEN) [2016-07-16 16:03:17.069] 0000000000000000 ffff8305356df480
>> ffff83009fd07ce8 ffff82d08014c394
>> (XEN) [2016-07-16 16:03:17.069] 0000000000000001 ffff8305356df480
>> 0000000000000293 ffff8305313b80cc
>> (XEN) [2016-07-16 16:03:17.069] 000000568012ffe5 ffff8305313b8000
>> ffff83009fd07cd8 ffff83009fd07e38
>> (XEN) [2016-07-16 16:03:17.070] 0000000000000000 ffff83054e5fc000
>> 00007fc25a33e004 ffff8305313b8000
>> (XEN) [2016-07-16 16:03:17.070] ffff83009fd07da8 ffff82d0801629c8
>> 0000000000000000 ffff83053b1191f0
>> (XEN) [2016-07-16 16:03:17.070] 0000000000000246 ffff83009fd07d28
>> ffff82d0801300ae 000000000000000e
>> (XEN) [2016-07-16 16:03:17.070] ffff83009fd07d78 ffff82d080171497
>> ffff83009fd07d78 000000020001d17b
>> (XEN) [2016-07-16 16:03:17.070] ffff83009fd07d68 0000000000000000
>> ffff83009fd07d68 ffff82d080130280
>> (XEN) [2016-07-16 16:03:17.070] ffff83009fd07d78 ffff82d08014d0aa
>> 0000000000000202 0000000000000000
>> (XEN) [2016-07-16 16:03:17.070] ffff8305313b8000 ffff88005716d320
>> 0000000000305000 00007fc25a33e004
>> (XEN) [2016-07-16 16:03:17.070] ffff83009fd07ef8 ffff82d080104b2c
>> 0000000000000206 0000000000000002
>> (XEN) [2016-07-16 16:03:17.070] ffff83009fd07df8 ffff82d08018c9db
>> 0000000000000cfe 0000000000000002
>> (XEN) [2016-07-16 16:03:17.070] 0000000000000002 ffff83054e5fc000
>> ffff83009fd07e48 ffff82d08019c119
>> (XEN) [2016-07-16 16:03:17.070] ffff83009fd07e38 0000000080121177
>> ffff83009fd07e38 0000000000000cfe
>> (XEN) [2016-07-16 16:03:17.070] ffff83009fd07f18 0000000000000206
>> 0000000c00000030 000056082bb90013
>> (XEN) [2016-07-16 16:03:17.070] 0000000200000056 00007fc200000013
>> 0000305600000000 000056082b87465d
>> (XEN) [2016-07-16 16:03:17.070] 00007ffe268206e0 00007fc25606b31f
>> 0000000000000000 000056082b8746cf
>> (XEN) [2016-07-16 16:03:17.070] 0000000000001000 fee5600026820730
>> 00007ffe26820740 000056082b8797be
>> (XEN) [2016-07-16 16:03:17.070] 00000000fee56000 0000430026820772
>> 00007ffe26820740 0000000000003056
>> (XEN) [2016-07-16 16:03:17.070] 00007ffe268206e0 ffff83009ff8a000
>> 00007ffe26820580 ffff88005716d320
>> (XEN) [2016-07-16 16:03:17.070] Xen call trace:
>> (XEN) [2016-07-16 16:03:17.070] [<ffff82d0801e39de>]
>> msixtbl_pt_unregister+0x7b/0xd9
>> (XEN) [2016-07-16 16:03:17.070] [<ffff82d08014c394>]
>> pt_irq_destroy_bind+0x2be/0x3f0
>> (XEN) [2016-07-16 16:03:17.070] [<ffff82d0801629c8>]
>> arch_do_domctl+0xc77/0x2414
>> (XEN) [2016-07-16 16:03:17.070] [<ffff82d080104b2c>]
>> do_domctl+0x19db/0x1d26
>> (XEN) [2016-07-16 16:03:17.070] [<ffff82d0802426bd>]
>> lstar_enter+0xdd/0x137
>> (XEN) [2016-07-16 16:03:17.070]
>> (XEN) [2016-07-16 16:03:17.070] Pagetable walk from 0000000000000000:
>> (XEN) [2016-07-16 16:03:17.070] L4[0x000] = 0000000000000000
>> ffffffffffffffff
>> (XEN) [2016-07-16 16:03:18.147]
>> (XEN) [2016-07-16 16:03:18.155] ****************************************
>> (XEN) [2016-07-16 16:03:18.175] Panic on CPU 0:
>> (XEN) [2016-07-16 16:03:18.187] FATAL PAGE FAULT
>> (XEN) [2016-07-16 16:03:18.200] [error_code=0000]
>> (XEN) [2016-07-16 16:03:18.214] Faulting linear address: 0000000000000000
>> (XEN) [2016-07-16 16:03:18.233] ****************************************
>> (XEN) [2016-07-16 16:03:18.252]
>> (XEN) [2016-07-16 16:03:18.261] Reboot in five seconds...
>>
> Can you paste the disassembly of msixtbl_pt_unregister() please? That
> is a dereference of %rdx which is NULL at this point, but I need to
> figure out which pointer it is supposed to be.
Hi Andrew,
# addr2line -e xen-syms ffff82d0801e3e7e
/usr/src/new/xen-unstable/xen/arch/x86/hvm/vmsi.c:535 (discriminator 1)
So the RIP points to:
void msixtbl_pt_unregister(struct domain *d, struct pirq *pirq)
{
struct irq_desc *irq_desc;
struct msi_desc *msi_desc;
struct pci_dev *pdev;
struct msixtbl_entry *entry;
ASSERT(pcidevs_locked());
ASSERT(spin_is_locked(&d->event_lock));
if ( !has_vlapic(d) )
return;
irq_desc = pirq_spin_lock_irq_desc(pirq, NULL);
if ( !irq_desc )
return;
msi_desc = irq_desc->msi_desc;
if ( !msi_desc )
goto out;
pdev = msi_desc->dev;
list_for_each_entry( entry, &d->arch.hvm_domain.msixtbl_list, list ) <---
HERE
if ( pdev == entry->pdev )
goto found;
out:
spin_unlock_irq(&irq_desc->lock);
return;
found:
if ( !atomic_dec_and_test(&entry->refcnt) )
del_msixtbl_entry(entry);
spin_unlock_irq(&irq_desc->lock);
}
Disassembly:
(gdb) info line msixtbl_pt_unregister
Line 513 of "vmsi.c" starts at address 0xffff82d0801e3e03
<msixtbl_pt_unregister> and ends at 0xffff82d0801e3e10
<msixtbl_pt_unregister+13>.
(gdb) disas 0xffff82d0801e3e03
Dump of assembler code for function msixtbl_pt_unregister:
0xffff82d0801e3e03 <+0>: push %rbp
0xffff82d0801e3e04 <+1>: mov %rsp,%rbp
0xffff82d0801e3e07 <+4>: push %r12
0xffff82d0801e3e09 <+6>: push %rbx
0xffff82d0801e3e0a <+7>: mov %rdi,%r12
0xffff82d0801e3e0d <+10>: mov %rsi,%rbx
0xffff82d0801e3e10 <+13>: callq 0xffff82d08014d585 <pcidevs_locked>
0xffff82d0801e3e15 <+18>: test %al,%al
0xffff82d0801e3e17 <+20>: jne 0xffff82d0801e3e1b
<msixtbl_pt_unregister+24>
0xffff82d0801e3e19 <+22>: ud2
0xffff82d0801e3e1b <+24>: lea 0xcc(%r12),%rdi
0xffff82d0801e3e23 <+32>: callq 0xffff82d080130544 <_spin_is_locked>
0xffff82d0801e3e28 <+37>: test %eax,%eax
0xffff82d0801e3e2a <+39>: jne 0xffff82d0801e3e2e
<msixtbl_pt_unregister+43>
0xffff82d0801e3e2c <+41>: ud2
0xffff82d0801e3e2e <+43>: testb $0x1,0x9dc(%r12)
0xffff82d0801e3e37 <+52>: je 0xffff82d0801e3ed7
<msixtbl_pt_unregister+212>
0xffff82d0801e3e3d <+58>: mov $0x0,%esi
0xffff82d0801e3e42 <+63>: mov %rbx,%rdi
0xffff82d0801e3e45 <+66>: callq 0xffff82d0801743a4
<pirq_spin_lock_irq_desc>
0xffff82d0801e3e4a <+71>: mov %rax,%rbx
0xffff82d0801e3e4d <+74>: test %rax,%rax
0xffff82d0801e3e50 <+77>: je 0xffff82d0801e3ed7
<msixtbl_pt_unregister+212>
0xffff82d0801e3e56 <+83>: mov 0x10(%rax),%rax
0xffff82d0801e3e5a <+87>: test %rax,%rax
0xffff82d0801e3e5d <+90>: je 0xffff82d0801e3e89
<msixtbl_pt_unregister+134>
0xffff82d0801e3e5f <+92>: mov 0x20(%rax),%rax
0xffff82d0801e3e63 <+96>: mov 0x5a0(%r12),%rdx
0xffff82d0801e3e6b <+104>: lea 0x5a0(%r12),%rdi
0xffff82d0801e3e73 <+112>: jmp 0xffff82d0801e3e7e
<msixtbl_pt_unregister+123>
0xffff82d0801e3e75 <+114>: cmp %rax,0x18(%rdx)
0xffff82d0801e3e79 <+118>: je 0xffff82d0801e3e94
<msixtbl_pt_unregister+145>
0xffff82d0801e3e7b <+120>: mov %rcx,%rdx
0xffff82d0801e3e7e <+123>: mov (%rdx),%rcx
0xffff82d0801e3e81 <+126>: prefetcht0 (%rcx)
0xffff82d0801e3e84 <+129>: cmp %rdi,%rdx
0xffff82d0801e3e87 <+132>: jne 0xffff82d0801e3e75
<msixtbl_pt_unregister+114>
0xffff82d0801e3e89 <+134>: lea 0x24(%rbx),%rdi
0xffff82d0801e3e8d <+138>: callq 0xffff82d080130514 <_spin_unlock_irq>
0xffff82d0801e3e92 <+143>: jmp 0xffff82d0801e3ed7
<msixtbl_pt_unregister+212>
0xffff82d0801e3e94 <+145>: lock decl 0x10(%rdx)
0xffff82d0801e3e98 <+149>: sete %al
0xffff82d0801e3e9b <+152>: test %al,%al
0xffff82d0801e3e9d <+154>: jne 0xffff82d0801e3ece
<msixtbl_pt_unregister+203>
0xffff82d0801e3e9f <+156>: mov (%rdx),%rcx
0xffff82d0801e3ea2 <+159>: mov 0x8(%rdx),%rax
0xffff82d0801e3ea6 <+163>: mov %rax,0x8(%rcx)
0xffff82d0801e3eaa <+167>: mov %rcx,(%rax)
0xffff82d0801e3ead <+170>: movabs $0x200200200200200,%rax
0xffff82d0801e3eb7 <+180>: mov %rax,0x8(%rdx)
0xffff82d0801e3ebb <+184>: lea 0x158(%rdx),%rdi
0xffff82d0801e3ec2 <+191>: lea -0xac1(%rip),%rsi #
0xffff82d0801e3408 <free_msixtbl_entry>
0xffff82d0801e3ec9 <+198>: callq 0xffff82d080122be0 <call_rcu>
0xffff82d0801e3ece <+203>: lea 0x24(%rbx),%rdi
0xffff82d0801e3ed2 <+207>: callq 0xffff82d080130514 <_spin_unlock_irq>
0xffff82d0801e3ed7 <+212>: pop %rbx
0xffff82d0801e3ed8 <+213>: pop %r12
0xffff82d0801e3eda <+215>: pop %rbp
0xffff82d0801e3edb <+216>: retq
End of assembler dump.
--
Sander
> Thanks,
> ~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |