[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] PVH Dom0 Intel IOMMU issues



On Mon, Apr 17, 2017 at 08:47:48AM +0100, Roger Pau Monné wrote:
>On Mon, Apr 17, 2017 at 07:32:45AM +0800, Chao Gao wrote:
>> On Fri, Apr 14, 2017 at 04:34:41PM +0100, Roger Pau Monné wrote:
>> >Hello,
>> >
>> >Although PVHv2 Dom0 is not yet finished, I've been trying the current code 
>> >on
>> >different hardware, and found that with pre-Haswell Intel hardware PVHv2 
>> >Dom0
>> >completely freezes the box when calling iommu_hwdom_init in 
>> >dom0_construct_pvh.
>> >OTOH the same doesn't happen when using a newer CPU (ie: haswell or newer).
>> >
>> >I'm not able to debug that in any meaningful way because the box seems to 
>> >lock
>> >up completely, even the watchdog NMI stops working. Here is the boot log, 
>> >up to
>> >the point where it freezes:
>> 
>> I try "dom0=pvh" with my skylake. An assertion failed. Is it a software bug?
>> 
>---8<---
>diff --git a/xen/arch/x86/hvm/vioapic.c b/xen/arch/x86/hvm/vioapic.c
>index 527ac2aadd..1df7710041 100644
>--- a/xen/arch/x86/hvm/vioapic.c
>+++ b/xen/arch/x86/hvm/vioapic.c
>@@ -625,6 +625,9 @@ int vioapic_init(struct domain *d)
>         nr_gsis += nr_pins;
>     }
> 
>+    printk("domain nr_gsis: %u vioapic gsis: %u nr_irqs_gsi: %u highest_gsi: 
>%u\n",
>+           hvm_domain_irq(d)->nr_gsis, nr_gsis, nr_irqs_gsi, highest_gsi());
>+
>     ASSERT(hvm_domain_irq(d)->nr_gsis == nr_gsis);
> 
>     d->arch.hvm_domain.nr_vioapics = nr_vioapics;

With the above patch,
(XEN) [   10.420001] PCI: MCFG area at 80000000 reserved in E820
(XEN) [   10.426854] PCI: Using MCFG for segment 0000 bus 00-ff
(XEN) [   10.433952] Intel VT-d iommu 6 supported page sizes: 4kB, 2MB, 1GB.
(XEN) [   10.441856] Intel VT-d iommu 5 supported page sizes: 4kB, 2MB, 1GB.
(XEN) [   10.449759] Intel VT-d iommu 4 supported page sizes: 4kB, 2MB, 1GB.
(XEN) [   10.457671] Intel VT-d iommu 3 supported page sizes: 4kB, 2MB, 1GB.
(XEN) [   10.465585] Intel VT-d iommu 2 supported page sizes: 4kB, 2MB, 1GB.
(XEN) [   10.473485] Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
(XEN) [   10.481394] Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
(XEN) [   10.489299] Intel VT-d iommu 7 supported page sizes: 4kB, 2MB, 1GB.
(XEN) [   10.497196] Intel VT-d Snoop Control enabled.
(XEN) [   10.503196] Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) [   10.510145] Intel VT-d Queued Invalidation enabled.
(XEN) [   10.516646] Intel VT-d Interrupt Remapping enabled.
(XEN) [   10.523173] Intel VT-d Posted Interrupt not enabled.
(XEN) [   10.529775] Intel VT-d Shared EPT tables enabled.
(XEN) [   10.548815] I/O virtualisation enabled
(XEN) [   10.554186]  - Dom0 mode: Relaxed
(XEN) [   10.559264] Interrupt remapping enabled
(XEN) [   10.564854] nr_sockets: 5
(XEN) [   10.569231] Enabled directed EOI with ioapic_ack_old on!
(XEN) [   10.577294] ENABLING IO-APIC IRQs
(XEN) [   10.582245]  -> Using old ACK method
(XEN) [   10.587967] ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) [   10.797645] TSC deadline timer enabled
(XEN) [   10.887286] Defaulting to alternative key handling; send 'A' to switch 
to normal mode.
(XEN) [   10.897864] mwait-idle: MWAIT substates: 0x2020
(XEN) [   10.899335] mwait-idle: v0.4.1 model 0x55
(XEN) [   10.900799] mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) [   10.902304] VMX: Supported advanced features:
(XEN) [   10.903781]  - APIC MMIO access virtualisation
(XEN) [   10.905258]  - APIC TPR shadow
(XEN) [   10.907138]  - Extended Page Tables (EPT)
(XEN) [   10.908782]  - Virtual-Processor Identifiers (VPID)
(XEN) [   10.910262]  - Virtual NMI
(XEN) [   10.911719]  - MSR direct-access bitmap
(XEN) [   10.913188]  - Unrestricted Guest
(XEN) [   10.914650]  - APIC Register Virtualization
(XEN) [   10.916126]  - Virtual Interrupt Delivery
(XEN) [   10.917596]  - Posted Interrupt Processing
(XEN) [   10.919066]  - VMCS shadowing
(XEN) [   10.920519]  - VM Functions
(XEN) [   10.921976]  - Virtualisation Exceptions
(XEN) [   10.923448]  - Page Modification Logging
(XEN) [   10.924918]  - TSC Scaling
(XEN) [   10.926371] HVM: ASIDs enabled.
(XEN) [   10.927829] HVM: VMX enabled
(XEN) [   10.929278] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [   10.930762] HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) [    0.000000] CMCI: threshold 0x2 too large for CPU56 bank 6, using 0x1
(XEN) [    0.000000] CMCI: threshold 0x2 too large for CPU56 bank 9, using 0x1
(XEN) [    0.000000] CMCI: threshold 0x2 too large for CPU56 bank 10, using 0x1
(XEN) [    0.000000] CMCI: threshold 0x2 too large for CPU56 bank 11, using 0x1
(XEN) [   13.216648] Brought up 112 CPUs
(XEN) [   13.739330] build-id: dc4540250abe5d96614d340c67069e390c37c21c
(XEN) [   13.740816] Running stub recovery selftests...
(XEN) [   13.742258] traps.c:3466: GPF (0000): ffff82d0bffff041 
[ffff82d0bffff041] -> ffff82d080359cf2
(XEN) [   13.745155] traps.c:813: Trap 12: ffff82d0bffff040 [ffff82d0bffff040] 
-> ffff82d080359cf2
(XEN) [   13.748046] traps.c:1215: Trap 3: ffff82d0bffff041 [ffff82d0bffff041] 
-> ffff82d080359cf2
(XEN) [   13.771452] TSC warp detected, disabling TSC_RELIABLE
(XEN) [   13.772897] ACPI sleep modes: S3
(XEN) [   13.774329] VPMU: disabled
(XEN) [   13.775757] mcheck_poll: Machine check polling timer started.
(XEN) [   13.777246] Dom0 has maximum 1448 PIRQs
(XEN) [   13.779426] domain nr_gsis: 104 vioapic gsis: 88 nr_irqs_gsi: 104 
highest_gsi: 103
(XEN) [   13.782371] Assertion 'hvm_domain_irq(d)->nr_gsis == nr_gsis' failed 
at vioapic.c:603
(XEN) [   13.785249] ----[ Xen-4.9-unstable  x86_64  debug=y   Not tainted ]----
(XEN) [   13.786715] CPU:    0
(XEN) [   13.788137] RIP:    e008:[<ffff82d0802ef764>] vioapic_init+0x13f/0x196
(XEN) [   13.789605] RFLAGS: 0000000000010287   CONTEXT: hypervisor
(XEN) [   13.791065] rax: ffff830837c7ea00   rbx: 0000000000000009   rcx: 
0000000000000000
(XEN) [   13.793948] rdx: ffff82d080457fff   rsi: 000000000000000a   rdi: 
ffff82d08044d6b8
(XEN) [   13.796819] rbp: ffff82d080457d28   rsp: ffff82d080457ce8   r8:  
ffff83083e000000
(XEN) [   13.799690] r9:  0000000000000006   r10: 00000000000be4a2   r11: 
0000000000000006
(XEN) [   13.802565] r12: 0000000000000008   r13: ffff830837d2e000   r14: 
0000000000000058
(XEN) [   13.805440] r15: ffff830837c7eb20   cr0: 000000008005003b   cr4: 
00000000003526e0
(XEN) [   13.808316] cr3: 000000006f84c000   cr2: 0000000000000000
(XEN) [   13.809768] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 
e008
(XEN) [   13.812648] Xen code around <ffff82d0802ef764> 
(vioapic_init+0x13f/0x196):
(XEN) [   13.815516]  00 00 44 3b 70 40 74 02 <0f> 0b 8b 45 cc 41 89 85 b0 02 
00 00 4c 89 ef e8
(XEN) [   13.818458] Xen stack trace from rsp=ffff82d080457ce8:
(XEN) [   13.819902]    ffff82d08029e7de 0000000937c7e010 ffff82d080457d08 
ffff830837d2e000
(XEN) [   13.822785]    0000000000000068 0000000000000001 0000000000000000 
0000000000000000
(XEN) [   13.825660]    ffff82d080457d48 ffff82d0802de276 ffff830837d2e000 
0000000000000000
(XEN) [   13.828535]    ffff82d080457d78 ffff82d08026d593 ffff82d080457d78 
ffff830837d2e000
(XEN) [   13.831410]    000000000000001f 0000000000000007 ffff82d080457de8 
ffff82d080205226
(XEN) [   13.834286]    ffff82d0804380e0 0000000000000004 ffff82d080457eb4 
0000000000000000
(XEN) [   13.837164]    ffff82d080457dc8 f7fa32231fcbfbff 000001212c100800 
00000000000000e0
(XEN) [   13.840038]    ffff830838543850 00000000000000e0 ffff82d08043b780 
000000000000006f
(XEN) [   13.842917]    ffff82d080457f08 ffff82d0803ee1be 000000000028fe80 
000000000000015c
(XEN) [   13.845795]    00000000000001df 0000000200000000 0000000000000002 
0000000000000002
(XEN) [   13.848669]    0000000000000002 0000000000000001 0000000000000001 
0000000000000001
(XEN) [   13.851544]    0000000000000001 0000000000000000 ffff82d080429a90 
0000000000000017
(XEN) [   13.854486]    0000001075ec7000 00000000013b7000 0000000001080000 
0000000000000000
(XEN) [   13.857356]    ffff830000000002 ffff830000097c60 ffff830000097ee0 
ffff830000097fb0
(XEN) [   13.860226]    0000000000000000 0000001100000000 0000000800000000 
000000010000006e
(XEN) [   13.863101]    0000000000000003 00000000000002f8 0000000000000000 
0000000000000000
(XEN) [   13.865975]    00000000000000e8 0000000068924a50 000000006c5b8a28 
000000006c5b8a40
(XEN) [   13.868847]    0000000000000000 ffff82d0802000f3 0000000000000000 
0000000000000000
(XEN) [   13.871723]    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN) [   13.874600]    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN) [   13.877472] Xen call trace:
(XEN) [   13.878892]    [<ffff82d0802ef764>] vioapic_init+0x13f/0x196
(XEN) [   13.880353]    [<ffff82d0802de276>] hvm_domain_initialise+0x268/0x365
(XEN) [   13.881825]    [<ffff82d08026d593>] arch_domain_create+0x4b0/0x6ac
(XEN) [   13.883294]    [<ffff82d080205226>] domain_create+0x3d8/0x593
(XEN) [   13.884759]    [<ffff82d0803ee1be>] __start_xen+0x20fa/0x24c3
(XEN) [   13.886221]    [<ffff82d0802000f3>] __high_start+0x53/0x58
(XEN) [   13.887681] 
(XEN) [   14.148367] 
(XEN) [   14.151511] ****************************************
(XEN) [   14.158124] Panic on CPU 0:
(XEN) [   14.162565] Assertion 'hvm_domain_irq(d)->nr_gsis == nr_gsis' failed 
at vioapic.c:603
(XEN) [   14.173326] ****************************************
(XEN) [   14.179947] 
(XEN) [   14.183083] Reboot in five seconds...
(XEN) [   19.190218] Resetting with ACPI MEMORY or I/O RESET_REG.

Thanks
Chao

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.