Re: [Xen-devel] [BUG] panic: "IO-APIC + timer doesn't work" - several people have reproduced

[Jason Andryuk <jandryuk@xxxxxxxxx>]

enabling vecOn Tue, Dec 31, 2019 at 5:43 AM Aaron Janse <aaron@xxxxxxxxx> wrote:
>
> On Tue, Dec 31, 2019, at 12:27 AM, Andrew Cooper wrote:
> > Is there any full boot log in the bad case?  Debugging via divination
> > isn't an effective way to get things done.
>
> Agreed. I included some more verbose logs towards the end of the email (typed 
> up by hand).
>
> Attached are pictures from a slow-motion video of my laptop booting. Note 
> that I also included a picture of a stack trace that happens immediately 
> before reboot. It doesn't look related, but I wanted to include it anyway.
>
> I think the original email should have said "4.8.5" instead of "4.0.5." 
> Regardless, everyone on this mailing list can now see all the boot logs that 
> I've seen.
>
> Attaching a serial console seems like it would be difficult to do on this 
> laptop, otherwise I would have sent the logs as a txt file.

I'm seeing Xen panic: "IO-APIC + timer doesn't work" on a Dell
Latitude 7200 2-in-1.  Fedora 31 Live USB image boots successfully.
No way to get serial output.  I manually recreated the output before
from the vga display.

Comparing the Linux and Xen, Xen does:
(XEN) I/O Virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) nr_sockets: 1
(XEN) Getting VERSION: 1060015
(XEN) Getting VERSION: 1060015
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) Getting ID: 0
(XEN) Getting LVT0: 700
(XEN) Getting LVT1: 400
(XEN) Suppress EOI broadcast on CPU#0
(XEN) enabled ExtINT on CPU#0
(XEN) ESR value before enabling vector: 0x40 after: 0
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) init IO_APIC IRQs
(XEN)  IO-APIC (apicid-pin) 2-0, 2-16, 2-17, ...<snip>... 2-119 not connected.
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) ..MP-BIOS bug: 8254 timer not connected to IO-APIC
(XEN) ...trying to set up timer (IRQ0) through 8259A ... failed
(XEN) ...trying to set up timer as Virtual Wire IRQ... failed.
(XEN) ...trying to set up timer as ExtINT IRQ...spurious 8259A interrupt: IRQ7.
(XEN) CPU0: no irq handler for vector e7 (IRQ -8)
(XEN) IRQ7 a=0001[0001,0000] v=60[ffffffff] t=IO-APIC-edge s=00000002
(XEN)  failed :(.

while linux apic=debug does:
kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
kernel: clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles:
0x1e44fb6c2ab, max_idle_ns: 440795206594 ns
kernel: Calibrating delay loop (skipped), value calculated using timer
frequency.. 4199.88 BogoMIPS (lpj=2099944)
.. and continues onward

Since linux doesn't print "...trying to set up timer (IRQ0) through
the 8259A ..." that seems to indicate Linux is seeing the timer
interrupt properly.
https://elixir.bootlin.com/linux/v5.3.7/source/arch/x86/kernel/apic/io_apic.c#L2198

I tested Linux with intel_iommu=on and that booted successfully.
Under Xen, this system sets iommu_x2apic_enabled = true, so
force_iommu is set and iommu=0 cannot disable the iommu.
fails.  Oh, I can disable x2apic and then disable iommu

x2apic=1 -> failure above
x2apic=0 iommu=0 -> failure above
clocksource=acpi -> doesn't help
clocksource=pit -> hangs after "load tracking window length 1073741824 ns"
noapic -> BUG in init_bsp_APIC

One other thing that might be noteworthy.  Linux only prints ACPI IRQ0
and IRQ9 used by override where Xen lists IRQ 0, 2 & 9.

Below is the re-constructed Xen console output.  The SMBIOS line is
the first thing displayed on the VGA output.  I skipped the full EFI
memory map dump since it is quite long.

I've also attached the Linux dmesg output.  Any pointers or
suggestions are most welcome.

Thanks,
Jason

(XEN) SMBIOS 3.2 present.
(XEN) APIC boot stats is `xapic`
(XEN) Using APIC driver default
(XEN) XSM Framework v1.0.0 initialized
(XEN) Flask: 128 avtab hash slots, 283 rules.
(XEN) Flask: 128 avtab hash slots, 283 rules.
(XEN) Flask:  4 users, 3 roles, 38 types, 2 bools
(XEN) Flask:  13 classes, 283 rules
(XEN) Flask:  Starting in enforcing mode.
(XEN) ACPI: PM-Timer IO Port: 0x1808 (32 bits)
(XEN) ACPI: v5 SLEEP INFO: control[1:1804], status[1:1800]
(XEN) ACPI: Invalid sleep control/status register data: 0:0x8:0x3 0:0x8:0x3
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:1804,1:0], pm1x_evt[1:1800,1:0]
(XEN) ACPI: 32/64X FACS address mismatch in FADT -
38c80c00/0000000000000000, using 32
(XEN) ACPI:             wakeup_vec[38c80c0c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-119
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a201 base: 0xfed00000
(XEN) PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff
(XEN) PCI: MCFG area at e0000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-ff
(XEN) [VT-D]dmar.c:563:  Non-existent device (0000:00:16.7) is
reported in RMRR (386fa000, 38779fff)'s scope!
(XEN) [VT-D]dmar.c:579:   Ignore the RMRR (386fa000, 38779fff) due to
device under its scope are not PCI discoverable
(XEN) ACPI: BGRT: invalidating v1 image at 0x3329c018
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 8 CPUs (0 hotplug CPUs)
(XEN) mapped APIC to ffff82cfffffb000 (fee00000)
(XEN) mapped IOAPIC to ffff82cfffffa000 (fec00000)
(XEN) IRQ limits: 120 GSI, 1544 MSI/MSI-X
(XEN) Switched to APIC driver x2apic_cluster
(XEN) xstate: size: 0x440 and states: 0x1f
(XEN) mce_intel.c:773: MCA Capability: firstbank 0, extended MCE MSR
0, BCAST, CMCI
(XEN) CPU0: Intel machine check reporting enabled
(XEN) Speculative mitigation facilities:
(XEN)   Hardware features: IBRS/IBPB STIBP L1D_FLUSH SSBD MD_CLEAR
IBRS_ALL RDCL_NO SKIP_L1DFL MDS_NO
(XEN)   Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN)   Xen settings: BTI-Thunk JMP, SPEC_CTRL: IBRS+ SSBD-, Other:
IBPB BRANCH_HARDEN
(XEN)   L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 39, Safe
address 8000000000
(XEN)   Support for HVM VMs: MSR_SPEC_CTRL RSB EAGER_FPU MD_CLEAR
(XEN)   Support for PV VMs: MSR_SPEC_CTRL RSB EAGER_FPU MD_CLEAR
(XEN)   XPTI (64-bit PV only): Dom0 disabled, DomU enabled (with PCID)
(XEN)   PV L1TF shadowing: Dom0 disabled, DomU disabled
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN)  load_precision_shift: 18
(XEN)  load_window_shift: 30
(XEN)  underload_balance_tolerance: 0
(XEN)  overload_balance_tolerance: -3
(XEN)  runqueues arrangement: socket
(XEN)  cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Adding cpu 0 to runqueue 0
(XEN)  First cpu on runqueue, activating
(XEN) Platform timer is 23.999MHz HPET
(XEN) Detected 2111.997 MHz processor.
(XEN) EFI memory map:
(XEN)  0000000000000-0000000003fff type=2 attr=000000000000000f
(XEN)  0000000004000-000000008dfff type=7 attr=000000000000000f
(XEN)  000000008e000-000000009dfff type=2 attr=000000000000000f
(XEN)  000000009e000-000000009efff type=0 attr=000000000000000f
(XEN)  000000009f000-000000009ffff type=3 attr=000000000000000f
<snip>
(XEN)  0000037d0c000-000038779ffff type=0 attr=000000000000000f
(XEN)  000003877a000-0000387f6ffff type=9 attr=000000000000000f
(XEN)  00000387f7000-000038c81ffff type=10 attr=000000000000000f
<snip>
(XEN)  00000489f4000-00000489fffff type=7 attr=000000000000000f
(XEN)  0000100000000-00004ac7fffff type=7 attr=000000000000000f
(XEN)  00000000a0000-00000000fffff type=0 attr=0000000000000000
(XEN)  0000048a00000-000004f7fffff type=0 attr=0000000000000000
(XEN)  00000e0000000-00000efffffff type=11 attr=800000000000100d
(XEN)  00000fe000000-00000fe010fff type=11 attr=8000000000000001
(XEN)  00000fec00000-00000fec00fff type=11 attr=8000000000000001
(XEN)  00000fed20000-00000fed7ffff type=0 attr=0000000000000000
(XEN)  00000fee00000-00000fee00fff type=11 attr=8000000000000001
(XEN)  00000ff000000-00000ffffffff type=11 attr=800000000000100d
(XEN) alt table ffff82d080483030 -> ffff82d0804910d8
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Posted Interrupt not enabled.
(XEN) Intel VT-d Shared EPT tables enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) nr_sockets: 1
(XEN) Getting VERSION: 1060015
(XEN) Getting VERSION: 1060015
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) Getting ID: 0
(XEN) Getting LVT0: 700
(XEN) Getting LVT1: 400
(XEN) Suppress EOI broadcast on CPU#0
(XEN) enabled ExtINT on CPU#0
(XEN) ESR value before enabling vector: 0x40  after: 0
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) init IO_APIC IRQs
(XEN)  IO-APIC (apicid-pin) 2-0, 2-16, 2-17, ...<snip>... 2-119 not connected.
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) ..MP-BIOS bug: 8254 timer not connected to IO-APIC
(XEN) ...trying to set up timer (IRQ0) through 8259A ... failed.
(XEN) ...trying to set up timer as Virtual Wire IRQ... failed.
(XEN) ...trying to set up timer as ExtINT IRQ...spurious 8259A interrupt: IRQ7.
(XEN) CPU0: no irq handler for vector 27 (IRQ -8)
(XEN) IRQ7 a=ffffffffffffffff[0001,0000] v=68[ffffffff] t=IO-APIC-edge
s=00000002
(XEN)  failed :(.

Attachment: dell-7200-dmesg.txt
Description: Text document

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel