[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Regression with Xen 4.4 when booting with AMD IOMMU on TA890FXE BIOSTAR

To: Konrad Rzeszutek Wilk <konrad@xxxxxxxxxx>
From: Boris Ostrovsky <boris.ostrovsky@xxxxxxxxxx>
Date: Thu, 19 Dec 2013 16:17:31 -0500
Cc: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, suravee.suthikulpanit@xxxxxxx, jbeulich@xxxxxxxx, xen-devel@xxxxxxxxxxxxxxxxxxxx
Delivery-date: Thu, 19 Dec 2013 21:17:41 +0000
List-id: Xen developer discussion <xen-devel.lists.xen.org>

On 12/19/2013 01:52 PM, Konrad Rzeszutek Wilk wrote:

On Thu, Dec 19, 2013 at 06:38:42PM +0000, Andrew Cooper wrote:

On 19/12/2013 18:25, Konrad Rzeszutek Wilk wrote:

Hey,

With Xen 4.1, 4.2 and 4.3 it boots fine (albeit it complains about the
IOMMU a bit -see attached). With Xen 4.4 it blows up:
(the xen-syms is at http://darnok.org/xen/xen-syms.gz)

This is with commit a0070f7a5ad8652c74c685a0ee5f10215402279d
"x86/memshr: fix preemption in relinquish_shared_pages()" though
I hadn't run Xen 4.4 on this box before so the issue might have
happend much earlier.

You are getting some console corruption, even with sync_console.

Also, can you try iommu=debug apic_verbosity=debug ?

Could this perhaps be because you have manually set up the interrupt for
com1 ?

No.

If I boot with 3726b101a5cfb2614d3fa2d8be29d6e0b4032a6e
("extract register definitions from ns16550 into a separated header")
it boots fine. If I got one commit further forward:
2ca9fbd739b8a72b16dd790d0fff7b75f5488fb8 ("AMD IOMMU: allocate IRTE
entries instead of using a static mapping") it blows up.

I also had booted Xen with just 'console=vga' and saw the same
blow up.

With iommu=debug and on 2ca9fbd739b8a72b16dd790d0fff7b75f5488fb8
I see this:

PXELINUX 3.82 2009-06-09  Copyright (C) 1994-2009 H. Peter Anvin et al
boot:
Loading xen.gz... ok
Loading vmlinuz... ok
Loading initramfs.cpio.gz... ok
  __  __            _  _   _  _                      _        _     _
  \ \/ /___ _ __   | || | | || |     _   _ _ __  ___| |_ __ _| |__ | | ___
   \  // _ \ '_ \  | || |_| || |_ __| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
   /  \  __/ | | | |__   _|__   _|__| |_| | | | \__ \ || (_| | |_) | |  __/
  /_/\_\___|_| |_|    |_|(_) |_|     \__,_|_| |_|___/\__\__,_|_.__/|_|\___|

(XEN) Xen version 4.4-unstable (konrad@(none)) (gcc (GCC) 4.4.4 20100503 (Red Hat 4.4.4-2)) debug=y Thu Dec 19 13:49:43 EST 2013

(XEN) Latest ChangeSet: Tue Jul 16 11:52:38 2013 +0200 git:2ca9fbd
(XEN) Console output is synchronous.
(XEN) Bootloader: unknown
(XEN) Command line: sync_console iommu=debug,verbose console=vga,com1 
com1=115200,8n1,0xD800,0 conring_size=2097152 loglvl=all guest_loglvl=all 
dom0_mem_max=max:6GB,2G
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009b400 (usable)
(XEN)  000000000009b400 - 00000000000a0000 (reserved)
(XEN)  00000000000e2000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000cffb0000 (usable)
(XEN)  00000000cffb0000 - 00000000cffbe000 (ACPI data)
(XEN)  00000000cffbe000 - 00000000cffe0000 (ACPI NVS)
(XEN)  00000000cffe0000 - 00000000d0000000 (reserved)
(XEN)  00000000fff00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000230000000 (usable)
(XEN) ACPI: RSDP 000FB380, 0024 (r2 ACPIAM)
(XEN) ACPI: XSDT CFFB0100, 0064 (r1 071610 XSDT1756 20100716 MSFT       97)
(XEN) ACPI: FACP CFFB0290, 00F4 (r4 071610 FACP1756 20100716 MSFT       97)
(XEN) ACPI: DSDT CFFB0450, 4D0D (r2  89FAD 89FAD716        1 INTL 20051117)
(XEN) ACPI: FACS CFFBE000, 0040
(XEN) ACPI: APIC CFFB0390, 007C (r2 071610 APIC1756 20100716 MSFT       97)
(XEN) ACPI: MCFG CFFB0410, 003C (r1 071610 OEMMCFG  20100716 MSFT       97)
(XEN) ACPI: OEMB CFFBE040, 0072 (r1 071610 OEMB1756 20100716 MSFT       97)
(XEN) ACPI: SRAT CFFBA450, 0108 (r3 AMD    FAM_F_10        2 AMD         1)
(XEN) ACPI: HPET CFFBA560, 0038 (r1 071610 OEMHPET  20100716 MSFT       97)
(XEN) ACPI: IVRS CFFBA5A0, 00B8 (r1  AMD     RD890S   202031 AMD         0)
(XEN) ACPI: SSDT CFFBA660, 0DA4 (r1 A M I  POWERNOW        1 AMD         1)
(XEN) System RAM: 8191MB (8387884kB)
(XEN) SRAT: PXM 0 -> APIC 0 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 1 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 2 -> Node 0
(XEN) SRAT:Allocated memnodemap from 229329000 - 22932c000
(XEN) NUMA: Using 8 for the hash shift.
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000ff780
(XEN) DMI present.
(XEN) Using APIC driver default
(XEN) EN) ACPI:             wakeup_vec[cffbe00c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 0:10 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
(XEN) Processor #1 0:10 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
(XEN) Processor #2 0:10 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
(XEN) Processor #3 0:10 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled)
(XEN) Processor #4 0:10 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x05] enabled)
(XEN) Processor #5 0:10 APIC version 16
(XEN) ACPI: IOAPIC (id[0x06] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 6, version 33, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8300 base: 0xfed00000
(XEN) ERST table was not found
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 6 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 24 GSI, 1144 MSI/MSI-X
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3000.216 MHz processor.
(XEN) Initing memory sharing.
(XEN) AMD Fam10h machine check reporting enabled
(XEN) PI: Not using MCFG for segment 0000 bus 00-ff
(XEN) AMD-Vi: Found MSI capability block at 0x54
(XEN) AMD-Vi: ACPI Table:
(XEN) AMD-Vi:  Signature IVRS
(XEN) AMD-Vi:  Length 0xb8
(XEN) AMD-Vi:  Revision 0x1
(XEN) AMD-Vi:  CheckSum 0x4f
(XEN) AMD-Vi:  OEM_Id AMD
(XEN) AMD-Vi:  OEM_Table_Id RD890S
(XEN) AMD-Vi:  OEM_Revision 0x202031
(XEN) AMD-Vi:  Creator_Id AMD
(XEN) AMD-Vi:  Creator_Revision 0
(XEN) AMD-Vi: IVRS Block: type 0x10 flags 0x3e len 0x88 id 0x2
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0 -> 0x2
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x10 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x100 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x100 -> 0x101
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x88 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x90 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x90 -> 0x92
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x98 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x98 -> 0x9a
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa0 flags 0xd7
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa3 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa4 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0 id 0 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x43 id 0x200 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0x200 -> 0x3ff alias 0xa4
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa5 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa8 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa9 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x400 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0xb0 flags 0
(XEN) AMD-Vi:  Dev_Id Range: 0xb0 -> 0xb2
(XEN) AMD-Vi: IVHD Device Entry: type 0 id 0 flags 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0xd7
(XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x2 handle 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0
(XEN) AMD-Vi: IVHD Special: 0000:00:00.1 variety 0x1 handle 0x6
(XEN) AMD-Vi: IOMMU 0 Enabled.
(XEN) AMD-Vi: per-device vector map logic is broken.  Using per-device-global 
maps instead until a fix is found.

Looks like IOAPIC definition is missing from IVRS. The one that's listed(0000:00:00.1 variety 0x1 handle 0x6) is usually unused IIRC. I'd expectan entry for the second one (that lives on the southbridge), such as

    AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x1 handle 0

And page faults that you are seeing may also be related to this since0xa0 is 00:14.0.

BIOS update might help with this problem (looks like Biostar has a newerBIOS for this board) but I think this should be fixed so maybe you wantto keep this BIOS until a fix is developed.

You could try booting with iommu=no-intremap to work around this in themeantime.


-boris

(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) ENABLING IO-APIC IRQs
((XEN) ..MP-BIOS bug: 8254 timer not connected to IO-APIC
(XEN) ...trying to set up timer (IRQ0) through the 8259A ...  failed.set up 
timer as Virtual Wire IRQ... works.
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 1048576 KiB.
(XEN) HVM: ASIDs enabled.
(XEN) SVM: Supper
(XEN) HVM: SVM enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) microcode: CPU1 collect_cpu_info: patch_id=0x10000bf
(XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 
0xfdf8010140, flags = 0x8
(XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 
0xfdf8010140, flags = 0x8
(XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 
0xfdf8010140, flags = 0x8
(XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 
0xfdf8010140, flags = 0x8
(XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 
0xfdf8010140, flags = 0x8
(XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 
0xfdf8010140, flags = 0x8
(XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 
0xfdf8010140, flags = 0x8
(XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 
0xfdf8010140, flags = 0x8
(XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 
0xfdf8010140, flags = 0x8
(XEN) AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0xa0, fault address = 
0xfdf8010140, flags = 0x8
(XEN) microcode: CPU2 collect_cpu_info: patch_id=0x10000bf
(XEN) microcode: CPU3 collect_cpu_info: patch_id=0x10000bf
(XEN) microcode: CPU4 collect_cpu_info: patch_id=0x10000bf
(XEN) Brought up 6 CPUs
(XEN) microcode: CPU5 collect_cpu_info: pat 'get_rte_index(rte) == offset' 
failed at iommu_intr.c:159
(XEN) ----[ Xen-4.4-unstable  x86_64  debug=y  Tainted:    C ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82d0801522e3>] amd_iommu_ioapic_update_ire+0x447/0x600
(XEN) RFLAGS: 0000000000010002   CONTEXT: hypervisor
(XEN) rax: ffff8302292f2004   rbx: 0000000000000015   rcx: 0000000000000000
(XEN) rdx: 0000000000000000   rsi: 0000000000000000   rdi: ffff82d0802f00c8
(XEN) rbp: ffff82d0802c7d88   rsp: ffff82d0802c7cb8   r8:  0000000000000020
(XEN) r9:  0000000000000000   r10: 000000000000000f   r11: ffff82d080226f00
(XEN) r12: 0000000000000000   r13: 0000000000000014   r14: 0000000000000000
(XEN) r15: ffff8302292cc380   cr0: 000000008005003b   cr4: 00000000000006f0
(XEN) cr3: 00000000cfc73000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff82d0802c7cb8:
(XEN)    0000000000000000 0100000000000086 0000000000000046 0000001480129548
(XEN)    ffff82d080127200 0000000000000000 0000000000000000 0000000000000000
(XEN)    ffff8302292e0070 0100000000000000 0000000100000002 ffff82d0802c7d48
(XEN)    ffff830200000001 ffff830200000001 0000001500000001 0000000000205000
(XEN)    0100000000010000 ffff82d08016ab76 0000000000010000 0000000000000000
(XEN)    0000000000000000 ffff830229276b50 ffff82d080263d24 0000000000000002
(XEN)    ffff82cffffff010 0000000000000000 ffff82d0802c7d98 ffff82d080143787
(XEN)    ffff82d0802c7df8 ffff82d0801654e8 0000000000000282 0000000000000292
(XEN)    ffff830229280000 0100000000000282 ffff82d0802c7e08 0000000000000002
(XEN)    ffff830229280000 0000000000000000 ffff82d080260d20 0000000000000000
(XEN)    ffff82d0802c7e38 ffff82d0801656b0 0000000000000006 0000000000000006
(XEN)    ffff83000006efb0 ffff83022931a7f0 ffff83022931a7f0 ffff82d080265080
(XEN)    ffff82d0802c7e48 ffff82d08029d0b7 ffff82d0802c7f08 ffff82d08029c7b7
(XEN)    0000000000000000 0000000000000000 ffff82d0802406bd 0000000001281000
(XEN)    0000000000000000 ffff83000006efb0 ffff82d0ffffffff 0000000000000005
(XEN)    ffff83000006eef0 ffff83000006efb0 ffff82d000000005 ffff82d000000006
(XEN)    0000000800000000 000000010000006e 0000000000000003 00000000000002f8
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff82d0801000b5
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN)    [<ffff82d0801522e3>] amd_iommu_ioapic_update_ire+0x447/0x600
(XEN)    [<ffff82d080143787>] iommu_update_ire_from_apic+0x34/0x36
(XEN)    [<ffff82d0801654e8>] set_ioapic_affinity_irq+0xa5/0x1d6
(XEN)    [<ffff82d0801656b0>] setup_ioapic_dest+0x97/0xca
(XEN)    [<ffff82d08029d0b7>] smp_cpus_done+0x51/0x61
(XEN)    [<ffff82d08029c7b7>] __start_xen+0x29dc/0x2d1a
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Assertion 'get_rte_index(rte) == offset' failed at iommu_intr.c:159
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

telnet> Connection closed.
[Connecting to system 32 ]

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

Follow-Ups:
- Re: [Xen-devel] Regression with Xen 4.4 when booting with AMD IOMMU on TA890FXE BIOSTAR
  - From: Konrad Rzeszutek Wilk

References:
- [Xen-devel] Regression with Xen 4.4 when booting with AMD IOMMU on TA890FXE BIOSTAR
  - From: Konrad Rzeszutek Wilk
- Re: [Xen-devel] Regression with Xen 4.4 when booting with AMD IOMMU on TA890FXE BIOSTAR
  - From: Andrew Cooper
- Re: [Xen-devel] Regression with Xen 4.4 when booting with AMD IOMMU on TA890FXE BIOSTAR
  - From: Konrad Rzeszutek Wilk

Prev by Date: [Xen-devel] Is it possible to run Xen ARM on Google/Samsung Nexus 10?
Next by Date: Re: [Xen-devel] OS porting on Xen on ARM
Previous by thread: Re: [Xen-devel] Regression with Xen 4.4 when booting with AMD IOMMU on TA890FXE BIOSTAR
Next by thread: Re: [Xen-devel] Regression with Xen 4.4 when booting with AMD IOMMU on TA890FXE BIOSTAR
Index(es):
- Date
- Thread

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.