Re: [Xen-devel] Unable to get VT-d working on Supermicro X11SSZ-QF

[Tamas K Lengyel <tamas.k.lengyel@xxxxxxxxx>]

Hi Xudong,

I've been also been adding some debug messages to see where iommu gets disabled and it seems that the VT-d init fails at http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/drivers/passthrough/vtd/iommu.c;h=82549376cfa64f311c0981bdd74418a0295d3ff9;hb=HEAD#l2139

Also, here is the full xl dmesg log.

ÂXen 4.7-unstable
(XEN) Xen version 4.7-unstable (root@) (gcc (Debian 4.9.2-10) 4.9.2) debug=y Sat Oct 17 17:04:30 MDT 2015
(XEN) Latest ChangeSet: Sat Oct 3 15:22:29 2015 -0400 git:a23ce42
(XEN) Bootloader: GRUB 2.02~beta2-22
(XEN) Command line: placeholder loglvl=all guest_loglvl=all iommu=verbose no-real-mode edd=off
(XEN) Video information:
(XEN)Â VGA is text mode 80x25, font 8x16
(XEN) Disc information:
(XEN)Â Found 0 MBR signatures
(XEN)Â Found 0 EDD information structures
(XEN) Multiboot-e820 RAM map:
(XEN)Â 0000000000000000 - 0000000000058000 (usable)
(XEN)Â 0000000000058000 - 0000000000059000 (reserved)
(XEN)Â 0000000000059000 - 000000000009f000 (usable)
(XEN)Â 000000000009f000 - 00000000000a0000 (reserved)
(XEN)Â 0000000000100000 - 0000000082ac3000 (usable)
(XEN)Â 0000000082ac3000 - 0000000082ac4000 (ACPI NVS)
(XEN)Â 0000000082ac4000 - 0000000082b0e000 (reserved)
(XEN)Â 0000000082b0e000 - 0000000086b7e000 (usable)
(XEN)Â 0000000086b7e000 - 0000000086f0a000 (reserved)
(XEN)Â 0000000086f0a000 - 00000000870cf000 (usable)
(XEN)Â 00000000870cf000 - 0000000087729000 (ACPI NVS)
(XEN)Â 0000000087729000 - 0000000087e6e000 (reserved)
(XEN)Â 0000000087e6e000 - 0000000087fff000 type 20
(XEN)Â 0000000087fff000 - 0000000088000000 (usable)
(XEN)Â 00000000e0000000 - 00000000f0000000 (reserved)
(XEN)Â 00000000fe000000 - 00000000fe011000 (reserved)
(XEN)Â 00000000fec00000 - 00000000fec01000 (reserved)
(XEN)Â 00000000fee00000 - 00000000fee01000 (reserved)
(XEN)Â 00000000ff000000 - 0000000100000000 (reserved)
(XEN)Â 0000000100000000 - 0000000873000000 (usable)
(XEN) ACPI Error (tbxfroot-0218): A valid RSDP was not found [20070126]
(XEN) System RAM: 32668MB (33452636kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-0000000873000000
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000fcc20
(XEN) DMI 3.0 present.
(XEN) Using APIC driver default
(XEN) Intel MultiProcessor Specification v1.4
(XEN)ÂÂÂÂ Virtual Wire compatibility mode.
(XEN) OEM ID: SMCI--MB Product ID: SUPERM APIC at: 0xfee00000
(XEN) Processor #0 6:14 APIC version 21
(XEN) Processor #2 6:14 APIC version 21
(XEN) Processor #4 6:14 APIC version 21
(XEN) Processor #6 6:14 APIC version 21
(XEN) I/O APIC #2 Version 32 at 0xfec00000.
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) Processors: 4
(XEN) SMP: Allowing 4 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 24 GSI, 760 MSI/MSI-X
(XEN) Not enabling x2APIC: depends on iommu_supports_eim.
(XEN) xstate_init: using cntxt_size: 0x440 and states: 0x1f
(XEN) mce_intel.c:735: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0 extended MCE MSR 0
(XEN) Intel machine check reporting enabled
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3408.136 MHz processor.
(XEN) Initing memory sharing.
(XEN) alt table ffff82d0802e9e10 -> ffff82d0802eb0c4
(XEN) spurious 8259A interrupt: IRQ7.
(XEN) I/O virtualisation disabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)Â -> Using old ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=0 pin2=0
(XEN) TSC deadline timer enabled
(XEN) Platform timer is 1.193MHz PIT
(XEN) Allocated console ring of 32 KiB.
(XEN) mwait-idle: does not run on family 6 model 94
(XEN) VMX: Supported advanced features:
(XEN)Â - APIC MMIO access virtualisation
(XEN)Â - APIC TPR shadow
(XEN)Â - Extended Page Tables (EPT)
(XEN)Â - Virtual-Processor Identifiers (VPID)
(XEN)Â - Virtual NMI
(XEN)Â - MSR direct-access bitmap
(XEN)Â - Unrestricted Guest
(XEN)Â - VMCS shadowing
(XEN)Â - VM Functions
(XEN)Â - Virtualisation Exceptions
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Brought up 4 CPUs
(XEN) ACPI sleep modes: S3
(XEN) VPMU: disabled
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) Dom0 has maximum 600 PIRQs
(XEN) ACPI is disabled, notifying Domain 0 (acpi=off)
(XEN) NX (Execute Disable) protection active
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_parse_binary: phdr: paddr=0x1000000 memsz=0x8b1000
(XEN) elf_parse_binary: phdr: paddr=0x1a00000 memsz=0x115000
(XEN) elf_parse_binary: phdr: paddr=0x1b15000 memsz=0x16a18
(XEN) elf_parse_binary: phdr: paddr=0x1b2c000 memsz=0x22b000
(XEN) elf_parse_binary: memory: 0x1000000 -> 0x1d57000
(XEN) elf_xen_parse_note: GUEST_OS = "linux"
(XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) elf_xen_parse_note: ENTRY = 0xffffffff81b2c1f0
(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
(XEN) elf_xen_parse_note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb|writable_descriptor_tables|auto_translated_physmap|supervisor_mode_kernel"
(XEN) elf_xen_parse_note: SUPPORTED_FEATURES = 0x90d
(XEN) elf_xen_parse_note: PAE_MODE = "yes"
(XEN) elf_xen_parse_note: LOADER = "generic"
(XEN) elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) elf_xen_parse_note: MOD_START_PFN = 0x1
(XEN) elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
(XEN) elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) elf_xen_addr_calc_check: addresses:
(XEN)ÂÂÂÂ virt_baseÂÂÂÂÂÂÂ = 0xffffffff80000000
(XEN)ÂÂÂÂ elf_paddr_offset = 0x0
(XEN)ÂÂÂÂ virt_offsetÂÂÂÂÂ = 0xffffffff80000000
(XEN)ÂÂÂÂ virt_kstartÂÂÂÂÂ = 0xffffffff81000000
(XEN)ÂÂÂÂ virt_kendÂÂÂÂÂÂÂ = 0xffffffff81d57000
(XEN)ÂÂÂÂ virt_entryÂÂÂÂÂÂ = 0xffffffff81b2c1f0
(XEN)ÂÂÂÂ p2m_baseÂÂÂÂÂÂÂÂ = 0xffffffffffffffff
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN)Â Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1d57000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)Â Dom0 alloc.:ÂÂ 0000000850000000->0000000858000000 (8201279 pages to be allocated)
(XEN)Â Init. ramdisk: 0000000871fbf000->0000000872ffff3b
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)Â Loaded kernel: ffffffff81000000->ffffffff81d57000
(XEN)Â Init. ramdisk: 0000000000000000->0000000000000000
(XEN)Â Phys-Mach map: ffffffff81d57000->ffffffff85c31400
(XEN)Â Start info:ÂÂÂ ffffffff85c32000->ffffffff85c324b4
(XEN)Â Page tables:ÂÂ ffffffff85c33000->ffffffff85c66000
(XEN)Â Boot stack:ÂÂÂ ffffffff85c66000->ffffffff85c67000
(XEN)Â TOTAL:ÂÂÂÂÂÂÂÂ ffffffff80000000->ffffffff86000000
(XEN)Â ENTRY ADDRESS: ffffffff81b2c1f0
(XEN) Dom0 has maximum 4 VCPUs
(XEN) elf_load_binary: phdr 0 at 0xffffffff81000000 -> 0xffffffff818b1000
(XEN) elf_load_binary: phdr 1 at 0xffffffff81a00000 -> 0xffffffff81b15000
(XEN) elf_load_binary: phdr 2 at 0xffffffff81b15000 -> 0xffffffff81b2ba18
(XEN) elf_load_binary: phdr 3 at 0xffffffff81b2c000 -> 0xffffffff81c5f000
(XEN) Scrubbing Free RAM on 1 nodes using 4 CPUs
(XEN) ....................................................................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 296kB init memory.
(XEN) traps.c:2682:d0v0 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) traps.c:2682:d0v0 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0bffff000 to 0xffffffff81572750.
(XEN) traps.c:2682:d0v0 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0bffff020 to 0xffffffff81574d20.
(XEN) traps.c:2682:d0v0 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) traps.c:2682:d0v0 Domain attempted WRMSR 0000000000000175 from 0xffff8300868f7fc0 to 0x0000000000000000.
(XEN) traps.c:2682:d0v0 Domain attempted WRMSR 0000000000000176 from 0xffff82d08023cda0 to 0xffffffff81574b20.
(XEN) traps.c:2682:d0v0 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:02.0
(XEN) PCI add device 0000:00:08.0
(XEN) PCI add device 0000:00:14.0
(XEN) PCI add device 0000:00:14.2
(XEN) PCI add device 0000:00:16.0
(XEN) PCI add device 0000:00:16.3
(XEN) PCI add device 0000:00:17.0
(XEN) PCI add device 0000:00:1d.0
(XEN) PCI add device 0000:00:1d.1
(XEN) PCI add device 0000:00:1d.2
(XEN) PCI add device 0000:00:1f.0
(XEN) PCI add device 0000:00:1f.2
(XEN) PCI add device 0000:00:1f.3
(XEN) PCI add device 0000:00:1f.4
(XEN) PCI add device 0000:00:1f.6
(XEN) PCI add device 0000:02:00.0
(XEN) PCI add device 0000:03:00.0
(XEN) PCI add device 0000:04:00.0

On Sun, Oct 18, 2015 at 8:11 PM, Hao, Xudong <xudong.hao@xxxxxxxxx> wrote:

VT-d is enabled by default on Xen, can you share the whole log of Xen âxl dmesgâ?

Â

-Thanks

Xudong

Â

From: xen-devel-bounces@xxxxxxxxxxxxx [mailto:xen-devel-bounces@xxxxxxxxxxxxx] On Behalf Of Tamas K Lengyel
Sent: Monday, October 19, 2015 4:59 AM
To: Xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>
Subject: [Xen-devel] Unable to get VT-d working on Supermicro X11SSZ-QF

Â

Hi all,

I've recently got a new Supermicro X11SSZ-QF board which has the Q170 chipset (http://www.supermicro.com/products/motherboard/Core/Q170/X11SSZ-QF.cfm). The CPU is a i7-6700. I'm unable to get VT-d working under Xen. I've verified that that VT-d is enabled in the BIOS and if I boot only Linux with intel_iommu=on dmesg shows the following:

# dmesg | grep -e DMAR -e IOMMU
[ÂÂÂ 0.000000] ACPI: DMAR 0x0000000087710980 0000A8 (v01 INTELÂ SKLÂÂÂÂÂ 00000001 INTL 00000001)
[ÂÂÂ 0.000000] DMAR: IOMMU enabled
[ÂÂÂ 0.043288] DMAR: Host address width 39
[ÂÂÂ 0.043291] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ÂÂÂ 0.043297] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e
[ÂÂÂ 0.043300] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ÂÂÂ 0.043305] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ÂÂÂ 0.043308] DMAR: RMRR base: 0x00000086c61000 end: 0x00000086c80fff
[ÂÂÂ 0.043309] DMAR: RMRR base: 0x00000088800000 end: 0x0000008affffff
[ 0.043312] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ÂÂÂ 0.043314] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ÂÂÂ 0.043316] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[ÂÂÂ 0.043317] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[ÂÂÂ 0.043711] DMAR-IR: Enabled IRQ remapping in xapic mode
[ÂÂÂ 0.519525] DMAR: No ATSR found
[ÂÂÂ 0.519572] DMAR: dmar0: Using Queued invalidation
[ÂÂÂ 0.519784] DMAR: dmar1: Using Queued invalidation
[ÂÂÂ 0.519790] DMAR: Setting RMRR:
[ÂÂÂ 0.519797] DMAR: Setting identity map for device 0000:00:02.0 [0x88800000 - 0x8affffff]
[ÂÂÂ 0.519810] DMAR: Setting identity map for device 0000:00:14.0 [0x86c61000 - 0x86c80fff]
[ÂÂÂ 0.519819] DMAR: Prepare 0-16MiB unity mapping for LPC
[ÂÂÂ 0.519824] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[ÂÂÂ 0.519845] DMAR: Intel(R) Virtualization Technology for Directed I/O

However,

Xen 4.6 and unstable both report
(XEN) I/O virtualisation disabled

Any help on trying to pin down the issue would be appreciated!

Thanks,

Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel