Re: [Xen-devel] [PATCH]x86:x2apic: Disable x2apic on x86-32 permanently

[Keir Fraser <keir@xxxxxxx>]

Okay. I think after 4.1 is branched we should simply remove 32-bit host
support entirely. There's really no good reason for it to persist imo.

 -- Keir

On 21/01/2011 15:19, "Li, Xin" <xin.li@xxxxxxxxx> wrote:

> Below is original log.  We found Xen hypervisor needs to map
> VT-d/x2apci/MSI/MSI-x resources using map_domain_page.  And on 32bit, the map
> page pool is trending to be used up.  Maybe we can fix it, but for large
> machines with many devices and CPUs, we probably will still meet the 32bit
> limitation.  So for new features which need map page in Xen hypervisor, I tend
> to enable on 64 bit only.  I'm not sure how important 32bit Xen hypervisor is,
> but I think 64 bit is becoming more and more popular.  Simply disabling is not
> technically elegant, but for longer term, I think it's the trend because it
> makes things simpler :).  Comments?
> Thanks!
> -Xin
> 
> root (hd0,2)
>  Filesystem type is ext2fs, partition type 0x83
> kernel (hd0,2)/boot/xen.gz dom0_mem=512M iommu=1 loglvl=all guest_loglvl=all
> un
> restricted_guest=1  console=com1 sync_console iommu=1 msi=1 conring_size=4M
>    [Multiboot-elf, <0x100000:0x14433c:0x34cc4>, shtab=0x279078,
> entry=0x100000]
> module (hd0,2)/boot/vmlinuz-2.6-xen ro root=/dev/sda3 console=hvc0
> earlyprintk=
> xen ignore_loglevel pci=assign-busses
>    [Multiboot-module @ 0x27a000, 0x3ede40 bytes]
> module (hd0,2)/boot/initrd-2.6-xen.img
>    [Multiboot-module @ 0x668000, 0x369800 bytes]
> 
>  __  __            _  _    _                    _        _     _
>  \ \/ /___ _ __   | || |  / |   _   _ _ __  ___| |_ __ _| |__ | | ___
>   \  // _ \ '_ \  | || |_ | |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
>   /  \  __/ | | | |__   _|| |__| |_| | | | \__ \ || (_| | |_) | |  __/
>  /_/\_\___|_| |_|    |_|(_)_|   \__,_|_| |_|___/\__\__,_|_.__/|_|\___|
> 
> (XEN) Xen version 4.1-unstable (build@xxxxxxxxxxxx) (gcc version 4.1.2
> 20080704 (Red Hat 4.1.2-44)) Sat Sep 11 06:03:28 CST 2010
> (XEN) Latest ChangeSet: Fri Sep 10 19:06:33 2010 +0100 22132:3985fea87987
> (XEN) Console output is synchronous.
> (XEN) Bootloader: GNU GRUB 0.97
> (XEN) Command line: dom0_mem=512M iommu=1 loglvl=all guest_loglvl=all
> unrestricted_guest=1  console=com1 sync_console iommu=1 msi=1 conring_size=4M
> (XEN) Video information:
> (XEN)  VGA is text mode 80x25, font 8x16
> (XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
> (XEN)  EDID info not retrieved because no DDC retrieval method detected
> (XEN) Disc information:
> (XEN)  Found 1 MBR signatures
> (XEN)  Found 1 EDD information structures
> (XEN) Xen-e820 RAM map:
> (XEN)  0000000000000000 - 000000000009bc00 (usable)
> (XEN)  000000000009bc00 - 00000000000a0000 (reserved)
> (XEN)  00000000000e0000 - 0000000000100000 (reserved)
> (XEN)  0000000000100000 - 0000000020050000 (usable)
> (XEN)  0000000020050000 - 0000000020250000 (reserved)
> (XEN)  0000000020250000 - 0000000040000000 (usable)
> (XEN)  0000000040000000 - 0000000040200000 (reserved)
> (XEN)  0000000040200000 - 000000007f553000 (usable)
> (XEN)  000000007f553000 - 000000007f5e7000 (reserved)
> (XEN)  000000007f5e7000 - 000000007f7e7000 (ACPI NVS)
> (XEN)  000000007f7e7000 - 000000007f7ff000 (ACPI data)
> (XEN)  000000007f7ff000 - 000000007f800000 (usable)
> (XEN)  000000007f800000 - 0000000080000000 (reserved)
> (XEN)  00000000f8000000 - 00000000fc000000 (reserved)
> (XEN)  00000000fec00000 - 00000000fec01000 (reserved)
> (XEN)  00000000fed10000 - 00000000fed14000 (reserved)
> (XEN)  00000000fed18000 - 00000000fed1a000 (reserved)
> (XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
> (XEN)  00000000fee00000 - 00000000fee01000 (reserved)
> (XEN)  00000000ff980000 - 00000000ffc00000 (reserved)
> (XEN)  00000000ffd80000 - 0000000100000000 (reserved)
> (XEN) System RAM: 2032MB (2081724kB)
> (XEN) ACPI: RSDP 000F0410, 0024 (r2  INTEL)
> (XEN) ACPI: XSDT 7F7FDE18, 006C (r1  INTEL  SNB-CPT  6222004 MSFT    10013)
> (XEN) ACPI: FACP 7F752D98, 00F4 (r4  INTEL  SNB-CPT  6222004 MSFT    10013)
> (XEN) ACPI: DSDT 7F725018, EACA (r2  INTEL  SNB-CPT        0 INTL 20100331)
> (XEN) ACPI: FACS 7F7E6D40, 0040
> (XEN) ACPI: APIC 7F7FCF18, 0092 (r2  INTEL  SNB-CPT  6222004 MSFT    10013)
> (XEN) ACPI: TCPA 7F79FD18, 0032 (r2                        0             0)
> (XEN) ACPI: MCFG 7F79FC98, 003C (r1 INTEL  SNDYBRDG  6222004 MSFT       97)
> (XEN) ACPI: HPET 7F79FC18, 0038 (r1 A M I   PCHHPET  6222004 AMI.        3)
> (XEN) ACPI: SSDT 7F753018, 1068 (r1 TrmRef PtidDevc     1000 INTL 20100331)
> (XEN) ACPI: BOOT 7F79FB98, 0028 (r1 INTEL  SNB-CPT   6222004 AMI     10013)
> (XEN) ACPI: SSDT 7F751018, 0996 (r1  PmRef    CpuPm     3000 INTL 20100331)
> (XEN) ACPI: DMAR 7F752F18, 00B0 (r1 INTEL      SNB         1 INTL        1)
> (XEN) No NUMA configuration found
> (XEN) Faking a node at 0000000000000000-000000007f800000
> (XEN) Xen heap: 9MB (9716kB)
> (XEN) Domain heap initialised
> (XEN) found SMP MP-table at 000fcaa0
> (XEN) DMI 2.6 present.
> (XEN) Using APIC driver default
> (XEN) ACPI: PM-Timer IO Port: 0x408
> (XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[404,0], pm1x_evt[400,0]
> (XEN) ACPI: 32/64X FACS address mismatch in FADT - 7f79df40/000000007f7e6d40,
> using 32
> (XEN) ACPI:                  wakeup_vec[7f79df4c], vec_size[20]
> (XEN) ACPI: Local APIC address 0xfee00000
> (XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
> (XEN) Processor #0 6:10 APIC version 21
> (XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
> (XEN) Processor #2 6:10 APIC version 21
> (XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
> (XEN) Processor #4 6:10 APIC version 21
> (XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
> (XEN) Processor #6 6:10 APIC version 21
> (XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
> (XEN) Processor #1 6:10 APIC version 21
> (XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
> (XEN) Processor #3 6:10 APIC version 21
> (XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
> (XEN) Processor #5 6:10 APIC version 21
> (XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
> (XEN) Processor #7 6:10 APIC version 21
> (XEN) ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
> (XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
> (XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
> (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
> (XEN) ACPI: IRQ0 used by override.
> (XEN) ACPI: IRQ2 used by override.
> (XEN) ACPI: IRQ9 used by override.
> (XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
> (XEN) ACPI: HPET id: 0x8086a701 base: 0xfed00000
> (XEN) Table is not found!
> (XEN) Using ACPI (MADT) for SMP configuration information
> (XEN) IRQ limits: 24 GSI, 1528 MSI/MSI-X
> (XEN) Early fatal page fault at e008:ff1da1c6 (cr2=ffffffb8, ec=0000)
> (XEN) Stack dump: ff279610 00000080 ff24bfb0 fffff000 00000000 ff24bdd4
> ffffc000 00000000
> ff1da1c6 0000e008 00010082 00000000 00000000 ff24bdd4 ff130e89 ff21828a
> 00000006 ffbfec64 ff279610 00bde0a0 00000000 ff24be04 ff1384f7 0007e95c
> 00000002 00000000 ff11d4ef 00000096 00000096 f73de0a0 ff279610 ff279638
> ff2795b8 ff24be34 ff13c740 ff279568 00000004 c9008020 00000000 ff21ac1c
> ff2795e4 00000096 ff279568 ff21a7bc 00000008 ff24be54 ff13e038 ff2795b8
> 00010970 02fa0000 ff279910 ff279910 00000007 ff24be74 ff144b0a ff279910
> ff24be74 ff232570 00000007 00000000 00100800 ff24bfa4 ff234bf7 ff21dc0c
> 00000080 00000008 00000000 00000000 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 0007bef0 00000000 00000000
> ff220260 00000000 00000000 1f000000 00000000 00000000 ff07bc50 ff07bfb0
> ff07bef0 00757640 00848000 00000015 ffffffff 00000000 00000000 00000000
> 00000000 00000000 00000000 7f7ff000 00000000 40000000 00000000 00000000
> ff07bef0 00000000 00000000 00000000 00000000 00000007 00000007 00000008
> 000001a4 000001a4 ff07bee0 01000000 00000000 ffd80000 00000000 ffd80000
> 00000000 00000015 ffffffff ff07bf24 0000097d 00000000 00000000 00000000
> 00000008 0000006e 00000001 00000003 000002f8 00000000 00000000 00220d70
> ff2737c0 00067ebc ff10006c 0007bfb0 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000 fffff000 00000000
> 
>> -----Original Message-----
>> From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
>> [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Keir Fraser
>> Sent: Friday, January 21, 2011 9:09 PM
>> To: Zhang, Fengzhe; xen-devel@xxxxxxxxxxxxxxxxxxx
>> Subject: Re: [Xen-devel] [PATCH]x86:x2apic: Disable x2apic on x86-32
>> permanently
>> 
>> Could you give more info on the bug, such as a backtrace? I wonder whether
>> this might be better fixed in some more elegant way. Also I can't eyeball
>> where map_domain_page() is being called during x2apic setup.
>> 
>>  Thanks,
>>  Keir
>> 
>> On 21/01/2011 12:21, "Zhang, Fengzhe" <fengzhe.zhang@xxxxxxxxx> wrote:
>> 
>>> x86:x2apic: Disable x2apic on x86-32 permanently
>>> 
>>> x2apic initialization on x86_32 uses vcpu pointer before it is initialized.
>>> As
>>> x2apic is unlikely to be used on x86_32, this patch disables x2apic
>>> permanently on x86_32. It also asserts the sanity of vcpu pointer before
>>> dereference to prevent further misuse.
>>> 
>>> Signed-off-by: Fengzhe Zhang <fengzhe.zhang@xxxxxxxxx>
>>> 
>>> diff -r 02c0af2bf280 xen/arch/x86/apic.c
>>> --- a/xen/arch/x86/apic.c Mon Jan 17 18:05:52 2011 +0000
>>> +++ b/xen/arch/x86/apic.c Wed Jan 19 03:24:16 2011 -0500
>>> @@ -961,6 +961,22 @@
>>>      if ( !cpu_has_x2apic )
>>>          return;
>>> 
>>> +#ifdef __i386__
>>> +    clear_bit(X86_FEATURE_X2APIC, boot_cpu_data.x86_capability);
>>> +    if (x2apic_enabled) {
>>> +        uint64_t msr_content;
>>> +
>>> +        rdmsrl(MSR_IA32_APICBASE, msr_content);
>>> +        msr_content &= ~(MSR_IA32_APICBASE_ENABLE |
>> MSR_IA32_APICBASE_EXTD);
>>> +        wrmsrl(MSR_IA32_APICBASE, msr_content);
>>> +        msr_content |= MSR_IA32_APICBASE_ENABLE;
>>> +        wrmsrl(MSR_IA32_APICBASE, msr_content);
>>> +        x2apic_enabled = 0;
>>> +    }
>>> +    printk("x2APIC disabled permanently on x86_32.\n");
>>> +    return;
>>> +#endif
>>> +
>>>      if ( !opt_x2apic )
>>>      {
>>>          if ( !x2apic_enabled )
>>> diff -r 02c0af2bf280 xen/arch/x86/x86_32/domain_page.c
>>> --- a/xen/arch/x86/x86_32/domain_page.c Mon Jan 17 18:05:52 2011 +0000
>>> +++ b/xen/arch/x86/x86_32/domain_page.c Wed Jan 19 03:24:16 2011 -0500
>>> @@ -53,6 +53,8 @@
>>>      perfc_incr(map_domain_page_count);
>>> 
>>>      v = mapcache_current_vcpu();
>>> +    /* Prevent vcpu pointer being used before initialize. */
>>> +    ASSERT((unsigned long)v != 0xfffff000);
>>> 
>>>      dcache = &v->domain->arch.mapcache;
>>>      vcache = &v->arch.mapcache;
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>>> http://lists.xensource.com/xen-devel
>> 
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>> http://lists.xensource.com/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel