Re: [Xen-devel] Crash in acpi_ps_peek_opcode when booting kernel 3.19 as Xen dom0

[Juergen Gross <jgross@xxxxxxxx>]

On 02/09/2015 02:33 PM, Stefan Bader wrote:
> On 09.02.2015 14:07, Stefan Bader wrote:
> > On 05.02.2015 20:36, Konrad Rzeszutek Wilk wrote:
> > > On Thu, Feb 05, 2015 at 03:33:02PM +0100, Stefan Bader wrote:
> > > > While experimenting/testing various kernel versions I discovered that trying to
> > > > boot a Haswell based hosts will always crash when booting as Xen dom0
> > > > (Xen-4.4.1). The same crash happens since v3.19-rc1 and still does happen with
> > > > v3.19-rc7. A bare metal boot is having no issues and also an Opteron based host
> > > > is having no issues (dom0 and bare metal).
> > > > Could be a table that the other host does not have and since its only happening
> > > > in dom0 maybe some cpu capability that needs to be passed on?
> > > Usually it means that the ACPI AML code is trying to do something with
> > > the IOAPIC or something wihch is not accessible.
> > >
> > > But this on the other hand looks to be trying to execute some AML code
> > > that is unknown. Any chance you cna disassemble it and perhaps also
> > > run with acpi debug options on to figure out where it blows up?
> > The weird thing here is that bare-metal on the same machine does work. And
> > previous kernels did work as well. So I think we can assume the ACPI tables are
> > ok. It could even be a red-herring. Well, likely is as booting with acpi=off
> > does hang instead of crashing.
> >
> > Since I got no clue, I did what we always do when we are dumbfound, I went ahead
> > and bisected 3.18..3.19-rc1. Unfortunately the very last kernel I build was
> > something in between good and bad. Good as it did not crash exactly but bad as
> > it did not come up in a usable state. So I would not be sure the claimed to be
> > offending commit is right. Could be one in the range of:
> >
> > G  * xen: use common page allocation function in p2m.c
> >     * xen: Delay remapping memory of pv-domain
> > g  * xen: Delay m2p_override initialization
> > -> * xen: Delay invalidating extra memory
> > B  * x86: Introduce function to get pmd entry pointer
> >
> > (G) really good, (g) somewhat not bad, (B) bad, (->) claimed first broken.
> Oh, since that all sounds related to E820 in some way:
>
> (XEN) Xen-e820 RAM map:
> (XEN)  0000000000000000 - 000000000009a400 (usable)
> (XEN)  000000000009a400 - 00000000000a0000 (reserved)
> (XEN)  00000000000e0000 - 0000000000100000 (reserved)
> (XEN)  0000000000100000 - 0000000030a48000 (usable)
> (XEN)  0000000030a48000 - 0000000030a49000 (reserved)
Hmm, this memory hole is at a rather low address. Could it be some
vital data (one of kernel, page tables, initrd or p2m map) is located
at this address?

This would be a problem similar to the one I ran into when trying to
test on a machine with 1TB of memory, where the p2m map was too big
to fit into contiguous memory.

Could you check the addresses where the hypervisor puts this data for
Dom0?


Juergen

> (XEN)  0000000030a49000 - 00000000a27f4000 (usable)
> (XEN)  00000000a27f4000 - 00000000a2ab4000 (reserved)
> (XEN)  00000000a2ab4000 - 00000000a2fb4000 (ACPI NVS)
> (XEN)  00000000a2fb4000 - 00000000a2feb000 (ACPI data)
> (XEN)  00000000a2feb000 - 00000000a3000000 (usable)
> (XEN)  00000000a3000000 - 00000000afa00000 (reserved)
> (XEN)  00000000e0000000 - 00000000f0000000 (reserved)
> (XEN)  00000000fec00000 - 00000000fec01000 (reserved)
> (XEN)  00000000fed00000 - 00000000fed04000 (reserved)
> (XEN)  00000000fed10000 - 00000000fed1a000 (reserved)
> (XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
> (XEN)  00000000fed84000 - 00000000fed85000 (reserved)
> (XEN)  00000000fee00000 - 00000000fee01000 (reserved)
> (XEN)  00000000ffc00000 - 0000000100000000 (reserved)
> (XEN)  0000000100000000 - 000000024e600000 (usable)
>
> and how it looks with a 3.18 boot:
>
> [    0.000000] e820: BIOS-provided physical RAM map:
> [    0.000000] Xen: [mem 0x0000000000000000-0x0000000000099fff] usable
> [    0.000000] Xen: [mem 0x000000000009a400-0x00000000000fffff] reserved
> [    0.000000] Xen: [mem 0x0000000000100000-0x0000000030a47fff] usable
> [    0.000000] Xen: [mem 0x0000000030a48000-0x0000000030a48fff] reserved
> [    0.000000] Xen: [mem 0x0000000030a49000-0x00000000a27f3fff] usable
> [    0.000000] Xen: [mem 0x00000000a27f4000-0x00000000a2ab3fff] reserved
> [    0.000000] Xen: [mem 0x00000000a2ab4000-0x00000000a2fb3fff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000a2fb4000-0x00000000a2feafff] ACPI data
> [    0.000000] Xen: [mem 0x00000000a2feb000-0x00000000a2ffffff] usable
> [    0.000000] Xen: [mem 0x00000000a3000000-0x00000000af9fffff] reserved
> [    0.000000] Xen: [mem 0x00000000e0000000-0x00000000efffffff] reserved
> [    0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
> [    0.000000] Xen: [mem 0x00000000fed00000-0x00000000fed03fff] reserved
> [    0.000000] Xen: [mem 0x00000000fed10000-0x00000000fed19fff] reserved
> [    0.000000] Xen: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved
> [    0.000000] Xen: [mem 0x00000000fed84000-0x00000000fed84fff] reserved
> [    0.000000] Xen: [mem 0x00000000fee00000-0x00000000feefffff] reserved
> [    0.000000] Xen: [mem 0x00000000ffc00000-0x00000000ffffffff] reserved
> [    0.000000] Xen: [mem 0x0000000100000000-0x00000001bdc59fff] usable
> [    0.000000] Xen: [mem 0x00000001bdc5a000-0x000000024e5fffff] unusable
>
> Not sure that helps much. I probably have to try comparing later output. But
> that will need a bit of time.
>
> -Stefan
>
> > So it seems one of the delaying changes has a very bad effect on that Sharkbay.
> > A bit odd since none of those sounds Intel/AMD geared. Could only be a different
> > usage of memory (my AMD box has considerably more memory and also no CPU with
> > GPU functionality as the Haswell).
> >
> > Jürgen, maybe some description that might trigger an idea for you...?
> >
> > -Stefan
> >
> > ---
> >
> > git bisect start
> > # good: [b2776bf7149bddd1f4161f14f79520f17fc1d71d] Linux 3.18
> > git bisect good b2776bf7149bddd1f4161f14f79520f17fc1d71d
> > # bad: [97bf6af1f928216fd6c5a66e8a57bfa95a659672] Linux 3.19-rc1
> > git bisect bad 97bf6af1f928216fd6c5a66e8a57bfa95a659672
> > # good: [70e71ca0af244f48a5dcf56dc435243792e3a495] Merge
> > git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
> > git bisect good 70e71ca0af244f48a5dcf56dc435243792e3a495
> > # good: [988adfdffdd43cfd841df734664727993076d7cb] Merge branch 'drm-next' of
> > git://people.freedesktop.org/~airlied/linux
> > git bisect good 988adfdffdd43cfd841df734664727993076d7cb
> > # good: [b024793188002b9eed452b5f6a04d45003ed5772] staging: rtl8723au:
> > phy_SsPwrSwitch92CU() was never called with bRegSSPwrLvl != 1
> > git bisect good b024793188002b9eed452b5f6a04d45003ed5772
> > # bad: [66dcff86ba40eebb5133cccf450878f2bba102ef] Merge tag 'for-linus' of
> > git://git.kernel.org/pub/scm/virt/kvm/kvm
> > git bisect bad 66dcff86ba40eebb5133cccf450878f2bba102ef
> > # bad: [d6666be6f0c43efb9475d1d35fbef9f8be61b7b1] Merge tag 'for-linus-20141215'
> > of git://git.infradead.org/linux-mtd
> > git bisect bad d6666be6f0c43efb9475d1d35fbef9f8be61b7b1
> > # bad: [94bbdb63d7ed5ca56b788e43d0ca4a8f9494c9e7] Merge tag 'fixes-for-linus' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
> > git bisect bad 94bbdb63d7ed5ca56b788e43d0ca4a8f9494c9e7
> > # good: [2dbfca5a181973558277b28b1f4c36362291f5e0] Merge branch 'for-next' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/cooloney/linux-leds
> > git bisect good 2dbfca5a181973558277b28b1f4c36362291f5e0
> > # bad: [0db2812a5240f2663b92d8d4b761122dd2e0c6c3] Merge
> > git://git.kernel.org/pub/scm/linux/kernel/git/cmetcalf/linux-tile
> > git bisect bad 0db2812a5240f2663b92d8d4b761122dd2e0c6c3
> > # bad: [f1d04b23b2015b4c3c0a8419677179b133afea08] Merge branch
> > 'devel/for-linus-3.19' into stable/for-linus-3.19
> > git bisect bad f1d04b23b2015b4c3c0a8419677179b133afea08
> > # bad: [792230c3a66b3d17d6dcca712866d24f2283d4a6] x86: Introduce function to get
> > pmd entry pointer
> > git bisect bad 792230c3a66b3d17d6dcca712866d24f2283d4a6
> > # good: [7108c9ce8f6e59f775b0c8250dba52b569b6cba2] xen: use common page
> > allocation function in p2m.c
> > # NOTE: This was the last really good
> > git bisect good 7108c9ce8f6e59f775b0c8250dba52b569b6cba2
> > # good: [97f4533a60ce5d0cb35ff44a190111f81a987620] xen: Delay m2p_override
> > initialization
> > # NOTE: This revision did not crash the usual way but was not useable either
> > # NOTE: Use of wrong bits in page-tables.
> > git bisect good 97f4533a60ce5d0cb35ff44a190111f81a987620
> > > > [    2.108038] ACPI: Core revision 20141107
> > > > [    2.108153] ACPI Warning: Unsupported module-level executable opcode 0x91 at
> > > > table offset 0x002B (20141107/psloop-225)
> > > > [    2.108264] ACPI Warning: Unsupported module-level executable opcode 0x91 at
> > > > table offset 0x0033 (20141107/psloop-225)
> > > > [    2.108375] ACPI Warning: Unsupported module-level executable opcode 0x95 at
> > > > table offset 0x0038 (20141107/psloop-225)
> > > > [    2.108489] ACPI Warning: Unsupported module-level executable opcode 0x95 at
> > > > table offset 0x0041 (20141107/psloop-225)
> > > > [    2.108613] ACPI Warning: Unsupported module-level executable opcode 0x7D at
> > > > table offset 0x040D (20141107/psloop-225)
> > > > [    2.108751] BUG: unable to handle kernel paging request at ffffc90000ee74e0
> > > > [    2.108835] IP: [<ffffffff814573db>] acpi_ps_peek_opcode+0xd/0x1f
> > > > [    2.108902] PGD 1f4be067 PUD 1f4bd067 PMD 1488f067 PTE 0
> > > > [    2.109018] Oops: 0000 [#1] SMP
> > > > [    2.109094] Modules linked in:
> > > > [    2.109153] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.19.0-031900rc7-generi
> > > > c #201502020035
> > > > [    2.109220] Hardware name: Intel Corporation Shark Bay Client platform/Flathe
> > > > ad Creek Crb, BIOS HSWLPTU1.86C.0109.R03.1301282055 01/28/2013
> > > > [    2.109295] task: ffffffff81c1c500 ti: ffffffff81c00000 task.ti: ffffffff81c0
> > > > 0000
> > > > [    2.109360] RIP: e030:[<ffffffff814573db>]  [<ffffffff814573db>] acpi_ps_peek
> > > > _opcode+0xd/0x1f
> > > > [    2.109445] RSP: e02b:ffffffff81c03ce8  EFLAGS: 00010283
> > > > [    2.109490] RAX: 000000000000000c RBX: ffff880014887000 RCX: ffffffff81c03d50
> > > > [    2.109539] RDX: ffffc90000ee74e0 RSI: ffff880014887030 RDI: ffff880014887030
> > > > [    2.109587] RBP: ffffffff81c03ce8 R08: ffffea0000522600 R09: ffffffff81432c4f
> > > > [    2.109635] R10: ffff880014899090 R11: 00000000000000ba R12: ffff880014887030
> > > > [    2.109684] R13: ffff880014887000 R14: ffffffff81c03d50 R15: 000000000000000d
> > > > [    2.109735] FS:  0000000000000000(0000) GS:ffff880018c00000(0000) knlGS:00000
> > > > 00000000000
> > > > [    2.109836] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > [    2.109881] CR2: ffffc90000ee74e0 CR3: 0000000001c15000 CR4: 0000000000042660
> > > > [    2.109930] Stack:
> > > > [    2.109968]  ffffffff81c03d38 ffffffff81456537 ffffffff81c03d28 ffffffff81457
> > > > a40
> > > > [    2.110104]  ffff880014887000 ffff880014887000 ffff8800148990c0 ffffc90000ee7
> > > > 4e0
> > > > [    2.110238]  ffff880014887030 0000000000000000 ffffffff81c03d78 ffffffff81456
> > > > 760
> > > > [    2.110373] Call Trace:
> > > > [    2.110413]  [<ffffffff81456537>] acpi_ps_get_next_arg+0x114/0x1f9
> > > > [    2.110461]  [<ffffffff81457a40>] ? acpi_ps_pop_scope+0x54/0x72
> > > > [    2.110508]  [<ffffffff81456760>] acpi_ps_get_arguments+0x91/0x228
> > > > [    2.110555]  [<ffffffff81456ad2>] acpi_ps_parse_loop+0x1db/0x311
> > > > [    2.110602]  [<ffffffff81457705>] acpi_ps_parse_aml+0x96/0x275
> > > > [    2.110649]  [<ffffffff8145322f>] acpi_ns_one_complete_parse+0xf7/0x114
> > > > [    2.110698]  [<ffffffff817d149a>] ? _raw_spin_lock_irqsave+0x1a/0x60
> > > > [    2.110746]  [<ffffffff8145326c>] acpi_ns_parse_table+0x20/0x38
> > > > [    2.110792]  [<ffffffff81452c20>] acpi_ns_load_table+0x4c/0x90
> > > > [    2.110840]  [<ffffffff817c50b5>] acpi_tb_load_namespace+0xa6/0x14a
> > > > [    2.110889]  [<ffffffff81d83269>] acpi_load_tables+0xc/0x35
> > > > [    2.110935]  [<ffffffff81454bf6>] ? acpi_ns_get_node+0xb7/0xc9
> > > > [    2.110982]  [<ffffffff81d825cf>] acpi_early_init+0x73/0x105
> > > > [    2.111029]  [<ffffffff81d3b083>] start_kernel+0x348/0x3f0
> > > > [    2.111075]  [<ffffffff81d3abcd>] ? set_init_arg+0x56/0x56
> > > > [    2.111121]  [<ffffffff81d3a5f8>] x86_64_start_reservations+0x2a/0x2c
> > > > [    2.111169]  [<ffffffff81d3e88c>] xen_start_kernel+0x4f5/0x4f7
> > > > [    2.111215] Code: 8a 87 60 05 87 81 5d c3 e8 73 cc 37 00 55 81 ff 00 01 00 00
> > > >   19 c0 48 89 e5 83 c0 02 5d c3 e8 5d cc 3
> > >
> > >
> > > > _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@xxxxxxxxxxxxx
> > > > http://lists.xen.org/xen-devel
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel