Re: [Xen-devel] [PATCH v2] mm/page_alloc: make bootscrub happen in idle-loop

[Sergey Dyasli <sergey.dyasli@xxxxxxxxxx>]

On 07/11/2018 18:20, Andrew Cooper wrote:
> On 09/10/18 16:21, Sergey Dyasli wrote:
>> Scrubbing RAM during boot may take a long time on machines with lots
>> of RAM. Add 'idle' option to bootscrub which marks all pages dirty
>> initially so they will eventually be scrubbed in idle-loop on every
>> online CPU.
>>
>> It's guaranteed that the allocator will return scrubbed pages by doing
>> eager scrubbing during allocation (unless MEMF_no_scrub was provided).
>>
>> Use the new 'idle' option as the default one.
>>
>> Signed-off-by: Sergey Dyasli <sergey.dyasli@xxxxxxxxxx>
> 
> This patch reliably breaks boot, although its not immediately obvious how:
> 
> (d9) (XEN) mcheck_poll: Machine check polling timer started.
> (d9) (XEN) xenoprof: Initialization failed. Intel processor family 6 model 60 
> is not supported
> (d9) (XEN) Dom0 has maximum 400 PIRQs
> (d9) (XEN) ----[ Xen-4.12-unstable  x86_64  debug=y   Not tainted ]----
> (d9) (XEN) CPU:    0
> (d9) (XEN) RIP:    e008:[<ffff82d080440ddb>] setup.c#cmdline_cook+0x1d/0x77
> (d9) (XEN) RFLAGS: 0000000000010282   CONTEXT: hypervisor
> (d9) (XEN) rax: ffff82d080406bdc   rbx: ffff8300c2c2c2c2   rcx: 
> 0000000000000000
> (d9) (XEN) rdx: 00000007c7ffffff   rsi: ffff83000045c24b   rdi: 
> ffff83000045c24b
> (d9) (XEN) rbp: ffff82d0804b7da8   rsp: ffff82d0804b7d98   r8:  
> ffff83003f057000
> (d9) (XEN) r9:  7fffffffffffffff   r10: 0000000000000000   r11: 
> 0000000000000001
> (d9) (XEN) r12: ffff83003f0d8100   r13: 0000000000000000   r14: 
> ffff82d0805f33d0
> (d9) (XEN) r15: 0000000000000002   cr0: 000000008005003b   cr4: 
> 00000000001526e0
> (d9) (XEN) cr3: 000000003fea7000   cr2: ffff8300c2c2c2c2
> (d9) (XEN) fsb: 0000000000000000   gsb: 0000000000000000   gss: 
> 0000000000000000
> (d9) (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
> (d9) (XEN) Xen code around <ffff82d080440ddb> 
> (setup.c#cmdline_cook+0x1d/0x77):
> (d9) (XEN)  05 5e fc ff 48 0f 44 d8 <80> 3b 20 75 09 48 83 c3 01 80 3b 20 74 
> f7 80 3d
> (d9) (XEN) Xen stack trace from rsp=ffff82d0804b7d98:
> (d9) (XEN)    0000000000000000 ffff8300c2c2c2c2 ffff82d0804b7ee8 
> ffff82d080443b7f
> (d9) (XEN)    0000000000000000 00000000003f3480 0000000000000002 
> 0000000000000002
> (d9) (XEN)    0000000000000002 0000000000000002 0000000000000002 
> 0000000000000001
> (d9) (XEN)    0000000000000001 0000000000000003 00000000000feffc 
> 0000000000000000
> (d9) (XEN)    00000000000feffd 0000000000000000 0000000000800163 
> 00000000feffd000
> (d9) (XEN)    ffff83000045c24b ffffffff00000002 0000000000000001 
> 0000000000000001
> (d9) (XEN)    ffff83000048da80 ffff82d08048db00 0000000000000000 
> 0000000000000000
> (d9) (XEN)    0000000000000000 0000000200000004 00000040ffffffff 
> 0000000000000400
> (d9) (XEN)    0000000800000000 000000010000006e 0000000000000003 
> 00000000000002f8
> (d9) (XEN)    0000000000000000 0000000000000000 0000000000000000 
> 0000000000000000
> (d9) (XEN)    0000000000000000 0000000000000000 0000000000000000 
> ffff82d0802000f3
> (d9) (XEN)    0000000000000000 0000000000000000 0000000000000000 
> 0000000000000000
> (d9) (XEN)    0000000000000000 0000000000000000 0000000000000000 
> 0000000000000000
> (d9) (XEN)    0000000000000000 0000000000000000 0000000000000000 
> 0000000000000000
> (d9) (XEN)    0000000000000000 0000000000000000 0000000000000000 
> 0000000000000000
> (d9) (XEN)    0000000000000000 0000000000000000 0000000000000000 
> 0000000000000000
> (d9) (XEN)    0000000000000000 0000000000000000 0000000000000000 
> 0000000000000000
> (d9) (XEN)    0000000000000000 0000000000000000 ffff83003f0ce000 
> 0000000000000000
> (d9) (XEN)    00000000001526e0 0000000000000000 0000000000000000 
> 0000060000000000
> (d9) (XEN)    0000001800000000
> (d9) (XEN) Xen call trace:
> (d9) (XEN)    [<ffff82d080440ddb>] setup.c#cmdline_cook+0x1d/0x77
> (d9) (XEN)    [<ffff82d080443b7f>] __start_xen+0x259c/0x292d
> (d9) (XEN)    [<ffff82d0802000f3>] __high_start+0x53/0x55
> (d9) (XEN) 
> (d9) (XEN) Pagetable walk from ffff8300c2c2c2c2:
> (d9) (XEN)  L4[0x106] = 800000003fea5063 ffffffffffffffff
> (d9) (XEN)  L3[0x003] = 000000003fea2063 ffffffffffffffff
> (d9) (XEN)  L2[0x016] = 0000000000000000 ffffffffffffffff
> (d9) (XEN) 
> (d9) (XEN) ****************************************
> (d9) (XEN) Panic on CPU 0:
> (d9) (XEN) FATAL PAGE FAULT
> (d9) (XEN) [error_code=0000]
> (d9) (XEN) Faulting linear address: ffff8300c2c2c2c2
> (d9) (XEN) ****************************************
> (d9) (XEN) 
> (d9) (XEN) Reboot in five seconds...
> 
> The low part of 0xffff8300c2c2c2c2 looks to be poisoned, so
> __va(mod[0].string) is obviously turning out to be junk.

0xc2 is a SCRUB_PATTERN, so my patch might have uncovered a real issue.
There are 2 implications of idle scrub:

    1. alloc_xenheap_pages() might return scrubbed memory (despite
       passing MEMF_no_scrub, and after secondary CPUs enter idle-loop)

    2. alloc_domheap_pages() will return scrubbed memory by default
       during Xen boot

What is the exact place of this crash? Maybe zeroing of allocated pages
is needed there? Can you reproduce the issue with Release build, where
scrub pattern is 0?

--
Thanks,
Sergey

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel