[Xen-users] Random page allocation failures when block-attaching an LVM snapshot

[Karol Jędrzejczyk <kjedrzejczyk@xxxxxxxxxxx>]

Dear list,
I have dozens of DomUs on several Debian Dom0s. I do backups of them by 
taking LVM snapshots on Dom0 and attaching them to DomU. Occasionally 
the block-attach fails with page allocation failure in DomU:
Sep  4 20:13:38 openvpn kernel: [696863.453080] xenwatch: page 
allocation failure: order:5, 
mode:0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null)
Sep  4 20:13:38 openvpn kernel: [696863.453093] xenwatch cpuset=/ 
mems_allowed=0
Sep  4 20:13:38 openvpn kernel: [696863.453099] CPU: 0 PID: 19 Comm: 
xenwatch Not tainted 4.19.0-5-amd64 #1 Debian 4.19.37-5+deb10u2
Sep  4 20:13:38 openvpn kernel: [696863.453103] Call Trace:
Sep  4 20:13:38 openvpn kernel: [696863.453111]  dump_stack+0x5c/0x80
Sep  4 20:13:38 openvpn kernel: [696863.453116] 
warn_alloc.cold.117+0x6c/0xec
Sep  4 20:13:38 openvpn kernel: [696863.453120] 
__alloc_pages_slowpath+0xc73/0xcb0
Sep  4 20:13:38 openvpn kernel: [696863.453127]  ? 
test_reply.isra.1+0x40/0x40
Sep  4 20:13:38 openvpn kernel: [696863.453131]  ? 
test_reply.isra.1+0x40/0x40
Sep  4 20:13:38 openvpn kernel: [696863.453134] 
__alloc_pages_nodemask+0x28b/0x2b0
Sep  4 20:13:38 openvpn kernel: [696863.453139]  kmalloc_order+0x14/0x30
Sep  4 20:13:38 openvpn kernel: [696863.453142] 
kmalloc_order_trace+0x1d/0xa0
Sep  4 20:13:38 openvpn kernel: [696863.453144]  ? 
test_reply.isra.1+0x40/0x40
Sep  4 20:13:38 openvpn kernel: [696863.453149] 
talk_to_blkback+0xc0/0xe00 [xen_blkfront]
Sep  4 20:13:38 openvpn kernel: [696863.453152]  ? xenbus_gather+0x13b/0x160
Sep  4 20:13:38 openvpn kernel: [696863.453155]  ? 
test_reply.isra.1+0x40/0x40
Sep  4 20:13:38 openvpn kernel: [696863.453157] 
blkback_changed+0x133/0xbf1 [xen_blkfront]
Sep  4 20:13:38 openvpn kernel: [696863.453160]  ? 
test_reply.isra.1+0x40/0x40
Sep  4 20:13:38 openvpn kernel: [696863.453162]  ? 
xenbus_read_driver_state+0x39/0x60
Sep  4 20:13:38 openvpn kernel: [696863.453165]  ? 
test_reply.isra.1+0x40/0x40
Sep  4 20:13:38 openvpn kernel: [696863.453168]  xenwatch_thread+0xcf/0x170
Sep  4 20:13:38 openvpn kernel: [696863.453173]  ? finish_wait+0x80/0x80
Sep  4 20:13:38 openvpn kernel: [696863.453177]  kthread+0x112/0x130
Sep  4 20:13:38 openvpn kernel: [696863.453180]  ? kthread_bind+0x30/0x30
Sep  4 20:13:38 openvpn kernel: [696863.453184]  ret_from_fork+0x35/0x40
Sep  4 20:13:38 openvpn kernel: [696863.453193] Mem-Info:
Sep  4 20:13:38 openvpn kernel: [696863.453197] active_anon:48394 
inactive_anon:1305 isolated_anon:0
Sep  4 20:13:38 openvpn kernel: [696863.453197]  active_file:19592 
inactive_file:27796 isolated_file:0
Sep  4 20:13:38 openvpn kernel: [696863.453197]  unevictable:0 dirty:24 
writeback:0 unstable:0
Sep  4 20:13:38 openvpn kernel: [696863.453197]  slab_reclaimable:8236 
slab_unreclaimable:6425
Sep  4 20:13:38 openvpn kernel: [696863.453197]  mapped:6505 shmem:1369 
pagetables:569 bounce:0
Sep  4 20:13:38 openvpn kernel: [696863.453197]  free:4144 free_pcp:0 
free_cma:0
Sep  4 20:13:38 openvpn kernel: [696863.453211] Node 0 
active_anon:193576kB inactive_anon:5220kB active_file:78368kB 
inactive_file:111184kB unevictable:0kB isolated(anon):0kB 
isolated(file):0kB mapped:26020kB dirty:96kB writeback:0kB shmem:5476kB 
shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB 
unstable:0kB all_unreclaimable? no
Sep  4 20:13:38 openvpn kernel: [696863.453219] Node 0 DMA free:8456kB 
min:100kB low:124kB high:148kB active_anon:2592kB inactive_anon:0kB 
active_file:1988kB inactive_file:1480kB unevictable:0kB writepending:8kB 
present:15996kB managed:15912kB mlocked:0kB kernel_stack:16kB 
pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep  4 20:13:38 openvpn kernel: [696863.453230] lowmem_reserve[]: 0 377 
377 377 377
Sep  4 20:13:38 openvpn kernel: [696863.453233] Node 0 DMA32 free:8120kB 
min:2432kB low:3040kB high:3648kB active_anon:190984kB 
inactive_anon:5220kB active_file:76380kB inactive_file:109704kB 
unevictable:0kB writepending:88kB present:507904kB managed:477256kB 
mlocked:0kB kernel_stack:2064kB pagetables:2276kB bounce:0kB 
free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep  4 20:13:38 openvpn kernel: [696863.453244] lowmem_reserve[]: 0 0 0 0 0
Sep  4 20:13:38 openvpn kernel: [696863.453248] Node 0 DMA: 506*4kB (UM) 
346*8kB (UM) 147*16kB (UM) 35*32kB (UM) 3*64kB (UM) 0*128kB 0*256kB 
0*512kB 0*1024kB 0*2048kB 0*4096kB = 8456kB
Sep  4 20:13:38 openvpn kernel: [696863.453258] Node 0 DMA32: 379*4kB 
(UMEH) 446*8kB (UMEH) 96*16kB (UMEH) 29*32kB (UMEH) 9*64kB (UH) 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8124kB
Sep  4 20:13:38 openvpn kernel: [696863.453268] 48760 total pagecache pages
Sep  4 20:13:38 openvpn kernel: [696863.453272] 0 pages in swap cache
Sep  4 20:13:38 openvpn kernel: [696863.453274] Swap cache stats: add 0, 
delete 0, find 0/0
Sep  4 20:13:38 openvpn kernel: [696863.453276] Free swap  = 0kB
Sep  4 20:13:38 openvpn kernel: [696863.453277] Total swap = 0kB
Sep  4 20:13:38 openvpn kernel: [696863.453279] 130975 pages RAM
Sep  4 20:13:38 openvpn kernel: [696863.453282] 0 pages HighMem/MovableOnly
Sep  4 20:13:38 openvpn kernel: [696863.453283] 7683 pages reserved
Sep  4 20:13:38 openvpn kernel: [696863.453285] 0 pages hwpoisoned
Sep  4 20:13:38 openvpn kernel: [696863.453289] vbd vbd-268441857: 12 
allocating ring_info structure
Sep  4 20:13:38 openvpn kernel: [696863.453347] vbd vbd-268441857: 
failed to write error node for device/vbd/268441857 (12 allocating 
ring_info structure)
Xen versions: 4.4.1 and 4.8.5-pre.

I can workaround the issue by re-attaching the snapshot, but how can I 
fix it for good?
--
Karol Jędrzejczyk

--
CONFIDENTIALITY NOTICE


This message and any attachment is intended 
exclusively for the individual or entity to which it is addressed. This 
communication may contain information that is proprietary, confidential, 
legally privileged or otherwise exempt from disclosure. The security and 
integrity of e-mails cannot be guaranteed. Any reply to this e-mail and any 
attachment received may be subject to Icotera monitoring.
If you are not 
the named addressee, you are not authorized to use, distribute, copy or 
take any action in reliance on this message. The unauthorized use, 
disclosure, or copying of this communication, or any attachment, is 
strictly prohibited and may be unlawful. No waiver of confidentiality or 
any applicable privilege is intended by any mistransmission. If you have 
received this message in error, please notify the sender immediately by 
replying to this e-mail and delete all copies of this message and any 
attachments.






_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users