[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] Random page allocation failures when block-attaching an LVM snapshot



Dear list,
I have dozens of DomUs on several Debian Dom0s. I do backups of them by taking LVM snapshots on Dom0 and attaching them to DomU. Occasionally the block-attach fails with page allocation failure in DomU:

Sep 4 20:13:38 openvpn kernel: [696863.453080] xenwatch: page allocation failure: order:5, mode:0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null) Sep 4 20:13:38 openvpn kernel: [696863.453093] xenwatch cpuset=/ mems_allowed=0 Sep 4 20:13:38 openvpn kernel: [696863.453099] CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.19.0-5-amd64 #1 Debian 4.19.37-5+deb10u2
Sep  4 20:13:38 openvpn kernel: [696863.453103] Call Trace:
Sep  4 20:13:38 openvpn kernel: [696863.453111]  dump_stack+0x5c/0x80
Sep 4 20:13:38 openvpn kernel: [696863.453116] warn_alloc.cold.117+0x6c/0xec Sep 4 20:13:38 openvpn kernel: [696863.453120] __alloc_pages_slowpath+0xc73/0xcb0 Sep 4 20:13:38 openvpn kernel: [696863.453127] ? test_reply.isra.1+0x40/0x40 Sep 4 20:13:38 openvpn kernel: [696863.453131] ? test_reply.isra.1+0x40/0x40 Sep 4 20:13:38 openvpn kernel: [696863.453134] __alloc_pages_nodemask+0x28b/0x2b0
Sep  4 20:13:38 openvpn kernel: [696863.453139]  kmalloc_order+0x14/0x30
Sep 4 20:13:38 openvpn kernel: [696863.453142] kmalloc_order_trace+0x1d/0xa0 Sep 4 20:13:38 openvpn kernel: [696863.453144] ? test_reply.isra.1+0x40/0x40 Sep 4 20:13:38 openvpn kernel: [696863.453149] talk_to_blkback+0xc0/0xe00 [xen_blkfront]
Sep  4 20:13:38 openvpn kernel: [696863.453152]  ? xenbus_gather+0x13b/0x160
Sep 4 20:13:38 openvpn kernel: [696863.453155] ? test_reply.isra.1+0x40/0x40 Sep 4 20:13:38 openvpn kernel: [696863.453157] blkback_changed+0x133/0xbf1 [xen_blkfront] Sep 4 20:13:38 openvpn kernel: [696863.453160] ? test_reply.isra.1+0x40/0x40 Sep 4 20:13:38 openvpn kernel: [696863.453162] ? xenbus_read_driver_state+0x39/0x60 Sep 4 20:13:38 openvpn kernel: [696863.453165] ? test_reply.isra.1+0x40/0x40
Sep  4 20:13:38 openvpn kernel: [696863.453168]  xenwatch_thread+0xcf/0x170
Sep  4 20:13:38 openvpn kernel: [696863.453173]  ? finish_wait+0x80/0x80
Sep  4 20:13:38 openvpn kernel: [696863.453177]  kthread+0x112/0x130
Sep  4 20:13:38 openvpn kernel: [696863.453180]  ? kthread_bind+0x30/0x30
Sep  4 20:13:38 openvpn kernel: [696863.453184]  ret_from_fork+0x35/0x40
Sep  4 20:13:38 openvpn kernel: [696863.453193] Mem-Info:
Sep 4 20:13:38 openvpn kernel: [696863.453197] active_anon:48394 inactive_anon:1305 isolated_anon:0 Sep 4 20:13:38 openvpn kernel: [696863.453197] active_file:19592 inactive_file:27796 isolated_file:0 Sep 4 20:13:38 openvpn kernel: [696863.453197] unevictable:0 dirty:24 writeback:0 unstable:0 Sep 4 20:13:38 openvpn kernel: [696863.453197] slab_reclaimable:8236 slab_unreclaimable:6425 Sep 4 20:13:38 openvpn kernel: [696863.453197] mapped:6505 shmem:1369 pagetables:569 bounce:0 Sep 4 20:13:38 openvpn kernel: [696863.453197] free:4144 free_pcp:0 free_cma:0 Sep 4 20:13:38 openvpn kernel: [696863.453211] Node 0 active_anon:193576kB inactive_anon:5220kB active_file:78368kB inactive_file:111184kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:26020kB dirty:96kB writeback:0kB shmem:5476kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no Sep 4 20:13:38 openvpn kernel: [696863.453219] Node 0 DMA free:8456kB min:100kB low:124kB high:148kB active_anon:2592kB inactive_anon:0kB active_file:1988kB inactive_file:1480kB unevictable:0kB writepending:8kB present:15996kB managed:15912kB mlocked:0kB kernel_stack:16kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Sep 4 20:13:38 openvpn kernel: [696863.453230] lowmem_reserve[]: 0 377 377 377 377 Sep 4 20:13:38 openvpn kernel: [696863.453233] Node 0 DMA32 free:8120kB min:2432kB low:3040kB high:3648kB active_anon:190984kB inactive_anon:5220kB active_file:76380kB inactive_file:109704kB unevictable:0kB writepending:88kB present:507904kB managed:477256kB mlocked:0kB kernel_stack:2064kB pagetables:2276kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep  4 20:13:38 openvpn kernel: [696863.453244] lowmem_reserve[]: 0 0 0 0 0
Sep 4 20:13:38 openvpn kernel: [696863.453248] Node 0 DMA: 506*4kB (UM) 346*8kB (UM) 147*16kB (UM) 35*32kB (UM) 3*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8456kB Sep 4 20:13:38 openvpn kernel: [696863.453258] Node 0 DMA32: 379*4kB (UMEH) 446*8kB (UMEH) 96*16kB (UMEH) 29*32kB (UMEH) 9*64kB (UH) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8124kB
Sep  4 20:13:38 openvpn kernel: [696863.453268] 48760 total pagecache pages
Sep  4 20:13:38 openvpn kernel: [696863.453272] 0 pages in swap cache
Sep 4 20:13:38 openvpn kernel: [696863.453274] Swap cache stats: add 0, delete 0, find 0/0
Sep  4 20:13:38 openvpn kernel: [696863.453276] Free swap  = 0kB
Sep  4 20:13:38 openvpn kernel: [696863.453277] Total swap = 0kB
Sep  4 20:13:38 openvpn kernel: [696863.453279] 130975 pages RAM
Sep  4 20:13:38 openvpn kernel: [696863.453282] 0 pages HighMem/MovableOnly
Sep  4 20:13:38 openvpn kernel: [696863.453283] 7683 pages reserved
Sep  4 20:13:38 openvpn kernel: [696863.453285] 0 pages hwpoisoned
Sep 4 20:13:38 openvpn kernel: [696863.453289] vbd vbd-268441857: 12 allocating ring_info structure Sep 4 20:13:38 openvpn kernel: [696863.453347] vbd vbd-268441857: failed to write error node for device/vbd/268441857 (12 allocating ring_info structure)

Xen versions: 4.4.1 and 4.8.5-pre.

I can workaround the issue by re-attaching the snapshot, but how can I fix it for good?
--
Karol Jędrzejczyk

--
CONFIDENTIALITY NOTICE


This message and any attachment is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, confidential, legally privileged or otherwise exempt from disclosure. The security and integrity of e-mails cannot be guaranteed. Any reply to this e-mail and any attachment received may be subject to Icotera monitoring. If you are not the named addressee, you are not authorized to use, distribute, copy or take any action in reliance on this message. The unauthorized use, disclosure, or copying of this communication, or any attachment, is strictly prohibited and may be unlawful. No waiver of confidentiality or any applicable privilege is intended by any mistransmission. If you have received this message in error, please notify the sender immediately by replying to this e-mail and delete all copies of this message and any attachments.







_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.