|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] xenwatch: page allocation failure: order:4, mode:0x10c0d0 xen_netback:xenvif_alloc: Could not allocate netdev for vif16.0
On Tue, Nov 19, 2013 at 02:02:57PM +0100, Sander Eikelenboom wrote: > Hi Wei, > > I ran into the following problem when trying to boot another guest after less > than a day of uptime. > (the system started 15 guests at boot already which went fine). dom0 is > allocated a fixed 1536M. > > Both host as pv guests run the same kernel, some hvm's run a slightly older > kernel (3.9 f.e.) > > The are quite some granttable messages in xl dmesg, i also included these and > a "vmstat -m" > This looks like a normal OOM. alloc_netdev is not able to allocate xenvif structure, which is only a few hundred / thousand bytes (don't remember the exact number). Are you using persistent grant in blkback? I think those grant table messages come from blkback hording grant references. It also implies it might be hording Dom0 pages. You can try to turn off persistent grant and see if it works. Wei. > -- > Sander > > [54807.299791] xenwatch: page allocation failure: order:4, mode:0x10c0d0 > [54807.317520] CPU: 5 PID: 54 Comm: xenwatch Not tainted 3.12.0-20131104 #1 > [54807.334747] Hardware name: MSI MS-7640/890FXA-GD70 (MS-7640) , BIOS > V1.8B1 09/13/2010 > [54807.351990] 0000000000000000 ffff8800591a3828 ffffffff81a6a8ba > 0000000000000006 > [54807.368857] 000000000010c0d0 ffff8800591a38b8 ffffffff81138770 > 0000000000000004 > [54807.385539] 000000000010c0d0 ffff8800591a38b8 ffffffff81a661de > ffff88005fd1c680 > [54807.401978] Call Trace: > [54807.418015] [<ffffffff81a6a8ba>] dump_stack+0x4f/0x84 > [54807.433971] [<ffffffff81138770>] warn_alloc_failed+0xf0/0x140 > [54807.449644] [<ffffffff81a661de>] ? > __alloc_pages_direct_compact+0x1ac/0x1be > [54807.465164] [<ffffffff8113bfaa>] __alloc_pages_nodemask+0x7aa/0x9d0 > [54807.480510] [<ffffffff810ed069>] ? trace_hardirqs_off_caller+0xb9/0x160 > [54807.495622] [<ffffffff81175277>] alloc_pages_current+0xb7/0x180 > [54807.510530] [<ffffffff81138059>] __get_free_pages+0x9/0x40 > [54807.525185] [<ffffffff8117cbdc>] __kmalloc+0x19c/0x1c0 > [54807.539538] [<ffffffff8190e9b4>] alloc_netdev_mqs+0x64/0x340 > [54807.553814] [<ffffffff8192ac20>] ? alloc_etherdev_mqs+0x20/0x20 > [54807.567777] [<ffffffff816dc3e4>] xenvif_alloc+0x64/0x2c0 > [54807.581473] [<ffffffff816dbc57>] netback_probe+0x287/0x2d0 > [54807.594971] [<ffffffff814bfe46>] xenbus_dev_probe+0x66/0x110 > [54807.608231] [<ffffffff81615105>] driver_probe_device+0x75/0x210 > [54807.621227] [<ffffffff81615350>] ? __driver_attach+0xb0/0xb0 > [54807.634071] [<ffffffff8161539b>] __device_attach+0x4b/0x60 > [54807.646626] [<ffffffff8161316e>] bus_for_each_drv+0x4e/0xa0 > [54807.658918] [<ffffffff81615058>] device_attach+0x98/0xb0 > [54807.671253] [<ffffffff816144b0>] bus_probe_device+0xb0/0xe0 > [54807.683379] [<ffffffff81612277>] device_add+0x3b7/0x700 > [54807.695145] [<ffffffff8161ca1d>] ? device_pm_sleep_init+0x4d/0x80 > [54807.706824] [<ffffffff816125d9>] device_register+0x19/0x20 > [54807.718145] [<ffffffff814bf9b1>] xenbus_probe_node+0x141/0x170 > [54807.729256] [<ffffffff81613236>] ? bus_for_each_dev+0x76/0xa0 > [54807.740091] [<ffffffff814bfbb0>] xenbus_dev_changed+0x1d0/0x1e0 > [54807.750811] [<ffffffff814bff26>] backend_changed+0x16/0x20 > [54807.761256] [<ffffffff814bdf3e>] xenwatch_thread+0x4e/0x140 > [54807.771371] [<ffffffff810bc1e0>] ? __init_waitqueue_head+0x60/0x60 > [54807.781443] [<ffffffff814bdef0>] ? xs_watch+0x60/0x60 > [54807.791310] [<ffffffff810bb716>] kthread+0xd6/0xe0 > [54807.800794] [<ffffffff81a752bb>] ? _raw_spin_unlock_irq+0x2b/0x70 > [54807.810150] [<ffffffff810bb640>] ? __init_kthread_worker+0x70/0x70 > [54807.819526] [<ffffffff81a762cc>] ret_from_fork+0x7c/0xb0 > [54807.828638] [<ffffffff810bb640>] ? __init_kthread_worker+0x70/0x70 > [54807.837482] Mem-Info: > [54807.846011] Node 0 DMA per-cpu: > [54807.854277] CPU 0: hi: 0, btch: 1 usd: 0 > [54807.862404] CPU 1: hi: 0, btch: 1 usd: 0 > [54807.870241] CPU 2: hi: 0, btch: 1 usd: 0 > [54807.877985] CPU 3: hi: 0, btch: 1 usd: 0 > [54807.885344] CPU 4: hi: 0, btch: 1 usd: 0 > [54807.892389] CPU 5: hi: 0, btch: 1 usd: 0 > [54807.899102] Node 0 DMA32 per-cpu: > [54807.905665] CPU 0: hi: 186, btch: 31 usd: 68 > [54807.911985] CPU 1: hi: 186, btch: 31 usd: 6 > [54807.917964] CPU 2: hi: 186, btch: 31 usd: 149 > [54807.923666] CPU 3: hi: 186, btch: 31 usd: 82 > [54807.929031] CPU 4: hi: 186, btch: 31 usd: 169 > [54807.934293] CPU 5: hi: 186, btch: 31 usd: 0 > [54807.939207] active_anon:12850 inactive_anon:8944 isolated_anon:0 > [54807.939207] active_file:70321 inactive_file:177850 isolated_file:0 > [54807.939207] unevictable:562 dirty:34 writeback:0 unstable:0 > [54807.939207] free:31143 slab_reclaimable:21805 slab_unreclaimable:12717 > [54807.939207] mapped:3344 shmem:276 pagetables:1211 bounce:0 > [54807.939207] free_cma:0 > [54807.966465] Node 0 DMA free:5644kB min:52kB low:64kB high:76kB > active_anon:0kB inactive_anon:4kB active_file:4kB inactive_file:0kB > unevictable:24kB isolated(anon):0kB isolated(file):0kB present:15968kB > managed:15884kB mlocked:24kB dirty:0kB writeback:0kB mapped:24kB shmem:0kB > slab_reclaimable:8992kB slab_unreclaimable:656kB kernel_stack:16kB > pagetables:56kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB > pages_scanned:0 all_unreclaimable? no > [54807.981676] lowmem_reserve[]: 0 1395 1395 1395 > [54807.987015] Node 0 DMA32 free:118928kB min:4748kB low:5932kB high:7120kB > active_anon:51028kB inactive_anon:35772kB active_file:281280kB > inactive_file:711400kB unevictable:2224kB isolated(anon):0kB > isolated(file):0kB present:1556480kB managed:1433216kB mlocked:2224kB > dirty:136kB writeback:0kB mapped:13168kB shmem:1104kB > slab_reclaimable:78228kB slab_unreclaimable:50212kB kernel_stack:3256kB > pagetables:4788kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB > pages_scanned:0 all_unreclaimable? no > [54808.011107] lowmem_reserve[]: 0 0 0 0 > [54808.017244] Node 0 DMA: 19*4kB (UEM) 18*8kB (UEM) 15*16kB (UEM) 14*32kB > (UEM) 8*64kB (UEM) 7*128kB (UEM) 3*256kB (U) 1*512kB (U) 0*1024kB 1*2048kB > (R) 0*4096kB = 5644kB > [54808.030315] Node 0 DMA32: 11490*4kB (UEM) 6743*8kB (UEM) 1133*16kB (EMR) > 6*32kB (MR) 0*64kB 1*128kB (R) 1*256kB (R) 0*512kB 0*1024kB 0*2048kB 0*4096kB > = 118608kB > [54808.044303] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 > hugepages_size=2048kB > [54808.051507] 249203 total pagecache pages > [54808.058647] 201 pages in swap cache > [54808.065986] Swap cache stats: add 2902, delete 2701, find 62055/62191 > [54808.073216] Free swap = 2092752kB > [54808.080397] Total swap = 2097148kB > [54808.093896] 393215 pages RAM > [54808.101004] 30940 pages reserved > [54808.108168] 339108 pages shared > [54808.115202] 258777 pages non-shared > [54808.122235] xen_netback:xenvif_alloc: Could not allocate netdev for vif16.0 > [54808.129391] vif vif-16-0: 12 creating interface > > > ~# vmstat -m > Cache Num Total Size Pages > ext4_groupinfo_4k 24426 24426 216 18 > ceph_osd_request 0 0 960 17 > xt_hashlimit 0 0 136 30 > nf_conntrack_ffffffff822d2900 1950 2100 400 20 > nf_conntrack_expect 0 0 288 28 > dm_snap_pending_exception 2496 2496 104 39 > kcopyd_job 91 133 4176 7 > dm_rq_target_io 0 0 416 19 > search 0 0 616 26 > blkif_cache 108 108 864 18 > cfq_io_cq 204 204 120 34 > cfq_queue 170 170 232 17 > bsg_cmd 0 0 312 26 > ceph_cap 0 0 128 32 > ceph_inode_info 0 0 2176 15 > gfs2_mblk 0 0 432 18 > gfs2_bufdata 0 0 88 46 > gfs2_inode 0 0 1280 25 > gfs2_glock(aspace) 0 0 960 17 > gfs2_glock 0 0 592 27 > btrfs_delayed_data_ref 0 0 96 42 > btrfs_delayed_ref_head 0 0 232 17 > btrfs_delayed_node 0 0 392 20 > btrfs_ordered_extent 0 0 472 17 > btrfs_extent_buffer 0 0 568 28 > btrfs_delalloc_work 0 0 184 22 > btrfs_path 0 0 144 28 > btrfs_transaction 0 0 600 27 > btrfs_trans_handle 0 0 160 25 > btrfs_inode 0 0 1944 16 > fuse_request 102 102 480 17 > fuse_inode 336 476 1152 28 > ntfs_big_inode_cache 0 0 1664 19 > ntfs_inode_cache 0 0 696 23 > cifs_small_rq 36 36 448 18 > cifs_request 4 4 16512 1 > cifs_inode_cache 0 0 1112 29 > isofs_inode_cache 0 0 960 17 > fat_inode_cache 0 0 1168 28 > fat_cache 0 0 40 102 > hugetlbfs_inode_cache 16 16 976 16 > jbd2_transaction_s 150 150 320 25 > jbd2_journal_handle 306 306 80 51 > journal_handle 0 0 56 73 > journal_head 1518 1800 112 36 > revoke_table 1536 1536 16 256 > revoke_record 768 768 32 128 > ext4_inode_cache 30702 30952 1704 19 > ext4_free_data 768 768 64 64 > ext4_allocation_context 180 180 136 30 > ext4_prealloc_space 156 156 152 26 > ext4_io_end 336 336 72 56 > ext4_extent_status 7361 7446 40 102 > ext3_inode_cache 0 0 1312 24 > ext3_xattr 3726 3956 88 46 > dquot 0 0 384 21 > kioctx 0 0 896 18 > pid_namespace 0 0 2208 14 > posix_timers_cache 108 108 296 27 > UNIX 198 198 1472 22 > Cache Num Total Size Pages > UDP-Lite 0 0 1280 25 > ip_fib_trie 438 438 56 73 > PING 0 0 1216 26 > UDP 150 150 1280 25 > tw_sock_TCP 399 399 192 21 > TCP 104 104 2368 13 > fscache_cookie_jar 0 0 192 21 > sgpool-128 54 78 5120 6 > sgpool-64 72 72 2560 12 > sgpool-32 325 325 1280 25 > sgpool-16 277 350 640 25 > blkdev_integrity 0 0 112 36 > blkdev_queue 165 165 2872 11 > blkdev_requests 315 315 376 21 > fsnotify_event_holder 0 0 24 170 > sock_inode_cache 272 272 960 17 > file_lock_cache 102 102 240 17 > shmem_inode_cache 2019 2072 1144 28 > Acpi-State 255 255 80 51 > Acpi-Namespace 1428 1428 40 102 > task_delay_info 1704 1704 168 24 > taskstats 144 144 328 24 > proc_inode_cache 1632 1664 976 16 > sigqueue 250 250 160 25 > bdev_cache 192 192 1344 24 > sysfs_dir_cache 23817 23968 144 28 > filp 1807 2400 320 25 > inode_cache 1861 2244 912 17 > dentry 27252 29440 248 16 > buffer_head 67283 102063 104 39 > vm_area_struct 4527 4708 184 22 > mm_struct 308 308 1152 28 > files_cache 253 253 704 23 > signal_cache 667 667 1408 23 > sighand_cache 407 490 2240 14 > task_struct 430 442 4240 7 > anon_vma 4676 4676 144 28 > shared_policy_node 850 850 48 85 > numa_policy 3528 3528 72 56 > radix_tree_node 7622 8050 568 28 > idr_layer_cache 240 240 2112 15 > dma-kmalloc-8192 0 0 8192 4 > dma-kmalloc-4096 0 0 4096 8 > dma-kmalloc-2048 0 0 2048 16 > dma-kmalloc-1024 0 0 1024 16 > dma-kmalloc-512 0 0 512 16 > dma-kmalloc-256 0 0 256 16 > dma-kmalloc-128 0 0 128 32 > dma-kmalloc-64 0 0 64 64 > dma-kmalloc-32 0 0 32 128 > dma-kmalloc-16 0 0 16 256 > dma-kmalloc-8 0 0 8 512 > dma-kmalloc-192 0 0 192 21 > dma-kmalloc-96 0 0 96 42 > kmalloc-8192 32 32 8192 4 > kmalloc-4096 835 916 4096 8 > kmalloc-2048 773 832 2048 16 > kmalloc-1024 2335 2336 1024 16 > kmalloc-512 1488 1536 512 16 > kmalloc-256 2535 2784 256 16 > Cache Num Total Size Pages > kmalloc-192 69883 70707 192 21 > kmalloc-128 2603 4992 128 32 > kmalloc-96 5628 5628 96 42 > kmalloc-64 41984 41984 64 64 > kmalloc-32 154792 156288 32 128 > kmalloc-16 6343 7936 16 256 > kmalloc-8 8192 8192 8 512 > kmem_cache_node 256 256 128 32 > kmem_cache 176 176 256 16 > > > (XEN) [2013-11-18 21:44:32] grant_table.c:1249:d1 Expanding dom (1) grant > table from (4) to (5) frames. > (XEN) [2013-11-18 21:44:32] grant_table.c:289:d0 Increased maptrack size to 8 > frames > (XEN) [2013-11-18 21:49:28] grant_table.c:1249:d1 Expanding dom (1) grant > table from (5) to (6) frames. > (XEN) [2013-11-19 00:00:26] grant_table.c:1249:d1 Expanding dom (1) grant > table from (6) to (7) frames. > (XEN) [2013-11-19 00:00:26] grant_table.c:1249:d1 Expanding dom (1) grant > table from (7) to (8) frames. > (XEN) [2013-11-19 00:00:26] grant_table.c:289:d0 Increased maptrack size to 9 > frames > (XEN) [2013-11-19 00:00:26] grant_table.c:289:d0 Increased maptrack size to > 10 frames > (XEN) [2013-11-19 00:00:42] grant_table.c:1249:d1 Expanding dom (1) grant > table from (8) to (9) frames. > (XEN) [2013-11-19 00:01:02] grant_table.c:1249:d1 Expanding dom (1) grant > table from (9) to (10) frames. > (XEN) [2013-11-19 00:09:27] grant_table.c:1249:d1 Expanding dom (1) grant > table from (10) to (11) frames. > (XEN) [2013-11-19 00:09:27] grant_table.c:289:d0 Increased maptrack size to > 11 frames > (XEN) [2013-11-19 04:15:26] grant_table.c:289:d0 Increased maptrack size to > 12 frames > (XEN) [2013-11-19 04:15:28] grant_table.c:1249:d12 Expanding dom (12) grant > table from (4) to (5) frames. > (XEN) [2013-11-19 04:15:40] grant_table.c:289:d0 Increased maptrack size to > 13 frames > (XEN) [2013-11-19 04:15:47] grant_table.c:1249:d10 Expanding dom (10) grant > table from (4) to (5) frames. > (XEN) [2013-11-19 04:15:47] grant_table.c:1249:d10 Expanding dom (10) grant > table from (5) to (6) frames. > (XEN) [2013-11-19 04:15:47] grant_table.c:289:d0 Increased maptrack size to > 14 frames > (XEN) [2013-11-19 04:15:52] grant_table.c:1249:d5 Expanding dom (5) grant > table from (4) to (5) frames. > (XEN) [2013-11-19 04:15:52] grant_table.c:1249:d5 Expanding dom (5) grant > table from (5) to (6) frames. > (XEN) [2013-11-19 04:15:52] grant_table.c:1249:d5 Expanding dom (5) grant > table from (6) to (7) frames. > (XEN) [2013-11-19 04:15:52] grant_table.c:289:d0 Increased maptrack size to > 15 frames > (XEN) [2013-11-19 04:15:52] grant_table.c:289:d0 Increased maptrack size to > 16 frames > (XEN) [2013-11-19 04:15:54] grant_table.c:1249:d8 Expanding dom (8) grant > table from (4) to (5) frames. > (XEN) [2013-11-19 04:15:54] grant_table.c:1249:d8 Expanding dom (8) grant > table from (5) to (6) frames. > (XEN) [2013-11-19 04:15:54] grant_table.c:1249:d8 Expanding dom (8) grant > table from (6) to (7) frames. > (XEN) [2013-11-19 04:15:54] grant_table.c:289:d0 Increased maptrack size to > 17 frames > (XEN) [2013-11-19 04:15:54] grant_table.c:289:d0 Increased maptrack size to > 18 frames > (XEN) [2013-11-19 04:15:56] grant_table.c:1249:d2 Expanding dom (2) grant > table from (4) to (5) frames. > (XEN) [2013-11-19 04:15:56] grant_table.c:1249:d2 Expanding dom (2) grant > table from (5) to (6) frames. > (XEN) [2013-11-19 04:15:56] grant_table.c:1249:d2 Expanding dom (2) grant > table from (6) to (7) frames. > (XEN) [2013-11-19 04:15:56] grant_table.c:289:d0 Increased maptrack size to > 19 frames > (XEN) [2013-11-19 04:15:57] grant_table.c:1249:d3 Expanding dom (3) grant > table from (4) to (5) frames. > (XEN) [2013-11-19 04:15:57] grant_table.c:1249:d3 Expanding dom (3) grant > table from (5) to (6) frames. > (XEN) [2013-11-19 04:15:57] grant_table.c:1249:d3 Expanding dom (3) grant > table from (6) to (7) frames. > (XEN) [2013-11-19 04:15:57] grant_table.c:289:d0 Increased maptrack size to > 20 frames > (XEN) [2013-11-19 04:16:00] grant_table.c:289:d0 Increased maptrack size to > 21 frames > (XEN) [2013-11-19 04:16:00] grant_table.c:1249:d13 Expanding dom (13) grant > table from (4) to (5) frames. > (XEN) [2013-11-19 04:16:00] grant_table.c:1249:d13 Expanding dom (13) grant > table from (5) to (6) frames. > (XEN) [2013-11-19 04:16:00] grant_table.c:289:d0 Increased maptrack size to > 22 frames > (XEN) [2013-11-19 04:16:00] grant_table.c:1249:d13 Expanding dom (13) grant > table from (6) to (7) frames. > (XEN) [2013-11-19 04:16:03] grant_table.c:1249:d9 Expanding dom (9) grant > table from (4) to (5) frames. > (XEN) [2013-11-19 04:16:03] grant_table.c:1249:d9 Expanding dom (9) grant > table from (5) to (6) frames. > (XEN) [2013-11-19 04:16:03] grant_table.c:1249:d9 Expanding dom (9) grant > table from (6) to (7) frames. > (XEN) [2013-11-19 04:16:03] grant_table.c:289:d0 Increased maptrack size to > 23 frames > (XEN) [2013-11-19 04:16:03] grant_table.c:289:d0 Increased maptrack size to > 24 frames > (XEN) [2013-11-19 04:16:03] grant_table.c:1249:d4 Expanding dom (4) grant > table from (4) to (5) frames. > (XEN) [2013-11-19 04:16:03] grant_table.c:1249:d4 Expanding dom (4) grant > table from (5) to (6) frames. > (XEN) [2013-11-19 04:16:03] grant_table.c:1249:d4 Expanding dom (4) grant > table from (6) to (7) frames. > (XEN) [2013-11-19 04:16:03] grant_table.c:289:d0 Increased maptrack size to > 25 frames > (XEN) [2013-11-19 04:16:03] grant_table.c:289:d0 Increased maptrack size to > 26 frames > (XEN) [2013-11-19 04:16:06] grant_table.c:1249:d10 Expanding dom (10) grant > table from (6) to (7) frames. > (XEN) [2013-11-19 04:16:22] grant_table.c:1249:d12 Expanding dom (12) grant > table from (5) to (6) frames. > (XEN) [2013-11-19 04:16:22] grant_table.c:1249:d12 Expanding dom (12) grant > table from (6) to (7) frames. > (XEN) [2013-11-19 04:16:22] grant_table.c:289:d0 Increased maptrack size to > 27 frames > (XEN) [2013-11-19 04:16:23] grant_table.c:289:d0 Increased maptrack size to > 28 frames > (XEN) [2013-11-19 04:16:24] grant_table.c:1249:d11 Expanding dom (11) grant > table from (4) to (5) frames. > (XEN) [2013-11-19 04:16:24] grant_table.c:1249:d11 Expanding dom (11) grant > table from (5) to (6) frames. > (XEN) [2013-11-19 04:16:24] grant_table.c:289:d0 Increased maptrack size to > 29 frames > (XEN) [2013-11-19 04:16:27] grant_table.c:1249:d7 Expanding dom (7) grant > table from (5) to (6) frames. > (XEN) [2013-11-19 04:16:27] grant_table.c:1249:d7 Expanding dom (7) grant > table from (6) to (7) frames. > (XEN) [2013-11-19 05:15:04] grant_table.c:289:d0 Increased maptrack size to > 30 frames > (XEN) [2013-11-19 05:15:05] grant_table.c:1249:d6 Expanding dom (6) grant > table from (4) to (5) frames. > (XEN) [2013-11-19 05:15:05] grant_table.c:1249:d6 Expanding dom (6) grant > table from (5) to (6) frames. > (XEN) [2013-11-19 05:15:05] grant_table.c:1249:d6 Expanding dom (6) grant > table from (6) to (7) frames. > (XEN) [2013-11-19 05:15:05] grant_table.c:289:d0 Increased maptrack size to > 31 frames > (XEN) [2013-11-19 12:49:37] AMD-Vi: Share p2m table with iommu: p2m table = > 0x52d3c5 _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |