Re: [Xen-users] 2.6 Kernel Out of memory

[Marcus Hardt <marcus.hardt@xxxxxxxxxx>]

Are you guys using loopback via NFS? That not working. Think it's linux, not 
xen related,

Marcus.


Quoting Sylvain COUTANT
> Hi guys,
>
>
>
> Another strange behavior with some test servers.
>
>
>
> They are booted using 2.6.11.12, xen-2.0.6, Debian.
>
> The only daemons running on dom0 are : xend, smtp, quagga (ospf), portmap
> (nfs client), sshd.
>
>
>
> Memory usage at startup is aroung 30 MB. Dom0 reserved memory is 256 MB.
>
>
>
> The only things that happen on this server are domU's start/stop (xm
> create, xm shutdown) and a ssh connection every 2 seconds to get the domU's
> states (via xm list).
>
>
>
> We know we could improve the domU states fetching but for now we are stuck
> to the ssh way of doing it.
>
>
>
> After a few hours, the kernel begins to get out of memory and kills every
> process around to gather some. The only log before the server completely
> stops are :
>
>
>
> Aug 22 22:03:37 xenu kernel: oom-killer: gfp_mask=0xd0
>
> Aug 22 22:03:37 xenu kernel: DMA per-cpu:
>
> Aug 22 22:03:37 xenu kernel: cpu 0 hot: low 32, high 96, batch 16
>
> Aug 22 22:03:37 xenu kernel: cpu 0 cold: low 0, high 32, batch 16
>
> Aug 22 22:03:37 xenu kernel: Normal per-cpu: empty
>
> Aug 22 22:03:37 xenu kernel: HighMem per-cpu: empty
>
> Aug 22 22:03:37 xenu kernel:
>
> Aug 22 22:03:37 xenu kernel: Free pages:        2304kB (0kB HighMem)
>
> Aug 22 22:03:37 xenu kernel: Active:2375 inactive:55381 dirty:0
> writeback:54298 unstable:1042 free:576 slab:3961 mapped:2375 pagetables:84
>
> Aug 22 22:03:37 xenu kernel: DMA free:2304kB min:2048kB low:2560kB
> high:3072kB active:9500kB inactive:221524kB present:262144kB
> pages_scanned:193051 all_unreclaimable? no
>
> Aug 22 22:03:37 xenu kernel: lowmem_reserve[]: 0 0 0
>
> Aug 22 22:03:37 xenu kernel: Normal free:0kB min:0kB low:0kB high:0kB
> active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
>
> Aug 22 22:03:37 xenu kernel: lowmem_reserve[]: 0 0 0
>
> Aug 22 22:03:37 xenu kernel: HighMem free:0kB min:128kB low:160kB
> high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0
> all_unreclaimable? no
>
> Aug 22 22:03:37 xenu kernel: lowmem_reserve[]: 0 0 0
>
> Aug 22 22:03:37 xenu kernel: DMA: 44*4kB 26*8kB 22*16kB 1*32kB 0*64kB
> 2*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2304kB
>
> Aug 22 22:03:37 xenu kernel: Normal: empty
>
> Aug 22 22:03:37 xenu kernel: HighMem: empty
>
> Aug 22 22:03:37 xenu kernel: Swap cache: add 0, delete 0, find 0/0, race
> 0+0
>
> Aug 22 22:03:37 xenu kernel: Free swap  = 0kB
>
> Aug 22 22:03:37 xenu kernel: Total swap = 0kB
>
> Aug 22 22:03:37 xenu kernel: Out of Memory: Killed process 1261 (exim4).
>
> Aug 22 22:03:37 xenu kernel: oom-killer: gfp_mask=0x1d2
>
> Aug 22 22:03:37 xenu kernel: DMA per-cpu:
>
> Aug 22 22:03:37 xenu kernel: cpu 0 hot: low 32, high 96, batch 16
>
> Aug 22 22:03:37 xenu kernel: cpu 0 cold: low 0, high 32, batch 16
>
> Aug 22 22:03:37 xenu kernel: Normal per-cpu: empty
>
> Aug 22 22:03:37 xenu kernel: HighMem per-cpu: empty
>
> Aug 22 22:03:37 xenu kernel:
>
> Aug 22 22:03:37 xenu kernel: Free pages:        2368kB (0kB HighMem)
>
> Aug 22 22:03:37 xenu kernel: Active:2316 inactive:55442 dirty:0
> writeback:52965 unstable:2458 free:592 slab:3936 mapped:2316 pagetables:80
>
> Aug 22 22:03:37 xenu kernel: DMA free:2368kB min:2048kB low:2560kB
> high:3072kB active:9264kB inactive:221768kB present:262144kB
> pages_scanned:267628 all_unreclaimable? no
>
> Aug 22 22:03:37 xenu kernel: lowmem_reserve[]: 0 0 0
>
> Aug 22 22:03:37 xenu kernel: Normal free:0kB min:0kB low:0kB high:0kB
> active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
>
> Aug 22 22:03:37 xenu kernel: lowmem_reserve[]: 0 0 0
>
> Aug 22 22:03:37 xenu kernel: HighMem free:0kB min:128kB low:160kB
> high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0
> all_unreclaimable? no
>
> Aug 22 22:03:37 xenu kernel: lowmem_reserve[]: 0 0 0
>
> Aug 22 22:03:37 xenu kernel: DMA: 70*4kB 19*8kB 23*16kB 1*32kB 0*64kB
> 2*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2368kB
>
> Aug 22 22:03:37 xenu kernel: Normal: empty
>
> Aug 22 22:03:37 xenu kernel: HighMem: empty
>
> Aug 22 22:03:37 xenu kernel: Swap cache: add 0, delete 0, find 0/0, race
> 0+0
>
> Aug 22 22:03:37 xenu kernel: Free swap  = 0kB
>
> Aug 22 22:03:37 xenu kernel: Total swap = 0kB
>
> Aug 22 22:03:43 xenu kernel: Out of Memory: Killed process 1287 (ospfd).
>
> Aug 22 22:03:43 xenu kernel: vlan0007: del 01:00:5e:00:00:06 mcast address
> from master interface
>
> Aug 22 22:03:43 xenu kernel: vlan0009: del 01:00:5e:00:00:05 mcast address
> from master interface
>
> Aug 22 22:03:43 xenu kernel: vlan0007: del 01:00:5e:00:00:05 mcast address
> from master interface
>
> Aug 22 22:04:55 xenu kernel: oom-killer: gfp_mask=0xd0
>
> Aug 22 22:04:55 xenu kernel: DMA per-cpu:
>
> Aug 22 22:04:55 xenu kernel: cpu 0 hot: low 32, high 96, batch 16
>
> Aug 22 22:04:55 xenu kernel: cpu 0 cold: low 0, high 32, batch 16
>
> Aug 22 22:04:55 xenu kernel: Normal per-cpu: empty
>
> Aug 22 22:04:55 xenu kernel: HighMem per-cpu: empty
>
> Aug 22 22:04:55 xenu kernel:
>
> Aug 22 22:04:55 xenu kernel: Free pages:        2000kB (0kB HighMem)
>
> Aug 22 22:04:55 xenu kernel: Active:2137 inactive:55782 dirty:0
> writeback:52203 unstable:3564 free:500 slab:3915 mapped:2136 pagetables:72
>
> Aug 22 22:04:55 xenu kernel: DMA free:2000kB min:2048kB low:2560kB
> high:3072kB active:8548kB inactive:223128kB present:262144kB
> pages_scanned:223163 all_unreclaimable? no
>
> Aug 22 22:04:55 xenu kernel: lowmem_reserve[]: 0 0 0
>
> Aug 22 22:04:55 xenu kernel: Normal free:0kB min:0kB low:0kB high:0kB
> active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
>
> Aug 22 22:04:55 xenu kernel: lowmem_reserve[]: 0 0 0
>
> Aug 22 22:04:55 xenu kernel: HighMem free:0kB min:128kB low:160kB
> high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0
> all_unreclaimable? no
>
> Aug 22 22:04:55 xenu kernel: lowmem_reserve[]: 0 0 0
>
> Aug 22 22:04:55 xenu kernel: DMA: 0*4kB 24*8kB 9*16kB 6*32kB 1*64kB 1*128kB
> 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2000kB
>
> Aug 22 22:04:55 xenu kernel: Normal: empty
>
> Aug 22 22:04:55 xenu kernel: HighMem: empty
>
> Aug 22 22:04:55 xenu kernel: Swap cache: add 0, delete 0, find 0/0, race
> 0+0
>
> Aug 22 22:04:55 xenu kernel: Free swap  = 0kB
>
> Aug 22 22:04:55 xenu kernel: Total swap = 0kB
>
> Aug 22 22:04:55 xenu kernel: Out of Memory: Killed process 1283 (zebra).
>
> Aug 22 22:04:55 xenu kernel: oom-killer: gfp_mask=0xd0
>
> Aug 22 22:04:55 xenu kernel: DMA per-cpu:
>
> Aug 22 22:04:55 xenu kernel: cpu 0 hot: low 32, high 96, batch 16
>
> Aug 22 22:04:55 xenu kernel: cpu 0 cold: low 0, high 32, batch 16
>
> Aug 22 22:04:55 xenu kernel: Normal per-cpu: empty
>
> Aug 22 22:04:55 xenu kernel: HighMem per-cpu: empty
>
> Aug 22 22:04:55 xenu kernel:
>
> Aug 22 22:04:55 xenu kernel: Free pages:        2320kB (0kB HighMem)
>
> Aug 22 22:04:55 xenu kernel: Active:2040 inactive:55767 dirty:0
> writeback:52067 unstable:3700 free:580 slab:3915 mapped:2039 pagetables:68
>
> Aug 22 22:04:55 xenu kernel: DMA free:2320kB min:2048kB low:2560kB
> high:3072kB active:8160kB inactive:223068kB present:262144kB
> pages_scanned:282123 all_unreclaimable? yes
>
> Aug 22 22:04:55 xenu kernel: lowmem_reserve[]: 0 0 0
>
> Aug 22 22:04:55 xenu kernel: Normal free:0kB min:0kB low:0kB high:0kB
> active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
>
> Aug 22 22:04:55 xenu kernel: lowmem_reserve[]: 0 0 0
>
> Aug 22 22:04:55 xenu kernel: HighMem free:0kB min:128kB low:160kB
> high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0
> all_unreclaimable? no
>
> Aug 22 22:04:55 xenu kernel: lowmem_reserve[]: 0 0 0
>
> Aug 22 22:04:55 xenu kernel: DMA: 34*4kB 43*8kB 11*16kB 6*32kB 1*64kB
> 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2320kB
>
> Aug 22 22:04:55 xenu kernel: Normal: empty
>
> Aug 22 22:04:55 xenu kernel: HighMem: empty
>
> Aug 22 22:04:55 xenu kernel: Swap cache: add 0, delete 0, find 0/0, race
> 0+0
>
> Aug 22 22:04:55 xenu kernel: Free swap  = 0kB
>
> Aug 22 22:04:55 xenu kernel: Total swap = 0kB
>
> Aug 22 22:04:55 xenu kernel: Out of Memory: Killed process 1142 (portmap).
>
> Aug 22 22:04:55 xenu kernel: oom-killer: gfp_mask=0x1d2
>
> Aug 22 22:04:55 xenu kernel: DMA per-cpu:
>
> Aug 22 22:05:07 xenu kernel: cpu 0 hot: low 32, high 96, batch 16
>
> Aug 22 22:05:07 xenu kernel: cpu 0 cold: low 0, high 32, batch 16
>
> Aug 22 22:05:07 xenu kernel: Normal per-cpu: empty
>
> Aug 22 22:05:07 xenu kernel: HighMem per-cpu: empty
>
> Aug 22 22:05:07 xenu kernel:
>
> Aug 22 22:05:07 xenu kernel: Free pages:        2256kB (0kB HighMem)
>
> Aug 22 22:05:07 xenu kernel: Active:2021 inactive:55790 dirty:0
> writeback:51611 unstable:4156 free:564 slab:3913 mapped:2021 pagetables:64
>
> Aug 22 22:05:07 xenu kernel: DMA free:2256kB min:2048kB low:2560kB
> high:3072kB active:8084kB inactive:223160kB present:262144kB
> pages_scanned:153229 all_unreclaimable? no
>
> Aug 22 22:05:07 xenu kernel: lowmem_reserve[]: 0 0 0
>
> Aug 22 22:05:07 xenu kernel: Normal free:0kB min:0kB low:0kB high:0kB
> active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
>
> Aug 22 22:05:07 xenu kernel: lowmem_reserve[]: 0 0 0
>
>
>
>
>
> Could someone help us to understand how this can be xen related ? We really
> have never see such things on any other server before !
>
>
>
> BR,
>
>
>
> --
>
> Sylvain COUTANT

-- 
Marcus

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users