|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] 0-order allocation failed
I don't know -- I've never seen Linux issue this particular message before. This is a 32Mb virtual domain, with 256Mb VD swap, and an NFS root filesystem. I don't have any real machines with only 32Mb RAM. ;-) I've duplicated it in two different nodes now, using the 02 Feb 1.2 build. I seem to be able to get one every few minutes by running "perl -e '$a="a"x100000000'" in a while loop on two guests at the same time. I was going to try to duplicate it in today's 1.2, but that's not booting -- I'll start another thread for that. Steve On Tue, Feb 10, 2004 at 08:51:42PM -0800, Kip Macy wrote: > > That is odd. What that means is that something tried to allocate a page > without passing in GFP_WAIT as one of the flags, and there were no pages > that could be freed in any of the zones. Does this not happen when > running 2.4.24 directly on the machine with the same amount of memory? > > > -Kip > > > On Tue, 10 Feb 2004 stevegt@xxxxxxxxxxxxx wrote: > > > Looks like yesterday's paging problems were an artifact of something on > > that particular dom0 disk image -- I haven't been able to reproduce it > > on other nodes, and after I re-imaged the same node (with the same > > image), the problem has gone away there too, so that rules out hardware. > > > > However, something else did pop up -- while trying to break things with > > "perl -e '$a="a"x100000000'", I got the following messages; do we care? > > This is in mainstream linux mm/page_alloc.c, and it's hard for me to > > tell from the code whether these are outright errors or whether they > > were recoverable. Does anyone know? > > > > DOM4: __alloc_pages: 0-order allocation failed (gfp=0xf0/0) > > DOM4: __alloc_pages: 0-order allocation failed (gfp=0xf0/0) > > DOM4: __alloc_pages: 0-order allocation failed (gfp=0xf0/0) > > DOM4: __alloc_pages: 0-order allocation failed (gfp=0xf0/0) > > > > This was still on last Monday's version of 1.2; I'll see if I can > > reproduce it in today's 1.2; I'm deploying that later tonight. > > > > Steve > > > > > > On Mon, Feb 09, 2004 at 10:34:34PM -0800, wrote: > > > Before anyone burns too much time on this, hang on -- I wasn't able to > > > duplicate the problem on another cluster node (both nodes were built > > > from the same SystemImager image). I'm looking for the reason why, and > > > will let you know as soon as I do. > > > > > > Steve > > > > > > On Mon, Feb 09, 2004 at 08:13:22PM -0800, wrote: > > > > Okay, the problem still exists when I bump the memory up to 256Mb, and > > > > never swap. I.E. I've found no workaround. Hasn't anyone else hit > > > > anything like this? > > > > > > > > Steve > > > > > > > > > > > > DOM3: xen_console_init > > > > DOM3: Linux version 2.4.24-xeno (stevegt@pathfinder) (gcc version > > > > 3.0.4) #16 Mon Feb 2 17:46:41 PST 2004 > > > > DOM3: On node 0 totalpages: 65536 > > > > DOM3: zone(0): 4096 pages. > > > > DOM3: zone(1): 61440 pages. > > > > DOM3: zone(2): 0 pages. > > > > DOM3: Kernel command line: > > > > ip=64.71.149.20:10.27.2.50:64.71.149.1:255.255.255.0::eth0:off > > > > root=/dev/nfs nfsroot=/export//xen/fs/stevegt/tcx/root 4 DOMID=20 > > > > DOM3: Initializing CPU#0 > > > > DOM3: Xen reported: 398.780 MHz processor. > > > > DOM3: Calibrating delay loop... 1592.52 BogoMIPS > > > > DOM3: Memory: 257132k/262144k available (1078k kernel code, 5012k > > > > reserved, 308k data, 52k init, 0k highmem) > > > > DOM3: Dentry cache hash table entries: 32768 (order: 6, 262144 bytes) > > > > DOM3: Inode cache hash table entries: 16384 (order: 5, 131072 bytes) > > > > DOM3: Mount cache hash table entries: 512 (order: 0, 4096 bytes) > > > > DOM3: Buffer cache hash table entries: 16384 (order: 4, 65536 bytes) > > > > DOM3: Page-cache hash table entries: 65536 (order: 6, 262144 bytes) > > > > DOM3: CPU: L1 I cache: 16K, L1 D cache: 16K > > > > DOM3: CPU: L2 cache: 512K > > > > DOM3: CPU: Intel Pentium II (Deschutes) stepping 01 > > > > DOM3: POSIX conformance testing by UNIFIX > > > > DOM3: Linux NET4.0 for Linux 2.4 > > > > DOM3: Based upon Swansea University Computer Society NET3.039 > > > > DOM3: Initializing RT netlink socket > > > > DOM3: Starting kswapd > > > > DOM3: Journalled Block Device driver loaded > > > > DOM3: Installing knfsd (copyright (C) 1996 okir@xxxxxxxxxxxx). > > > > DOM3: Xeno console successfully installed > > > > DOM3: Starting Xeno Balloon driver > > > > DOM3: pty: 256 Unix98 ptys configured > > > > DOM3: RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 > > > > blocksize > > > > DOM3: loop: loaded (max 8 devices) > > > > DOM3: NET4: Linux TCP/IP 1.0 for NET4.0 > > > > DOM3: IP Protocols: ICMP, UDP, TCP > > > > DOM3: IP: routing cache hash table of 2048 buckets, 16Kbytes > > > > DOM3: TCP: Hash tables configured (established 16384 bind 16384) > > > > DOM3: IP-Config: Complete: > > > > DOM3: device=eth0, addr=64.71.149.20, mask=255.255.255.0, > > > > gw=64.71.149.1, > > > > DOM3: host=64.71.149.20, domain=, nis-domain=(none), > > > > DOM3: bootserver=10.27.2.50, rootserver=10.27.2.50, rootpath= > > > > DOM3: ip_conntrack version 2.1 (2048 buckets, 16384 max) - 292 bytes > > > > per conntrack > > > > DOM3: ip_tables: (C) 2000-2002 Netfilter core team > > > > DOM3: NET4: Unix domain sockets 1.0/SMP for Linux NET4.0. > > > > DOM3: Looking up port of RPC 100003/2 on 10.27.2.50 > > > > DOM3: Looking up port of RPC 100005/1 on 10.27.2.50 > > > > DOM3: VFS: Mounted root (nfs filesystem). > > > > DOM3: Freeing unused kernel memory: 52k freed > > > > DOM3: INIT: version 2.84 booting > > > > DOM3: Activating swap. > > > > DOM3: Adding Swap: 262136k swap-space (priority -1) > > > > DOM3: Checking root file system... > > > > DOM3: fsck 1.27 (8-Mar-2002) > > > > DOM3: 10.27.2.50:/export/xen/fs/stevegt/tcx: NFS file system. > > > > DOM3: System time was Tue Feb 10 02:14:36 UTC 2004. > > > > DOM3: Setting the System Clock using the Hardware Clock as > > > > reference... > > > > DOM3: modprobe: modprobe: Can't locate module char-major-10-135 > > > > DOM3: modprobe: modprobe: Can't locate module char-major-4 > > > > DOM3: hwclock is unable to get I/O port access: the iopl(3) call > > > > failed. > > > > DOM3: modprobe: modprobe: Can't locate module char-major-10-135 > > > > DOM3: modprobe: modprobe: Can't locate module char-major-4 > > > > DOM3: System Clock set. System local time is now Tue Feb 10 02:14:36 > > > > UTC 2004. > > > > DOM3: Calculating module dependencies... depmod: cannot read ELF > > > > header from /lib/modules/2.4.24-xeno/modules.dep > > > > DOM3: depmod: cannot read ELF header from > > > > /lib/modules/2.4.24-xeno/modules.generic_string > > > > DOM3: depmod: /lib/modules/2.4.24-xeno/modules.ieee1394map is not an > > > > ELF file > > > > DOM3: depmod: /lib/modules/2.4.24-xeno/modules.isapnpmap is not an ELF > > > > file > > > > DOM3: depmod: cannot read ELF header from > > > > /lib/modules/2.4.24-xeno/modules.parportmap > > > > DOM3: depmod: /lib/modules/2.4.24-xeno/modules.pcimap is not an ELF > > > > file > > > > DOM3: depmod: cannot read ELF header from > > > > /lib/modules/2.4.24-xeno/modules.pnpbiosmap > > > > DOM3: depmod: /lib/modules/2.4.24-xeno/modules.usbmap is not an ELF > > > > file > > > > DOM3: done. > > > > DOM3: Loading modules: > > > > DOM3: Checking all file systems... > > > > DOM3: fsck 1.27 (8-Mar-2002) > > > > DOM3: Setting kernel variables. > > > > DOM3: Loading the saved-state of the serial devices... > > > > DOM3: Mounting local filesystems... > > > > DOM3: nothing was mounted > > > > DOM3: Running 0dns-down to make sure resolv.conf is ok...done. > > > > DOM3: Cleaning: /etc/network/ifstate. > > > > DOM3: Setting up IP spoofing protection: rp_filter. > > > > DOM3: Configuring network interfaces: done. > > > > DOM3: Mounting remote filesystems... > > > > DOM3: > > > > DOM3: Setting the System Clock using the Hardware Clock as > > > > reference... > > > > DOM3: System Clock set. Local time: Tue Feb 10 02:14:37 UTC 2004 > > > > DOM3: > > > > DOM3: Cleaning: /tmp /var/lock /var/run. > > > > DOM3: Initializing random number generator... done. > > > > DOM3: Recovering nvi editor sessions... done. > > > > DOM3: INIT: Entering runlevel: 4 > > > > DOM3: Starting system log daemon: syslogd. > > > > DOM3: Starting kernel log daemon: klogd. > > > > DOM3: Starting internet superserver: inetd. > > > > DOM3: Starting PCMCIA services: module directory > > > > /lib/modules/2.4.24-xeno/pcmcia not found. > > > > DOM3: Starting OpenBSD Secure Shell server: sshd. > > > > DOM3: Starting deferred execution scheduler: atd. > > > > DOM3: Starting periodic command scheduler: cron. > > > > DOM3: INIT: no more processes left in this runlevel > > > > DOM3: Unable to handle kernel paging request at virtual address > > > > 20000001 > > > > DOM3: printing eip: > > > > DOM3: c0007743 > > > > DOM3: *pde=00000000(00000000) > > > > DOM3: Oops: 0000 > > > > DOM3: CPU: 0 > > > > DOM3: EIP: 0819:[<c0007743>] Not tainted > > > > DOM3: EFLAGS: 00010202 > > > > DOM3: eax: 00000001 ebx: 20000001 ecx: c3ebde6c edx: c3ebde6c > > > > DOM3: esi: c3ebc000 edi: c0114254 ebp: c46c1060 esp: c3ebdce4 > > > > DOM3: ds: 0821 es: 0821 ss: 0821 > > > > DOM3: Process cc (pid: 13793, stackpage=c3ebd000)<1> > > > > DOM3: Stack: 20000001 c3ebc000 c002c728 c3ebc000 c3ebddb0 c0114254 > > > > ffffffb0 c3ebc000 > > > > DOM3: c003e789 c3ebde6c c014b5ac c003e314 c3ebde6c 00000000 > > > > c0114250 00000000 > > > > DOM3: 00000000 00000000 01082003 8d588810 00000000 00000000 > > > > 00000000 00000000 > > > > DOM3: Call Trace: [<c002c728>] [<c003e789>] [<c003e314>] [<c002ccc7>] > > > > [<c002cf4a>] > > > > DOM3: [<c002cf61>] [<c0090033>] [<c00914bf>] > > > > DOM3: > > > > DOM3: <1>Unable to handle kernel paging request at virtual address > > > > 20000001 > > > > DOM3: printing eip: > > > > DOM3: c000af0f > > > > DOM3: *pde=00000000(00000000) > > > > DOM3: Oops: 0002 > > > > DOM3: CPU: 0 > > > > DOM3: EIP: 0819:[<c000af0f>] Not tainted > > > > DOM3: EFLAGS: 00010282 > > > > DOM3: eax: 20000001 ebx: c485c0a0 ecx: c3ebc264 edx: c3ebc264 > > > > DOM3: esi: 00000000 edi: 20000001 ebp: 0000000b esp: c3ebdbb4 > > > > DOM3: ds: 0821 es: 0821 ss: 0821 > > > > DOM3: Process cc (pid: 13793, stackpage=c3ebd000)<1> > > > > DOM3: Stack: c485c0a0 00000000 c3ebc000 0000000b 0000000b c000b55f > > > > 20000001 0000001f > > > > DOM3: 00000000 cf4227e0 20000001 c0091a87 0000000b 00000000 > > > > c485c0bc c0096305 > > > > DOM3: c0129928 c3ebdcb0 00000000 c3ebc000 00000000 20000001 > > > > c46c1060 00000000 > > > > DOM3: Call Trace: [<c000b55f>] [<c0091a87>] [<c0096305>] [<c002eb19>] > > > > [<c0018a25>] > > > > DOM3: [<c0018c46>] [<c0018feb>] [<c0018ed4>] [<c006e759>] > > > > [<c0091768>] [<c0007743>] > > > > DOM3: [<c002c728>] [<c003e789>] [<c003e314>] [<c002ccc7>] > > > > [<c002cf4a>] [<c002cf61>] > > > > DOM3: [<c0090033>] [<c00914bf>] > > > > DOM3: > > > > DOM3: <1>Unable to handle kernel NULL pointer dereference at virtual > > > > address 00000001 > > > > DOM3: printing eip: > > > > DOM3: c000b623 > > > > DOM3: *pde=00000000(00000000) > > > > DOM3: Oops: 0002 > > > > DOM3: CPU: 0 > > > > DOM3: EIP: 0819:[<c000b623>] Not tainted > > > > DOM3: EFLAGS: 00010202 > > > > DOM3: eax: 00000000 ebx: 00000001 ecx: c3ebc264 edx: c3ebc264 > > > > DOM3: esi: 00000002 edi: c3ebc000 ebp: 0000000b esp: c3ebdaa0 > > > > DOM3: ds: 0821 es: 0821 ss: 0821 > > > > DOM3: Process cc (pid: 13793, stackpage=c3ebd000)<1> > > > > DOM3: Stack: 0000001f 00000002 20000001 20000001 c0091a87 0000000b > > > > 00000000 00000002 > > > > DOM3: c0096305 c0129928 c3ebdb80 00000002 c3ebc000 00000002 > > > > 20000001 0000000b > > > > DOM3: 63303039 ffffffff c3ebc000 00000002 38383130 38643538 > > > > 00030001 64303030 > > > > DOM3: Call Trace: [<c0091a87>] [<c0096305>] [<c0008996>] [<c000f797>] > > > > [<c000f991>] > > > > DOM3: [<c0091768>] [<c000af0f>] [<c000b55f>] [<c0091a87>] > > > > [<c0096305>] [<c002eb19>] > > > > DOM3: [<c0018a25>] [<c0018c46>] [<c0018feb>] [<c0018ed4>] > > > > [<c006e759>] [<c0091768>] > > > > DOM3: [<c0007743>] [<c002c728>] [<c003e789>] [<c003e314>] > > > > [<c002ccc7>] [<c002cf4a>] > > > > DOM3: [<c002cf61>] [<c0090033>] [<c00914bf>] > > > > DOM3: > > > > > > > > > > > > > > > > > > > > On Mon, Feb 09, 2004 at 05:25:00PM -0800, wrote: > > > > > Hi All, > > > > > > > > > > I seem to be able to reproduce a null pointer dereference and paging > > > > > request errors in 1.2. Can anyone give me any pointers on tracking > > > > > down > > > > > what is causing it? > > > > > > > > > > This is with a 32Mb virtual domain, running debian woody, NFS root, > > > > > 256Mb swap in a local VD, while running a process which builds > > > > > openldap, > > > > > python2.2.3, and related packages. I'm not sure which package, if any > > > > > in particular, is causing this; could be just anything that causes a > > > > > similar workload. This particular set of messages appeared before the > > > > > virtual domain locked up during the openldap build... > > > > > > > > > > Steve > > > > > > > > > > > > > > > DOM26: Unable to handle kernel paging request at virtual address > > > > > 20000001 > > > > > DOM26: printing eip: > > > > > DOM26: c0007743 > > > > > DOM26: *pde=00000000(00000000) > > > > > DOM26: Oops: 0000 > > > > > DOM26: CPU: 0 > > > > > DOM26: EIP: 0819:[<c0007743>] Not tainted > > > > > DOM26: EFLAGS: 00010202 > > > > > DOM26: eax: 00000001 ebx: 20000001 ecx: c0a79e6c edx: c0a79e6c > > > > > DOM26: esi: c0a78000 edi: c0114254 ebp: c1e5f580 esp: c0a79ce4 > > > > > DOM26: ds: 0821 es: 0821 ss: 0821 > > > > > DOM26: Process sh (pid: 10086, stackpage=c0a79000)<1> > > > > > DOM26: Stack: 20000001 c0a78000 c002c728 c0a78000 c0a79db0 c0114254 > > > > > ffffffb0 c0a78000 > > > > > DOM26: c003e789 c0a79e6c c014b5ac c003e314 c0a79e6c 00000000 > > > > > c0114250 c0a79de8 > > > > > DOM26: c1419640 80000000 00000000 00000000 00000000 00000000 > > > > > 00000000 00000000 > > > > > DOM26: Call Trace: [<c002c728>] [<c003e789>] [<c003e314>] [<c002ccc7>] > > > > > [<c002cf4a>] > > > > > DOM26: [<c002cf61>] [<c0090033>] [<c00914bf>] > > > > > DOM26: > > > > > DOM26: <1>Unable to handle kernel paging request at virtual address > > > > > 20000001 > > > > > DOM26: printing eip: > > > > > DOM26: c000af0f > > > > > DOM26: *pde=00000000(00000000) > > > > > DOM26: Oops: 0002 > > > > > DOM26: CPU: 0 > > > > > DOM26: EIP: 0819:[<c000af0f>] Not tainted > > > > > DOM26: EFLAGS: 00010282 > > > > > DOM26: eax: 20000001 ebx: c1ed5b20 ecx: c0a78264 edx: c0a78264 > > > > > DOM26: esi: 00000000 edi: 20000001 ebp: 0000000b esp: c0a79bb4 > > > > > DOM26: ds: 0821 es: 0821 ss: 0821 > > > > > DOM26: Process sh (pid: 10086, stackpage=c0a79000)<1> > > > > > DOM26: Stack: c1ed5b20 00000000 c0a78000 0000000b 0000000b c000b55f > > > > > 20000001 0000001f > > > > > DOM26: 00000000 c140d6c0 20000001 c0091a87 0000000b 00000000 > > > > > c1ed5b3c c0096305 > > > > > DOM26: c0129928 c0a79cb0 00000000 c0a78000 00000000 20000001 > > > > > c1e5f580 00000000 > > > > > DOM26: Call Trace: [<c000b55f>] [<c0091a87>] [<c0096305>] [<c002eb19>] > > > > > [<c0018a25>] > > > > > DOM26: [<c0018c46>] [<c0018feb>] [<c0018ed4>] [<c006e759>] > > > > > [<c0091768>] [<c0007743>] > > > > > DOM26: [<c002c728>] [<c003e789>] [<c003e314>] [<c002ccc7>] > > > > > [<c002cf4a>] [<c002cf61>] > > > > > DOM26: [<c0090033>] [<c00914bf>] > > > > > DOM26: > > > > > DOM26: <1>Unable to handle kernel NULL pointer dereference at virtual > > > > > address 00000001 > > > > > DOM26: printing eip: > > > > > DOM26: c000b623 > > > > > DOM26: *pde=00000000(00000000) > > > > > DOM26: Oops: 0002 > > > > > DOM26: CPU: 0 > > > > > DOM26: EIP: 0819:[<c000b623>] Not tainted > > > > > DOM26: EFLAGS: 00010202 > > > > > DOM26: eax: 00000000 ebx: 00000001 ecx: c0a78264 edx: c0a78264 > > > > > DOM26: esi: 00000002 edi: c0a78000 ebp: 0000000b esp: c0a79aa0 > > > > > DOM26: ds: 0821 es: 0821 ss: 0821 > > > > > DOM26: Process sh (pid: 10086, stackpage=c0a79000)<1> > > > > > DOM26: Stack: 0000001f 00000002 20000001 20000001 c0091a87 0000000b > > > > > 00000000 00000002 > > > > > DOM26: c0096305 c0129928 c0a79b80 00000002 c0a78000 00000002 > > > > > 20000001 0000000b > > > > > DOM26: 63303039 c101fc58 c0a78000 00000002 c101fc58 ffffffff > > > > > 00030001 c001e621 > > > > > DOM26: Call Trace: [<c0091a87>] [<c0096305>] [<c001e621>] [<c001f6c0>] > > > > > [<c0008996>] > > > > > DOM26: [<c00200d7>] [<c00204e1>] [<c001464d>] [<c0014c92>] > > > > > [<c0091768>] [<c000af0f>] > > > > > DOM26: [<c000b55f>] [<c0091a87>] [<c0096305>] [<c002eb19>] > > > > > [<c0018a25>] [<c0018c46>] > > > > > DOM26: [<c0018feb>] [<c0018ed4>] [<c006e759>] [<c0091768>] > > > > > [<c0007743>] [<c002c728>] > > > > > DOM26: [<c003e789>] [<c003e314>] [<c002ccc7>] [<c002cf4a>] > > > > > [<c002cf61>] [<c0090033>] > > > > > DOM26: [<c00914bf>] > > > > > DOM26: > > > > > > > > > > > > > > > > > > > > -- > > > > > Stephen G. Traugott (KG6HDQ) > > > > > UNIX/Linux Infrastructure Architect, TerraLuna LLC > > > > > stevegt@xxxxxxxxxxxxx > > > > > http://www.stevegt.com -- http://Infrastructures.Org > > > > > > > > -- > > > > Stephen G. Traugott (KG6HDQ) > > > > UNIX/Linux Infrastructure Architect, TerraLuna LLC > > > > stevegt@xxxxxxxxxxxxx > > > > http://www.stevegt.com -- http://Infrastructures.Org > > > > > > -- > > > Stephen G. Traugott (KG6HDQ) > > > UNIX/Linux Infrastructure Architect, TerraLuna LLC > > > stevegt@xxxxxxxxxxxxx > > > http://www.stevegt.com -- http://Infrastructures.Org > > > > -- > > Stephen G. Traugott (KG6HDQ) > > UNIX/Linux Infrastructure Architect, TerraLuna LLC > > stevegt@xxxxxxxxxxxxx > > http://www.stevegt.com -- http://Infrastructures.Org > > > > > > ------------------------------------------------------- > > The SF.Net email is sponsored by EclipseCon 2004 > > Premiere Conference on Open Tools Development and Integration > > See the breadth of Eclipse activity. February 3-5 in Anaheim, CA. > > http://www.eclipsecon.org/osdn > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@xxxxxxxxxxxxxxxxxxxxx > > https://lists.sourceforge.net/lists/listinfo/xen-devel > > > -- Stephen G. Traugott (KG6HDQ) UNIX/Linux Infrastructure Architect, TerraLuna LLC stevegt@xxxxxxxxxxxxx http://www.stevegt.com -- http://Infrastructures.Org ------------------------------------------------------- The SF.Net email is sponsored by EclipseCon 2004 Premiere Conference on Open Tools Development and Integration See the breadth of Eclipse activity. February 3-5 in Anaheim, CA. http://www.eclipsecon.org/osdn _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxxxxxxxxxx https://lists.sourceforge.net/lists/listinfo/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |