Re: [Xen-devel] [Xen-users] xend (12, 'Cannot allocate memory')

[Kai Meyer <kai@xxxxxxxxxxxxx>]

Well, that's no so great. It thinks I have barely a fraction of what I'm 
supposed to have available. Free shows 10GB available, xm info shows 1 
MB available. If I stop a virtual machine, the memory in xminfo 
increases by the amount of memory used by that xen machine, which is to 
be expected.
Is there a way to force Xen to refresh that number, or let me force an 
override?
-Kai Meyer

Keir Fraser wrote:
> Given you are running dom0 with plenty of swap, it's most likely that Xen
> has run out of memory to allocate to your new domU, rather than the
> allocation failure coming from within dom0. Probably auto-ballooning is
> screwing up somehow (that's the bit of xend that gives away memory back to
> Xen to make sure it has enough memory to create your new domU).
>
> Sorry I don't have a precise diagnosis for you, but I think looking at Xen's
> allocator is at least the right direction for you to take. 'xm info' tells
> you how much memory Xen thinks it has.
>
>  -- Keir
>
> On 26/3/08 14:54, "Kai Meyer" <kai@xxxxxxxxxxxxx> wrote:
>
>   
> > I'm getting the error:
> > xend (12, 'Cannot allocate memory')
> >  From the command line after the Dom-0 goes through a period of heavy
> > memory usage. I don't have any hard numbers on how to duplicate it, but
> > I'll give you the steps I've gone through to duplicate the problem.
> >
> > First, some information about my setup. It's a new setup, and has been
> > in heavy development for the past few months. I'm new to Xen as of this
> > project we started in January. As far as I can tell, this problem has
> > existed since we started, and may have even happened before but we were
> > too new to think it wasn't our fault (which it may still be....)  I've
> > found the same error addressed here:
> > http://archive.netbsd.se/?ml=xen-devel&a=2007-04&t=3887921
> > The 'hack' to fix this problem is already built into my
> > XendDomainInfo.py script already, so I'm a little more than lost.
> >
> > [root@xen1 ~]# xm info
> > host                   : xen1.fiber.net
> > release                : 2.6.18-53.1.13.el5xen
> > version                : #1 SMP Tue Feb 12 13:33:07 EST 2008
> > machine                : x86_64
> > nr_cpus                : 8
> > nr_nodes               : 1
> > sockets_per_node       : 2
> > cores_per_socket       : 4
> > threads_per_core       : 1
> > cpu_mhz                : 1866
> > hw_caps                :
> > bfebfbff:20100800:00000000:00000140:0004e3bd:00000000:00000001
> > total_memory           : 16382
> > free_memory            : 2
> > xen_major              : 3
> > xen_minor              : 1
> > xen_extra              : .0-53.1.13.el5
> > xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> > hvm-3.0-x86_32p hvm-3.0-x86_64
> > xen_pagesize           : 4096
> > platform_params        : virt_start=0xffff800000000000
> > xen_changeset          : unavailable
> > cc_compiler            : gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)
> > cc_compile_by          : mockbuild
> > cc_compile_domain      :
> > cc_compile_date        : Tue Feb 12 12:55:35 EST 2008
> > xend_config_format     : 2
> >
> >
> > Here's the error in action:
> > [root@xen1 centos-5.1_core_image]# xm create centos-5.1_core_image
> > Using config file "./centos-5.1_core_image".
> > Error: (12, 'Cannot allocate memory')
> > [root@xen1 centos-5.1_core_image]#
> >
> >
> > Here is the output from xend.log that occurs when I get this error:
> > [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
> > (XendDomainInfo:200) XendDomainInfo.create(['vm', ['name',
> > 'centos-5.1_core_image'], ['memory', 4024], ['maxmem', 4024],
> > ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash',
> > 'restart'], ['vcpus', 6], ['image', ['hvm', ['kernel',
> > '/usr/lib/xen/boot/hvmloader'], ['device_model',
> > '/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot',
> > 'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority',
> > '/root/.Xauthority'], ['keymap', 'en-us']]], ['device', ['vbd',
> > ['uname', 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'],
> > ['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname',
> > 'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'],
> > ['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr1'], ['mac',
> > '00:16:3e:4c:c0:01'], ['type', 'ioemu']]]])
> > [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
> > (XendDomainInfo:306) parseConfig: config is ['vm', ['name',
> > 'centos-5.1_core_image'], ['memory', 4024], ['maxmem', 4024],
> > ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash',
> > 'restart'], ['vcpus', 6], ['image', ['hvm', ['kernel',
> > '/usr/lib/xen/boot/hvmloader'], ['device_model',
> > '/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot',
> > 'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority',
> > '/root/.Xauthority'], ['keymap', 'en-us']]], ['device', ['vbd',
> > ['uname', 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'],
> > ['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname',
> > 'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'],
> > ['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr1'], ['mac',
> > '00:16:3e:4c:c0:01'], ['type', 'ioemu']]]]
> > [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
> > (XendDomainInfo:411) parseConfig: result is {'shadow_memory': None,
> > 'start_time': None, 'uuid': None, 'on_crash': 'restart', 'on_reboot':
> > 'restart', 'localtime': None, 'image': ['hvm', ['kernel',
> > '/usr/lib/xen/boot/hvmloader'], ['device_model',
> > '/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot',
> > 'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority',
> > '/root/.Xauthority'], ['keymap', 'en-us']], 'on_poweroff': 'destroy',
> > 'bootloader_args': None, 'cpus': None, 'name': 'centos-5.1_core_image',
> > 'backend': [], 'vcpus': 6, 'cpu_weight': None, 'features': None,
> > 'vcpu_avail': None, 'memory': 4024, 'device': [('vbd', ['vbd', ['uname',
> > 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'], ['dev',
> > 'hda'], ['mode', 'w']]), ('vbd', ['vbd', ['uname',
> > 'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'],
> > ['mode', 'w']]), ('vif', ['vif', ['bridge', 'xenbr1'], ['mac',
> > '00:16:3e:4c:c0:01'], ['type', 'ioemu']])], 'bootloader': None, 'cpu':
> > None, 'maxmem': 4024}
> > [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
> > (XendDomainInfo:1349) XendDomainInfo.construct: None
> > [2008-03-19 14:07:43 xend 3069] DEBUG (balloon:127) Balloon: 2276 KiB
> > free; need 2048; done.
> > [2008-03-19 14:07:43 xend.XendDomainInfo 3069] ERROR
> > (XendDomainInfo:212) Domain construction failed
> > Traceback (most recent call last):
> >  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
> > line 204, in create
> >    vm.construct()
> >  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
> > line 1369, in construct
> >    hvm = hvm)
> > Error: (12, 'Cannot allocate memory')
> >
> >
> > Here's the configuration file being used:
> > name = "centos-5.1_core_image"
> > maxmem = 4024
> > memory = 4024
> > vcpus = 6
> > builder = "hvm"
> > kernel = "/usr/lib/xen/boot/hvmloader"
> > boot = "cda"
> > pae = 1
> > acpi = 0
> > apic = 0
> > on_poweroff = "destroy"
> > on_reboot = "restart"
> > on_crash = "restart"
> > device_model = "/usr/lib64/xen/bin/qemu-dm"
> > sdl = 0
> > vnc = 1
> > vncunused = 1
> > keymap = "en-us"
> > disk = [ 'file:/xen/domains2/centos-5.1_core_image/disk.img,hda,w' ,
> > 'file:/xen/domains2/centos-5.1_core_image/swap.img,hdb,w' ]
> > vif = [ "mac=00:16:3e:4c:c0:01,bridge=xenbr1,type=ioemu" ]
> > serial = "pty"
> >
> > At the time the error occured, my memory looked like this:
> > [root@xen1 centos-5.1_core_image]# free
> >             total       used       free     shared    buffers     cached
> > Mem:      15845376   15821080      24296          0     113596   15081468
> > -/+ buffers/cache:     626016   15219360
> > Swap:      4192956          4    4192952
> >
> > The only way I've been able to duplicate this error is by making
> > multiple copies of the 6GB disk images (created with dd, so ls shows 6GB
> > but du shows 2GB) for different virtual machines and booting them up. I
> > can get up to about 8 new virtual machines roughly, before I start
> > seeing my memory error. If you're familiar with IBM's 'xen_deploy.pl'
> > script, I've modified it slightly to meet our needs, and launch our
> > systems with that tool. It's basic functionality is to read in a xen
> > configuration file you want to use as a template, and output a modified
> > one based on arguments you pass to the scripts, and then create a floppy
> > disk image and copy the image from the template config file, and then
> > boot the new system.
> >
> > I've read in previous emails that this error occurs because of a bug in
> > the hvmloader, and look for an update. That email response was for a 3.0
> > version of xen, and I'm on 3.1.
> >
> > What other information can I provide to help diagnose this problem?
> >
> > -Kai Meyer
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxxxxxxxx
> > http://lists.xensource.com/xen-devel
> >     
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
>   

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel