|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-bugs] [Bug 1027] New: very slow domU creation
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1027 Summary: very slow domU creation Product: Xen Version: unstable Platform: x86-64 OS/Version: Linux-2.6 Status: NEW Severity: critical Priority: P2 Component: Tools AssignedTo: xen-bugs@xxxxxxxxxxxxxxxxxxx ReportedBy: tejasvia@xxxxxxxxx First a little information about my setup: I have a RHEL4 AMD64 based box that I use to run Xen VM's. I was previously running the pre-3.1 unstable sources (2.6.16 kernel) and things seemed to be working fine upto a point where I could start as many as 60 VM's on this box that has 8GB of physical mem. I used to use the blktap driver for exporting the filesystem images (thanks to the fix of http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=743) I recently synced up to the latest unstable sources having the 2.6.18 kernel. Apart from issues with the blktap driver (which I shall log a separate bug for), I am seeing very long domU create times. (in the order of 4 minutes per domU). I see xenstored & the hotplug scripts chewing a lot of CPU. SInce blktap appears broken, I am using the loop driver to load images. (but will eventually need to shift to blktap to start 60 VM's) Given that I start ~60 VM's, this long startup time is going to make my wait for VM's to start very long. Can somebody please let me know if this is a known issue or a necessary side-effect of some other change ? The following is a sample log of an "xm create" of the first domU on the machine: ========= 2007-07-16 15:26:43 3842] DEBUG (XendDomainInfo:1430) XendDomainInfo.constructDomain [2007-07-16 15:26:43 3842] DEBUG (balloon:114) Balloon: 307952 KiB free; need 2048; done. [2007-07-16 15:26:43 3842] DEBUG (XendDomain:434) Adding Domain: 13 [2007-07-16 15:26:43 3842] DEBUG (XendDomainInfo:1493) XendDomainInfo.initDomain: 13 256 [2007-07-16 15:26:43 3842] DEBUG (XendDomainInfo:1524) _initDomain:shadow_memory=0x0, memory_static_max=0x640 0000, memory_static_min=0x0. [2007-07-16 15:26:43 3842] DEBUG (balloon:114) Balloon: 307944 KiB free; need 102400; done. [2007-07-16 15:26:43 3842] INFO (image:128) buildDomain os=linux dom=13 vcpus=2 [2007-07-16 15:26:43 3842] DEBUG (image:197) domid = 13 [2007-07-16 15:26:43 3842] DEBUG (image:198) memsize = 100 [2007-07-16 15:26:43 3842] DEBUG (image:199) image = /usr0/VM/vmlinuz-2.6.16.13-xen [2007-07-16 15:26:43 3842] DEBUG (image:200) store_evtchn = 1 [2007-07-16 15:26:43 3842] DEBUG (image:201) console_evtchn = 2 [2007-07-16 15:26:43 3842] DEBUG (image:202) cmdline = root=/dev/hda1 ro selinux=0 3 [2007-07-16 15:26:43 3842] DEBUG (image:203) ramdisk = [2007-07-16 15:26:43 3842] DEBUG (image:204) vcpus = 2 [2007-07-16 15:26:43 3842] DEBUG (image:205) features = [2007-07-16 15:26:43 3842] INFO (XendDomainInfo:1333) createDevice: vbd : {'uuid': '5c5018d8-8cd1-a10d-427c-1 c1f97ab1613', 'bootable': 1, 'driver': 'paravirtualised', 'dev': 'hda1', 'uname': 'file:/usr0/VM/compute-1-8- vm30/compute-1-8-vm30-fs.img', 'mode': 'w'} [2007-07-16 15:26:43 3842] DEBUG (DevController:115) DevController: writing {'virtual-device': '769', 'device -type': 'disk', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backen d/vbd/13/769'} to /local/domain/13/device/vbd/769. [2007-07-16 15:26:43 3842] DEBUG (DevController:117) DevController: writing {'domain': 'compute-1-8-vm30', 'f rontend': '/local/domain/13/device/vbd/769', 'uuid': '5c5018d8-8cd1-a10d-427c-1c1f97ab1613', 'dev': 'hda1', ' state': '1', 'params': '/usr0/VM/compute-1-8-vm30/compute-1-8-vm30-fs.img', 'mode': 'w', 'online': '1', 'fron tend-id': '13', 'type': 'file'} to /local/domain/0/backend/vbd/13/769. [2007-07-16 15:26:43 3842] INFO (XendDomainInfo:1333) createDevice: vbd : {'uuid': '004530ee-279f-ca09-29ce-3 6e95e29f62c', 'bootable': 0, 'driver': 'paravirtualised', 'dev': 'hda2', 'uname': 'file:/usr0/VM/compute-1-8- vm30/compute-1-8-vm30-swap.img', 'mode': 'w'} [2007-07-16 15:26:43 3842] DEBUG (DevController:115) DevController: writing {'virtual-device': '770', 'device -type': 'disk', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backen d/vbd/13/770'} to /local/domain/13/device/vbd/770. [2007-07-16 15:26:43 3842] DEBUG (DevController:117) DevController: writing {'domain': 'compute-1-8-vm30', 'f rontend': '/local/domain/13/device/vbd/770', 'uuid': '004530ee-279f-ca09-29ce-36e95e29f62c', 'dev': 'hda2', ' state': '1', 'params': '/usr0/VM/compute-1-8-vm30/compute-1-8-vm30-swap.img', 'mode': 'w', 'online': '1', 'fr ontend-id': '13', 'type': 'file'} to /local/domain/0/backend/vbd/13/770. [2007-07-16 15:26:43 3842] INFO (XendDomainInfo:1333) createDevice: vbd : {'uuid': '24ff2f58-1b1f-2718-bf39-a 0eaa07d76a8', 'bootable': 0, 'driver': 'paravirtualised', 'dev': 'hda3', 'uname': 'file:/usr0/VM/nonw.img', ' mode': 'r'} [2007-07-16 15:26:43 3842] DEBUG (DevController:115) DevController: writing {'virtual-device': '771', 'device -type': 'disk', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backen d/vbd/13/771'} to /local/domain/13/device/vbd/771. [2007-07-16 15:26:43 3842] DEBUG (DevController:117) DevController: writing {'domain': 'compute-1-8-vm30', 'f rontend': '/local/domain/13/device/vbd/771', 'uuid': '24ff2f58-1b1f-2718-bf39-a0eaa07d76a8', 'dev': 'hda3', ' state': '1', 'params': '/usr0/VM/nonw.img', 'mode': 'r', 'online': '1', 'frontend-id': '13', 'type': 'file'} to /local/domain/0/backend/vbd/13/771. [2007-07-16 15:26:43 3842] INFO (XendDomainInfo:1333) createDevice: vif : {'mac': '00:16:3e:37:29:82', 'uuid' : '24120c27-5db4-a5fb-264b-2fba7a79b836'} [2007-07-16 15:26:50 3842] DEBUG (DevController:115) DevController: writing {'mac': '00:16:3e:37:29:82', 'han dle': '0', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif /13/0'} to /local/domain/13/device/vif/0. [2007-07-16 15:26:50 3842] DEBUG (DevController:117) DevController: writing {'domain': 'compute-1-8-vm30', 'h andle': '0', 'uuid': '24120c27-5db4-a5fb-264b-2fba7a79b836', 'script': '/etc/xen/scripts/vif-bridge', 'state' : '1', 'frontend': '/local/domain/13/device/vif/0', 'mac': '00:16:3e:37:29:82', 'online': '1', 'frontend-id': '13', 'type': 'netfront'} to /local/domain/0/backend/vif/13/0. [2007-07-16 15:26:56 3842] DEBUG (XendDomainInfo:1981) Storing VM details: {'on_xend_stop': 'ignore', 'shadow _memory': '0', 'uuid': 'c0a52e8b-0712-edb3-85aa-552c4fb3779f', 'on_reboot': 'restart', 'start_time': '1184614 010.78', 'on_poweroff': 'destroy', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0 ', 'vcpus': '2', 'vcpu_avail': '3', 'image': "(linux (kernel /usr0/VM/vmlinuz-2.6.16.13-xen) (args 'root=/dev /hda1 ro selinux=0 3') (notes (FEATURES 'writable_page_tables|writable_descriptor_tables|auto_translated_phys map|pae_pgdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE -2147483648) (GUEST_VERSION 2.6) (PADDR_OFFSET -2 147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) (LOADER generic) (SUSPEND_CANCEL 1) (ENTRY -21453864 96) (XEN_VERSION xen-3.0)))", 'name': 'compute-1-8-vm30'} [2007-07-16 15:27:02 3842] DEBUG (XendDomainInfo:838) Storing domain details: {'console/ring-ref': '2210012', 'image/entry': '-2145386496', 'console/port': '2', 'store/ring-ref': '2210013', 'image/loader': 'generic', ' vm': '/vm/c0a52e8b-0712-edb3-85aa-552c4fb3779f', 'control/platform-feature-multiprocessor-suspend': '1', 'ima ge/guest-os': 'linux', 'cpu/1/availability': 'online', 'image/features/writable-descriptor-tables': '1', 'ima ge/virt-base': '-2147483648', 'memory/target': '102400', 'image/guest-version': '2.6', 'image/features/superv isor-mode-kernel': '1', 'console/limit': '1048576', 'image/paddr-offset': '-2147483648', 'image/hypercall-pag e': '-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-abo ve-4gb': '1', 'image/features/writable-page-tables': '1', 'image/features/auto-translated-physmap': '1', 'nam e': 'compute-1-8-vm30', 'domid': '13', 'image/xen-version': 'xen-3.0', 'store/port': '1'} [2007-07-16 15:27:27 3842] DEBUG (DevController:115) DevController: writing {'protocol': 'x86_64-abi', 'state ': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/13/0'} to /local/domain/13/device/cons ole/0. [2007-07-16 15:27:27 3842] DEBUG (DevController:117) DevController: writing {'domain': 'compute-1-8-vm30', 'p rotocol': 'vt100', 'uuid': '4617b955-87a8-c9e3-638a-616d0bd28b67', 'frontend': '/local/domain/13/device/conso le/0', 'state': '1', 'location': '2', 'online': '1', 'frontend-id': '13'} to /local/domain/0/backend/console/ 13/0. [2007-07-16 15:27:41 3842] DEBUG (XendDomainInfo:838) Storing domain details: {'console/ring-ref': '2210012', 'image/entry': '-2145386496', 'console/port': '2', 'store/ring-ref': '2210013', 'image/loader': 'generic', ' vm': '/vm/c0a52e8b-0712-edb3-85aa-552c4fb3779f', 'control/platform-feature-multiprocessor-suspend': '1', 'ima ge/guest-os': 'linux', 'cpu/1/availability': 'online', 'image/features/writable-descriptor-tables': '1', 'ima ge/virt-base': '-2147483648', 'memory/target': '102400', 'image/guest-version': '2.6', 'image/features/superv isor-mode-kernel': '1', 'console/limit': '1048576', 'image/paddr-offset': '-2147483648', 'image/hypercall-pag e': '-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-abo ve-4gb': '1', 'image/features/writable-page-tables': '1', 'image/features/auto-translated-physmap': '1', 'nam e': 'compute-1-8-vm30', 'domid': '13', 'image/xen-version': 'xen-3.0', 'store/port': '1'} [2007-07-16 15:28:43 3842] DEBUG (XendDomainInfo:922) XendDomainInfo.handleShutdownWatch [2007-07-16 15:29:41 3842] DEBUG (DevController:148) Waiting for devices vif. [2007-07-16 15:29:44 3842] DEBUG (DevController:153) Waiting for 0. [2007-07-16 15:29:49 3842] DEBUG (DevController:541) hotplugStatusCallback /local/domain/0/backend/vif/13/0/h otplug-status. [2007-07-16 15:29:51 3842] DEBUG (DevController:555) hotplugStatusCallback 1. [2007-07-16 15:29:53 3842] DEBUG (DevController:148) Waiting for devices usb. [2007-07-16 15:29:56 3842] DEBUG (DevController:148) Waiting for devices vbd. [2007-07-16 15:29:58 3842] DEBUG (DevController:153) Waiting for 769. [2007-07-16 15:30:03 3842] DEBUG (DevController:541) hotplugStatusCallback /local/domain/0/backend/vbd/13/769 /hotplug-status. [2007-07-16 15:30:03 3842] DEBUG (DevController:555) hotplugStatusCallback 1. [2007-07-16 15:30:05 3842] DEBUG (DevController:153) Waiting for 770. [2007-07-16 15:30:08 3842] DEBUG (DevController:541) hotplugStatusCallback /local/domain/0/backend/vbd/13/770 /hotplug-status. [2007-07-16 15:30:10 3842] DEBUG (DevController:555) hotplugStatusCallback 1. [2007-07-16 15:30:12 3842] DEBUG (DevController:153) Waiting for 771. [2007-07-16 15:30:17 3842] DEBUG (DevController:541) hotplugStatusCallback /local/domain/0/backend/vbd/13/771 /hotplug-status. [2007-07-16 15:30:20 3842] DEBUG (DevController:555) hotplugStatusCallback 1. [2007-07-16 15:30:22 3842] DEBUG (DevController:148) Waiting for devices irq. [2007-07-16 15:30:24 3842] DEBUG (DevController:148) Waiting for devices vkbd. [2007-07-16 15:30:27 3842] DEBUG (DevController:148) Waiting for devices vfb. [2007-07-16 15:30:29 3842] DEBUG (DevController:148) Waiting for devices console. [2007-07-16 15:30:31 3842] DEBUG (DevController:153) Waiting for 0. [2007-07-16 15:30:31 3842] DEBUG (DevController:148) Waiting for devices pci. [2007-07-16 15:30:34 3842] DEBUG (DevController:148) Waiting for devices ioports. [2007-07-16 15:30:36 3842] DEBUG (DevController:148) Waiting for devices tap. [2007-07-16 15:30:38 3842] DEBUG (DevController:148) Waiting for devices vtpm. [2007-07-16 15:30:41 3842] INFO (XendDomain:1120) Domain compute-1-8-vm30 (13) unpaused. -- Configure bugmail: http://bugzilla.xensource.com/bugzilla/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee. _______________________________________________ Xen-bugs mailing list Xen-bugs@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-bugs
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |