Re: [Wg-openstack] Oracle VM with Nova/libvirtd/libguestfs

[Mark Duggan <mark.duggan@xxxxxxxxxx>]

Hi Lars,

Thanks for fielding this one.

The system I'm using the following xen pkg versions:
xen-4.4.4-65.el6.x86_64
xenpvboot-0.1-8.el6.noarch
xen-tools-4.4.4-65.el6.x86_64

With the following libvirt pkg versions:
libvirt-1.2.14-4.el6.x86_64
libvirt-client-1.2.14-4.el6.x86_64
libvirt-daemon-1.2.14-4.el6.x86_64
libvirt-daemon-config-network-1.2.14-4.el6.x86_64
libvirt-daemon-config-nwfilter-1.2.14-4.el6.x86_64
libvirt-daemon-driver-interface-1.2.14-4.el6.x86_64
libvirt-daemon-driver-libxl-1.2.14-4.el6.x86_64
libvirt-daemon-driver-network-1.2.14-4.el6.x86_64
libvirt-daemon-driver-nodedev-1.2.14-4.el6.x86_64
libvirt-daemon-driver-nwfilter-1.2.14-4.el6.x86_64
libvirt-daemon-driver-qemu-1.2.14-4.el6.x86_64
libvirt-daemon-driver-secret-1.2.14-4.el6.x86_64
libvirt-daemon-driver-storage-1.2.14-4.el6.x86_64
libvirt-daemon-driver-xen-1.2.14-4.el6.x86_64
libvirt-daemon-kvm-1.2.14-4.el6.x86_64
libvirt-daemon-xen-1.2.14-4.el6.x86_64
libvirt-debuginfo-1.2.14-4.el6.x86_64
libvirt-python-1.2.14-4.el6.x86_64

Kernel versions is:
4.1.12-23.el6uek.x86_64

I'm running a Kilo nova_compute as a docker container, directly on my
my OVM/Xen instance.  Libvirtd is running natively on OVM.

So, what I'm seeing is that when I  try to create a VM from nova
with something like:

"docker-ostk nova boot --image cirros1 --flavor m1small manual20"

The process  gets as far as a libvirt talking to libguestfs but
it seems to get into trouble before requesting xl to instantiate the vm.
Here's a snip from the nova_compute logs:

.....
{"log":"2016-01-12 12:25:51.093 1 DEBUG oslo_concurrency.processutils 
[req-f945b5cf-a5d5-4eb2-bd3f-c1eb27d87cf7 
a20819f72c784bf184e2ba424bf1fc00 a8bfdaf593c644a5bbd2397f5b51d208 - - -] 
CMD \"qemu-img resize 
/var/lib/nova/instances/4cb9b1e6-1cdb-4634-b4d3-87da829e06db/disk 
21474836480\" returned: 0 in 0.065s execute 
/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:254\n","stream":"stderr","time":"2016-01-12T12:25:51.09462224Z"}
{"log":"2016-01-12 12:25:51.094 1 DEBUG nova.virt.disk.api 
[req-f945b5cf-a5d5-4eb2-bd3f-c1eb27d87cf7 
a20819f72c784bf184e2ba424bf1fc00 a8bfdaf593c644a5bbd2397f5b51d208 - - -] 
Checking if we can extend filesystem inside 
/var/lib/nova/instances/4cb9b1e6-1cdb-4634-b4d3-87da829e06db/disk. 
CoW=True is_image_extendable 
/usr/lib/python2.7/site-packages/nova/virt/disk/api.py:227\n","stream":"stderr","time":"2016-01-12T12:25:51.0952897Z"}
{"log":"2016-01-12 12:25:51.095 1 DEBUG nova.virt.disk.vfs.api 
[req-f945b5cf-a5d5-4eb2-bd3f-c1eb27d87cf7 
a20819f72c784bf184e2ba424bf1fc00 a8bfdaf593c644a5bbd2397f5b51d208 - - -] 
Instance for image 
imgfile=/var/lib/nova/instances/4cb9b1e6-1cdb-4634-b4d3-87da829e06db/disk imgfmt=qcow2 
partition=None instance_for_image 
/usr/lib/python2.7/site-packages/nova/virt/disk/vfs/api.py:46\n","stream":"stderr","time":"2016-01-12T12:25:51.095806962Z"}
{"log":"2016-01-12 12:25:51.095 1 DEBUG nova.virt.disk.vfs.api 
[req-f945b5cf-a5d5-4eb2-bd3f-c1eb27d87cf7 
a20819f72c784bf184e2ba424bf1fc00 a8bfdaf593c644a5bbd2397f5b51d208 - - -] 
Using primary VFSGuestFS instance_for_image 
/usr/lib/python2.7/site-packages/nova/virt/disk/vfs/api.py:50\n","stream":"stderr","time":"2016-01-12T12:25:51.096208082Z"}
{"log":"2016-01-12 12:25:51.114 1 DEBUG oslo_concurrency.lockutils [-] 
Releasing semaphore 
\"refresh_cache-4cb9b1e6-1cdb-4634-b4d3-87da829e06db\" lock 
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:404\n","stream":"stderr","time":"2016-01-12T12:25:51.114488866Z"}
{"log":"2016-01-12 12:25:51.114 1 DEBUG nova.compute.manager [-] 
[instance: 4cb9b1e6-1cdb-4634-b4d3-87da829e06db] Instance network_info: 
|[VIF({'profile': {}, 'ovs_interfaceid': 
u'afb4f71e-e472-4be4-9973-11e5d601f94d', 'preserve_on_delete': False, 
'network': Network({'bridge': 'br-int', 'subnets': [Subnet({'ips': 
[FixedIP({'meta': {}, 'version': 4, 'type': 'fixed', 'floating_ips': [], 
'address': u'10.10.10.59'})], 'version': 4, 'meta': {'dhcp_server': 
u'10.10.10.2'}, 'dns': [], 'routes': [], 'cidr': u'10.10.10.0/24', 
'gateway': IP({'meta': {}, 'version': 4, 'type': 'gateway', 'address': 
u'10.10.10.1'})})], 'meta': {'injected': False, 'tenant_id': 
u'a8bfdaf593c644a5bbd2397f5b51d208'}, 'id': 
u'20b334c8-c738-4220-9c9e-5f898ccdbe9d', 'label': u'marks'}), 'devname': 
u'tapafb4f71e-e4', 'vnic_type': u'normal', 'qbh_params': None, 'meta': 
{}, 'details': {u'port_filter': True, u'ovs_hybrid_plug': True}, 
'address': u'fa:16:3e:1e:56:0f', 'active': False, 'type': u'ovs', 'id': 
u'afb4f71e-e472-4be4-9973-11e5d601f94d', 'qbg_params': None})]| 
_allocate_network_async 
/usr/lib/python2.7/site-packages/nova/compute/manager.py:1797\n","stream":"stderr","time":"2016-01-12T12:25:51.115128397Z"}
{"log":"2016-01-12 12:25:51.398 1 WARNING nova.virt.disk.api 
[req-f945b5cf-a5d5-4eb2-bd3f-c1eb27d87cf7 
a20819f72c784bf184e2ba424bf1fc00 a8bfdaf593c644a5bbd2397f5b51d208 - - -] 
Unable to mount image 
/var/lib/nova/instances/4cb9b1e6-1cdb-4634-b4d3-87da829e06db/disk with 
error libguestfs installed but not usable (could not create appliance 
through 
libvirt.\n","stream":"stderr","time":"2016-01-12T12:25:51.399873589Z"}
{"log":"\n","stream":"stderr","time":"2016-01-12T12:25:51.399926259Z"}
{"log":"Try running qemu directly without libvirt using this environment 
variable:\n","stream":"stderr","time":"2016-01-12T12:25:51.399940827Z"}
{"log":"export 
LIBGUESTFS_BACKEND=direct\n","stream":"stderr","time":"2016-01-12T12:25:51.399981544Z"}
{"log":"\n","stream":"stderr","time":"2016-01-12T12:25:51.399994315Z"}
{"log":"Original error from libvirt: Cannot access storage file 
'/tmp/libguestfsBgj7ba/overlay2' (as uid:107, gid:107): No such file or 
directory [code=38 domain=18]). Cannot 
resize.\n","stream":"stderr","time":"2016-01-12T12:25:51.400006722Z"}
........

I've attached the full log.

Here's the relevant section from libvirtd.log (full log also attached):

2016-01-12 12:21:24.141+0000: 8605: debug : 
qemuDomainCheckDiskPresence:2606 : Checking for disk presence
/usr/lib64/libvirt.so.0(virLogMessage+0x89)[0x7efe1a351e99]
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so(qemuDomainCheckDiskPresence+0x59)[0x7efde9446429]
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so(qemuProcessStart+0xe1d)[0x7efde94625dd]
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so(+0xd75f7)[0x7efde94b95f7]
/usr/lib64/libvirt.so.0(virDomainCreateXML+0x99)[0x7efe1a40e049]
libvirtd(+0x3e1fe)[0x56110a6691fe]
/usr/lib64/libvirt.so.0(virNetServerProgramDispatch+0x4f8)[0x7efe1a4875d8]
libvirtd(+0x5050e)[0x56110a67b50e]
libvirtd(+0x51028)[0x56110a67c028]
/usr/lib64/libvirt.so.0(+0x3712cb905c)[0x7efe1a37c05c]
/usr/lib64/libvirt.so.0(+0x3712cb8569)[0x7efe1a37b569]
/lib64/libpthread.so.0(+0x329ea07a51)[0x7efe17a53a51]
/lib64/libc.so.6(clone+0x6d)[0x7efe177a09ad]

2016-01-12 12:21:24.142+0000: 8605: info : virObjectRef:296 : 
OBJECT_REF: obj=0x7efdd41014b0
2016-01-12 12:21:24.142+0000: 8605: debug : 
virStorageFileGetMetadata:2938 : path=/tmp/libguestfseSpFZD/overlay2 
format=14 uid=107 gid=107 probe=0, report_broken=1
2016-01-12 12:21:24.142+0000: 8605: debug : 
virStorageFileGetMetadataRecurse:2821 : 
path=/tmp/libguestfseSpFZD/overlay2 format=14 uid=107 gid=107 probe=0
2016-01-12 12:21:24.142+0000: 8605: debug : 
virStorageFileBackendFileInit:1386 : initializing FS storage file 
0x7efdfc00b9e0 (file:/tmp/libguestfseSpFZD/overlay2)[107:107]
2016-01-12 12:21:24.154+0000: 8605: error : 
virStorageFileGetMetadataRecurse:2836 : Cannot access storage file 
'/tmp/libguestfseSpFZD/overlay2' (as uid:107, gid:107): No such file or 
directory
2016-01-12 12:21:24.154+0000: 8605: debug : 
virStorageFileBackendFileDeinit:1369 : deinitializing FS storage file 
0x7efdfc00b9e0 (file:/tmp/libguestfseSpFZD/overlay2)
2016-01-12 12:21:24.154+0000: 8605: info : virObjectUnref:259 : 
OBJECT_UNREF: obj=0x7efdd41014b0
2016-01-12 12:21:24.154+0000: 8605: debug : virFileClose:102 : Closed fd 31
2016-01-12 12:21:24.155+0000: 8605: info : virObjectRef:296 : 
OBJECT_REF: obj=0x7efdd41014b0
2016-01-12 12:21:24.155+0000: 8605: debug : qemuProcessStop:5089 : 
Shutting down vm=0x7efdfc004a20 name=guestfs-m5jiy8mxifhlmro8 id=3 pid=0 
flags=2
I'm also including the libxl-driver log, in case there is anything 
relevant in it.
Thanks,

Mark







On 01/12/2016 03:22 PM, Lars Kurth wrote:
> Mark,
> I am adding our openstack WG list. We do have a WG meeting tomorrow at
> 17:00 UK time. We can discuss the issues there. You are also welcome to
> join the call
> Regards
> Lars

Attachment: libvirtd.log.gz
Description: GNU Zip compressed data

Attachment: libxl-driver.log.gz
Description: GNU Zip compressed data

Attachment: nova_compute.log.gz
Description: GNU Zip compressed data

_______________________________________________
Wg-openstack mailing list
Wg-openstack@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-openstack