[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] Xen with Ceph over libvirt not working ...

Hi everybody,

long time ago, when I was on this list, Xen 3.0 was not yet released :-)

But, I'm still with Xen :-D

In the past I was using Xen with DRBD and GlusterFS as storage, but now,
I have to go with Ceph too.

Software I'm using:
- CentOS 7.5.1804
- Kernel: 4.9.86-30.el7             @centos-virt-xen-48
- Xen: 4.8.3-5.el7                  @centos-virt-xen-48
- libvirt-xen: 4.1.0-2.xen48.el7    @centos-virt-xen-48
- Ceph: 2:12.2.5-0.el7              @Ceph

Xen itself works, as does Ceph too, but using the Ceph-RBD from Xen via
libvirt fails - accessing them directly after doing "rbd map" and
defining the "/dev/rbdNN" as physical device in the vm-config works, but
with many vm's it is too "confusing".

As far as I understood, there is still no Ceph-support in Xen, since it
was announced in 2013, so the way to go is with libvirt?

I was following this guide, to setup Ceph with libvirt:

# ceph auth get-or-create client.libvirt mon 'profile rbd' osd 'profile
rbd pool=devel-pool'
-> [client.libvirt]
       key = AQBThwFbGFRYFxxxxxxxxxxxxxxxxxxxxxxxxx==
# ceph auth ls
-> ...
       key: AQBThwFbGFRYFxxxxxxxxxxxxxxxxxxxxxxxxx==
       caps: [mon] profile rbd
       caps: [osd] profile rbd pool=devel-pool
# vi secret.xml
<secret ephemeral='no' private='no'>
        <usage type='ceph'>
                <name>client.libvirt secret</name>

# virsh secret-define --file secret.xml
-> Secret 07f3a0fe-0000-1111-2222-333333333333 created
# ceph auth get-key client.libvirt > client.libvirt.key
# cat client.libvirt.key
-> AQBThwFbGFRYFxxxxxxxxxxxxxxxxxxxxxxxxx==
# virsh secret-set-value --secret 07f3a0fe-0000-1111-2222-333333333333
--base64 $(cat client.libvirt.key)
-> Secret value set

# vi xml/testvm3.xml
<domain type='xen'>
    <disk type='network' device='disk'>
      <source protocol='rbd' name='devel-pool/testvm3.rbd'>
        <host name="" port="6789"/>
        <host name="" port="6789"/>
        <host name="" port="6789"/>
      <auth username='libvirt'>
        <secret type='ceph' uuid='07f3a0fe-0000-1111-2222-333333333333'/>
      <target dev='xvda' bus='xen'/>

# virsh define xml/testvm3.xml
-> Domain testvm3 defined from xml/testvm3.xml
# virsh start --console testvm3
error: Failed to start domain testvm3
error: internal error: libxenlight failed to create new domain 'testvm3'

So "somthing" goes wrong:

# cat /var/log/libvirt/libxl/libxl-driver.log
-> ...
2018-05-20 15:28:15.270+0000: libxl:
libxl_bootloader.c:634:bootloader_finished: bootloader failed - consult
logfile /var/log/xen/bootloader.7.log
2018-05-20 15:28:15.270+0000: libxl:
libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [26640]
exited with error status 1
2018-05-20 15:28:15.271+0000: libxl:
libxl_create.c:1259:domcreate_rebuild_done: cannot (re-)build domain: -3

# cat /var/log/xen/bootloader.7.log
Traceback (most recent call last):
  File "/usr/lib64/xen/bin/pygrub", line 896, in <module>
    part_offs = get_partition_offsets(file)
  File "/usr/lib64/xen/bin/pygrub", line 113, in get_partition_offsets
    image_type = identify_disk_image(file)
  File "/usr/lib64/xen/bin/pygrub", line 56, in identify_disk_image
    fd = os.open(file, os.O_RDONLY)
OSError: [Errno 2] No such file or directory:

So, as far as I "read" the logs, Xen does not find the RBD-device, but I
have no clue, how I can solve this :-( Any idea how it might work?

Thanks a lot,
kind regards,


Xen-users mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.