[Xen-users] Xen with Ceph over libvirt not working ...

[thg <nospam@xxxxxxxxx>]

Hi everybody,

long time ago, when I was on this list, Xen 3.0 was not yet released :-)

But, I'm still with Xen :-D

In the past I was using Xen with DRBD and GlusterFS as storage, but now,
I have to go with Ceph too.

Software I'm using:
- CentOS 7.5.1804
- Kernel: 4.9.86-30.el7             @centos-virt-xen-48
- Xen: 4.8.3-5.el7                  @centos-virt-xen-48
- libvirt-xen: 4.1.0-2.xen48.el7    @centos-virt-xen-48
- Ceph: 2:12.2.5-0.el7              @Ceph

Xen itself works, as does Ceph too, but using the Ceph-RBD from Xen via
libvirt fails - accessing them directly after doing "rbd map" and
defining the "/dev/rbdNN" as physical device in the vm-config works, but
with many vm's it is too "confusing".

As far as I understood, there is still no Ceph-support in Xen, since it
was announced in 2013, so the way to go is with libvirt?


I was following this guide, to setup Ceph with libvirt:
<http://docs.ceph.com/docs/master/rbd/libvirt/>:

# ceph auth get-or-create client.libvirt mon 'profile rbd' osd 'profile
rbd pool=devel-pool'
-> [client.libvirt]
       key = AQBThwFbGFRYFxxxxxxxxxxxxxxxxxxxxxxxxx==
# ceph auth ls
-> ...
   client.libvirt
       key: AQBThwFbGFRYFxxxxxxxxxxxxxxxxxxxxxxxxx==
       caps: [mon] profile rbd
       caps: [osd] profile rbd pool=devel-pool
       ...
# vi secret.xml
->
<secret ephemeral='no' private='no'>
        <usage type='ceph'>
                <name>client.libvirt secret</name>
        </usage>
</secret>

# virsh secret-define --file secret.xml
-> Secret 07f3a0fe-0000-1111-2222-333333333333 created
# ceph auth get-key client.libvirt > client.libvirt.key
# cat client.libvirt.key
-> AQBThwFbGFRYFxxxxxxxxxxxxxxxxxxxxxxxxx==
# virsh secret-set-value --secret 07f3a0fe-0000-1111-2222-333333333333
--base64 $(cat client.libvirt.key)
-> Secret value set

# vi xml/testvm3.xml
->
<domain type='xen'>
  <name>testvm3</name>
  ...
  <devices>
    <disk type='network' device='disk'>
      <source protocol='rbd' name='devel-pool/testvm3.rbd'>
        <host name="10.20.30.1" port="6789"/>
        <host name="10.20.30.2" port="6789"/>
        <host name="10.20.30.3" port="6789"/>
      </source>
      <auth username='libvirt'>
        <secret type='ceph' uuid='07f3a0fe-0000-1111-2222-333333333333'/>
      </auth>
      <target dev='xvda' bus='xen'/>
    </disk>
    ...

# virsh define xml/testvm3.xml
-> Domain testvm3 defined from xml/testvm3.xml
# virsh start --console testvm3
error: Failed to start domain testvm3
error: internal error: libxenlight failed to create new domain 'testvm3'


So "somthing" goes wrong:

# cat /var/log/libvirt/libxl/libxl-driver.log
-> ...
2018-05-20 15:28:15.270+0000: libxl:
libxl_bootloader.c:634:bootloader_finished: bootloader failed - consult
logfile /var/log/xen/bootloader.7.log
2018-05-20 15:28:15.270+0000: libxl:
libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [26640]
exited with error status 1
2018-05-20 15:28:15.271+0000: libxl:
libxl_create.c:1259:domcreate_rebuild_done: cannot (re-)build domain: -3

# cat /var/log/xen/bootloader.7.log
->
Traceback (most recent call last):
  File "/usr/lib64/xen/bin/pygrub", line 896, in <module>
    part_offs = get_partition_offsets(file)
  File "/usr/lib64/xen/bin/pygrub", line 113, in get_partition_offsets
    image_type = identify_disk_image(file)
  File "/usr/lib64/xen/bin/pygrub", line 56, in identify_disk_image
    fd = os.open(file, os.O_RDONLY)
OSError: [Errno 2] No such file or directory:
'rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789'


So, as far as I "read" the logs, Xen does not find the RBD-device, but I
have no clue, how I can solve this :-( Any idea how it might work?


Thanks a lot,
-- 
kind regards,

thg

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users