[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] Error with DRBD9 and Xen 4.6.5 with HVM Guest



I also did a bit of debugging in the hotplug script (block-drbd), and in the "add" case statement, at the point where write_dev "$dev" is called, $dev is set as "/dev/drbd116". If this is the case, why is qemu getting instantiated as "-drive file=disk1-firewall2" instead of "-drive file=/dev/drbd116"? or is this just a red herring ?

On 09/07/18 11:28, Justin Pretorius wrote:


On 09/07/18 09:08, Roger Pau Monné wrote:
On Thu, Sep 06, 2018 at 04:41:11PM +0200, Justin Pretorius wrote:
Hello

This is my first ever email list help request, apologies if I have not 100%
got the protocol correct.

I am currently using Xen on Ubuntu 16.04, and am using a number of PVM
guests successfully using DRBD9 as primary root disk, but when testing an
HVM PfSense (BSD) guest, I am having some issues.

DRBD9 is successfully set up and when creating the guest using the config as
follows

disk=['drbd:disk1-firewall2,hda,w']

The guest will not create with the following info

xl -v create /etc/xen/firewall2.cfg

xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x5b844
xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15b844
xc: detail: VIRTUAL MEMORY ARRANGEMENT:
xc: detail:   Loader:   0000000000100000->000000000015b844
xc: detail:   Modules:  0000000000000000->0000000000000000
xc: detail:   TOTAL:    0000000000000000->000000001f800000
xc: detail:   ENTRY:    0000000000100630
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail:   4KB PAGES: 0x0000000000000200
xc: detail:   2MB PAGES: 0x00000000000000fb
xc: detail:   1GB PAGES: 0x0000000000000000
xc: detail: elf_load_binary: phdr 0 at 0x7f979ffaa000 -> 0x7f979fffbdd1
domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
libxl: error: libxl_dm.c:1872:device_model_spawn_outcome: domain 74 device
model: spawn failed (rc=-3)
libxl: error: libxl_create.c:1344:domcreate_devmodel_started: device model
did not start: -3
libxl: error: libxl_dm.c:1987:kill_device_model: Device Model already exited
libxl: error: libxl.c:1610:libxl__destroy_domid: non-existant domain 74
libxl: error: libxl.c:1568:domain_destroy_callback: unable to destroy guest
with domid 74
libxl: error: libxl.c:1495:domain_destroy_cb: destruction of domain 74
failed

Contents of guest log

qemu-system-i386: -drive
file=disk1-firewall2,if=ide,index=0,media=disk,format=raw,cache=writeback:
Could not open 'disk1-firewall2': No such file or directory

This is wrong, file= should point to some block device created by the
drbd hotplug script.

Does this mean there is an issue with the handoff from Xen to qemu while building whichever command is required to start the HVM guest. I have used the hotplug script in the documented method which works for PV guests. Are you saying that using the following format

disk=['drbd:disk1-firewall2,hda,w']

Is not the correct way for HVM guests? I have confirmed, that using the actual dev mount works correctly (disk=['phy:/dev/vg/disk1-firewall2_00,hda,w']), but this will not live migrate, or even start on any node besides the last node that was primary.



Contents of hotplug log

disk1-firewall2: State change failed: (-12) Device is held open by someone Command 'drbdsetup secondary disk1-firewall2' terminated with exit code 11

I'm afraid this is due to block hotplug scripts being run twice for
HVM guests, once for the PV disk and another time for the emulated
disk.


This could be a red herring, the row show here may be from previous attempts to start the guest. Subsequent creates have not seemed to create this line. I have deleted the hotplug.log and rerun, and the file is still empty, I have restarted the xen service, in case a new file handle was required, but it still is empty after a few failed create attempts.


Could you please paste the full output of `xl -vvv create ...`, that
should contain all the calls to the hotplug scripts.


Parsing config from firewall2.cfg
libxl: debug: libxl_create.c:1563:do_domain_create: ao 0x1462fb0: create: how=(nil) callback=(nil) poller=0x1462960 libxl: debug: libxl_device.c:337:libxl__device_disk_set_backend: Disk vdev=hda spec.backend=unknown libxl: debug: libxl_device.c:260:disk_try_backend: Disk vdev=hda, uses script=... assuming phy backend libxl: debug: libxl_device.c:366:libxl__device_disk_set_backend: Disk vdev=hda, using backend phy
libxl: debug: libxl_create.c:947:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:324:libxl__bootloader_run: not a PV domain, skipping bootloader libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1460808: deregister unregistered
xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x5b844
xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15b844
xc: detail: VIRTUAL MEMORY ARRANGEMENT:
xc: detail:   Loader:   0000000000100000->000000000015b844
xc: detail:   Modules:  0000000000000000->0000000000000000
xc: detail:   TOTAL:    0000000000000000->000000001f800000
xc: detail:   ENTRY:    0000000000100630
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail:   4KB PAGES: 0x0000000000000200
xc: detail:   2MB PAGES: 0x00000000000000fb
xc: detail:   1GB PAGES: 0x0000000000000000
xc: detail: elf_load_binary: phdr 0 at 0x7f53733d5000 -> 0x7f5373426dd1
domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
libxl: debug: libxl_device.c:337:libxl__device_disk_set_backend: Disk vdev=hda spec.backend=phy libxl: debug: libxl_device.c:260:disk_try_backend: Disk vdev=hda, uses script=... assuming phy backend libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x1463b60 wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: register slotnum=3 libxl: debug: libxl_create.c:1586:do_domain_create: ao 0x1462fb0: inprogress: poller=0x1462960, flags=i libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1463b60 wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: event epath=/local/domain/0/backend/vbd/85/768/state libxl: debug: libxl_event.c:884:devstate_callback: backend /local/domain/0/backend/vbd/85/768/state wanted state 2 still waiting state 1 libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1463b60 wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: event epath=/local/domain/0/backend/vbd/85/768/state libxl: debug: libxl_event.c:880:devstate_callback: backend /local/domain/0/backend/vbd/85/768/state wanted state 2 ok libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch w=0x1463b60 wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: deregister slotnum=3 libxl: debug: libxl_device.c:991:device_backend_callback: calling device_backend_cleanup libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1463b60: deregister unregistered libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and environment ready libxl: debug: libxl_device.c:1088:device_hotplug: calling hotplug script: /etc/xen/scripts/block-drbd add libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/block-drbd add libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1463c60: deregister unregistered libxl: debug: libxl_linux.c:246:libxl__get_hotplug_script_info: num_exec 1, not running hotplug scripts libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to execute libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1463c60: deregister unregistered libxl: debug: libxl_dm.c:1796:libxl__spawn_local_dm: Spawning device-model /usr/bin/qemu-system-i386 with arguments: libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: /usr/bin/qemu-system-i386
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -xen-domid
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   85
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -chardev
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-85,server,nowait
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -no-shutdown
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -mon
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: chardev=libxl-cmd,mode=control
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -chardev
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-85,server,nowait
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -mon
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: chardev=libxenstat-cmd,mode=control
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -nodefaults
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -name
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   firewall2
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -vnc
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   :0,to=99
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -display
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   none
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -serial
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   pty
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -device
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: cirrus-vga,vgamem_mb=8
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -boot
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   order=c
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -net
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   none
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -machine
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   xenfv
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -m
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   504
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -drive
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: file=disk1-firewall2,if=ide,index=0,media=disk,format=raw,cache=writeback libxl: debug: libxl_dm.c:1800:libxl__spawn_local_dm: Spawning device-model /usr/bin/qemu-system-i386 with additional environment: libxl: debug: libxl_dm.c:1802:libxl__spawn_local_dm: XEN_QEMU_CONSOLE_LIMIT=1048576 libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x1460b00 wpath=/local/domain/0/device-model/85/state token=3/1: register slotnum=3 libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1460b00 wpath=/local/domain/0/device-model/85/state token=3/1: event epath=/local/domain/0/device-model/85/state libxl: debug: libxl_exec.c:399:spawn_watch_event: domain 85 device model: spawn watch p=(null) libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch w=0x1460b00 wpath=/local/domain/0/device-model/85/state token=3/1: deregister slotnum=3 libxl: error: libxl_dm.c:1872:device_model_spawn_outcome: domain 85 device model: spawn failed (rc=-3) libxl: error: libxl_create.c:1344:domcreate_devmodel_started: device model did not start: -3 libxl: error: libxl_dm.c:1987:kill_device_model: Device Model already exited libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x1467550 wpath=/local/domain/0/backend/vbd/85/768/state token=3/2: register slotnum=3 libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1467550 wpath=/local/domain/0/backend/vbd/85/768/state token=3/2: event epath=/local/domain/0/backend/vbd/85/768/state libxl: debug: libxl_event.c:880:devstate_callback: backend /local/domain/0/backend/vbd/85/768/state wanted state 6 ok libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch w=0x1467550 wpath=/local/domain/0/backend/vbd/85/768/state token=3/2: deregister slotnum=3 libxl: debug: libxl_device.c:991:device_backend_callback: calling device_backend_cleanup libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1467550: deregister unregistered libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and environment ready libxl: debug: libxl_device.c:1088:device_hotplug: calling hotplug script: /etc/xen/scripts/block-drbd remove libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/block-drbd remove libxl: debug: libxl_event.c:545:watchfd_callback: watch epath=/local/domain/0/backend/vbd/85/768/state token=3/2: empty slot libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1467650: deregister unregistered libxl: debug: libxl_linux.c:246:libxl__get_hotplug_script_info: num_exec 1, not running hotplug scripts libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to execute libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1467650: deregister unregistered libxl: debug: libxl_linux.c:267:libxl__get_hotplug_script_info: backend_kind 6, no need to execute scripts libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to execute libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch w=0x1467ca0: deregister unregistered libxl: debug: libxl.c:1738:devices_destroy_cb: forked pid 26093 for destroy of domain 85 libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x1462fb0: complete, rc=-3
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x1462fb0: destroy
libxl: debug: libxl.c:1477:libxl_domain_destroy: ao 0x1462250: create: how=(nil) callback=(nil) poller=0x1462960
libxl: error: libxl.c:1610:libxl__destroy_domid: non-existant domain 85
libxl: error: libxl.c:1568:domain_destroy_callback: unable to destroy guest with domid 85 libxl: error: libxl.c:1495:domain_destroy_cb: destruction of domain 85 failed libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x1462250: complete, rc=-21 libxl: debug: libxl.c:1486:libxl_domain_destroy: ao 0x1462250: inprogress: poller=0x1462960, flags=ic
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x1462250: destroy
xc: debug: hypercall buffer: total allocations:648 total releases:648
xc: debug: hypercall buffer: current allocations:0 maximum allocations:3
xc: debug: hypercall buffer: cache current size:3
xc: debug: hypercall buffer: cache hits:632 misses:3 toobig:13



And the guest is not started

If i try to attach to dom 0 :

xl block-list 0
Vdev  BE  handle state evt-ch ring-ref BE-path

xl block-attach 0 'drbd:disk1-firewall2,hda,w'      (no errors or output)

xl block-list 0

Vdev  BE  handle state evt-ch ring-ref BE-path
768   0   0      4     102    8        /local/domain/0/backend/vbd/0/768

xl block-detach 0 768   (no errors or output)

xl block-list 0

Vdev  BE  handle state evt-ch ring-ref BE-path

So it mounts and unmounts fine manually, but not via the hotplug script?

When you attach to Dom0 you are also using the hotplug script.


OK, which is possibly a useless test then, as the hotplug script then works as expected for PV, and when run manually via block-attach, so I think this confirms the script in fact works, just not in the HVM case.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.