Re: [Xen-users] Error with DRBD9 and Xen 4.6.5 with HVM Guest

[Justin Pretorius <justin.pretorius@xxxxxxxxx>]

I also did a bit of debugging in the hotplug script (block-drbd), and in 
the "add" case statement, at the point where write_dev "$dev" is called, 
$dev is set as "/dev/drbd116". If this is the case, why is qemu getting 
instantiated as "-drive file=disk1-firewall2" instead of "-drive 
file=/dev/drbd116"? or is this just a red herring ?
On 09/07/18 11:28, Justin Pretorius wrote:
> On 09/07/18 09:08, Roger Pau Monné wrote:
> > On Thu, Sep 06, 2018 at 04:41:11PM +0200, Justin Pretorius wrote:
> > > Hello
> > >
> > > This is my first ever email list help request, apologies if I have 
> > > not 100%
> > > got the protocol correct.
> > >
> > > I am currently using Xen on Ubuntu 16.04, and am using a number of PVM
> > > guests successfully using DRBD9 as primary root disk, but when 
> > > testing an
> > > HVM PfSense (BSD) guest, I am having some issues.
> > >
> > > DRBD9 is successfully set up and when creating the guest using the 
> > > config as
> > > follows
> > >
> > > disk=['drbd:disk1-firewall2,hda,w']
> > >
> > > The guest will not create with the following info
> > >
> > > xl -v create /etc/xen/firewall2.cfg
> > >
> > > xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x5b844
> > > xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15b844
> > > xc: detail: VIRTUAL MEMORY ARRANGEMENT:
> > > xc: detail:   Loader:   0000000000100000->000000000015b844
> > > xc: detail:   Modules:  0000000000000000->0000000000000000
> > > xc: detail:   TOTAL:    0000000000000000->000000001f800000
> > > xc: detail:   ENTRY:    0000000000100630
> > > xc: detail: PHYSICAL MEMORY ALLOCATION:
> > > xc: detail:   4KB PAGES: 0x0000000000000200
> > > xc: detail:   2MB PAGES: 0x00000000000000fb
> > > xc: detail:   1GB PAGES: 0x0000000000000000
> > > xc: detail: elf_load_binary: phdr 0 at 0x7f979ffaa000 -> 0x7f979fffbdd1
> > > domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
> > > libxl: error: libxl_dm.c:1872:device_model_spawn_outcome: domain 74 
> > > device
> > > model: spawn failed (rc=-3)
> > > libxl: error: libxl_create.c:1344:domcreate_devmodel_started: device 
> > > model
> > > did not start: -3
> > > libxl: error: libxl_dm.c:1987:kill_device_model: Device Model already 
> > > exited
> > > libxl: error: libxl.c:1610:libxl__destroy_domid: non-existant domain 74
> > > libxl: error: libxl.c:1568:domain_destroy_callback: unable to destroy 
> > > guest
> > > with domid 74
> > > libxl: error: libxl.c:1495:domain_destroy_cb: destruction of domain 74
> > > failed
> > >
> > > Contents of guest log
> > >
> > > qemu-system-i386: -drive
> > > file=disk1-firewall2,if=ide,index=0,media=disk,format=raw,cache=writeback: 
> > >
> > > Could not open 'disk1-firewall2': No such file or directory
> > This is wrong, file= should point to some block device created by the
> > drbd hotplug script.
> Does this mean there is an issue with the handoff from Xen to qemu while 
> building whichever command is required to start the HVM guest. I have 
> used the hotplug script in the documented method which works for PV 
> guests. Are you saying that using the following format
> disk=['drbd:disk1-firewall2,hda,w']
>
> Is not the correct way for HVM guests? I have confirmed, that using the 
> actual dev mount works correctly 
> (disk=['phy:/dev/vg/disk1-firewall2_00,hda,w']), but this will not live 
> migrate, or even start on any node besides the last node that was primary.
> > > Contents of hotplug log
> > >
> > > disk1-firewall2: State change failed: (-12) Device is held open by 
> > > someone
> > > Command 'drbdsetup secondary disk1-firewall2' terminated with exit 
> > > code 11
> > I'm afraid this is due to block hotplug scripts being run twice for
> > HVM guests, once for the PV disk and another time for the emulated
> > disk.
> This could be a red herring, the row show here may be from previous 
> attempts to start the guest. Subsequent creates have not seemed to 
> create this line. I have deleted the hotplug.log and rerun, and the file 
> is still empty, I have restarted the xen service, in case a new file 
> handle was required, but it still is empty after a few failed create 
> attempts.
>
> > Could you please paste the full output of `xl -vvv create ...`, that
> > should contain all the calls to the hotplug scripts.
> Parsing config from firewall2.cfg
> libxl: debug: libxl_create.c:1563:do_domain_create: ao 0x1462fb0: 
> create: how=(nil) callback=(nil) poller=0x1462960
> libxl: debug: libxl_device.c:337:libxl__device_disk_set_backend: Disk 
> vdev=hda spec.backend=unknown
> libxl: debug: libxl_device.c:260:disk_try_backend: Disk vdev=hda, uses 
> script=... assuming phy backend
> libxl: debug: libxl_device.c:366:libxl__device_disk_set_backend: Disk 
> vdev=hda, using backend phy
> libxl: debug: libxl_create.c:947:initiate_domain_create: running bootloader
> libxl: debug: libxl_bootloader.c:324:libxl__bootloader_run: not a PV 
> domain, skipping bootloader
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
> w=0x1460808: deregister unregistered
> xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x5b844
> xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15b844
> xc: detail: VIRTUAL MEMORY ARRANGEMENT:
> xc: detail:   Loader:   0000000000100000->000000000015b844
> xc: detail:   Modules:  0000000000000000->0000000000000000
> xc: detail:   TOTAL:    0000000000000000->000000001f800000
> xc: detail:   ENTRY:    0000000000100630
> xc: detail: PHYSICAL MEMORY ALLOCATION:
> xc: detail:   4KB PAGES: 0x0000000000000200
> xc: detail:   2MB PAGES: 0x00000000000000fb
> xc: detail:   1GB PAGES: 0x0000000000000000
> xc: detail: elf_load_binary: phdr 0 at 0x7f53733d5000 -> 0x7f5373426dd1
> domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
> libxl: debug: libxl_device.c:337:libxl__device_disk_set_backend: Disk 
> vdev=hda spec.backend=phy
> libxl: debug: libxl_device.c:260:disk_try_backend: Disk vdev=hda, uses 
> script=... assuming phy backend
> libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch 
> w=0x1463b60 wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: 
> register slotnum=3
> libxl: debug: libxl_create.c:1586:do_domain_create: ao 0x1462fb0: 
> inprogress: poller=0x1462960, flags=i
> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1463b60 
> wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: event 
> epath=/local/domain/0/backend/vbd/85/768/state
> libxl: debug: libxl_event.c:884:devstate_callback: backend 
> /local/domain/0/backend/vbd/85/768/state wanted state 2 still waiting 
> state 1
> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1463b60 
> wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: event 
> epath=/local/domain/0/backend/vbd/85/768/state
> libxl: debug: libxl_event.c:880:devstate_callback: backend 
> /local/domain/0/backend/vbd/85/768/state wanted state 2 ok
> libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch 
> w=0x1463b60 wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: 
> deregister slotnum=3
> libxl: debug: libxl_device.c:991:device_backend_callback: calling 
> device_backend_cleanup
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
> w=0x1463b60: deregister unregistered
> libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and 
> environment ready
> libxl: debug: libxl_device.c:1088:device_hotplug: calling hotplug 
> script: /etc/xen/scripts/block-drbd add
> libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to 
> execute: /etc/xen/scripts/block-drbd add
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
> w=0x1463c60: deregister unregistered
> libxl: debug: libxl_linux.c:246:libxl__get_hotplug_script_info: num_exec 
> 1, not running hotplug scripts
> libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to 
> execute
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
> w=0x1463c60: deregister unregistered
> libxl: debug: libxl_dm.c:1796:libxl__spawn_local_dm: Spawning 
> device-model /usr/bin/qemu-system-i386 with arguments:
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: 
> /usr/bin/qemu-system-i386
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -xen-domid
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   85
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -chardev
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: 
> socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-85,server,nowait
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -no-shutdown
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -mon
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: 
> chardev=libxl-cmd,mode=control
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -chardev
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: 
> socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-85,server,nowait
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -mon
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: 
> chardev=libxenstat-cmd,mode=control
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -nodefaults
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -name
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   firewall2
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -vnc
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   :0,to=99
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -display
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   none
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -serial
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   pty
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -device
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: cirrus-vga,vgamem_mb=8
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -boot
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   order=c
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -net
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   none
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -machine
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   xenfv
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -m
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   504
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -drive
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: 
> file=disk1-firewall2,if=ide,index=0,media=disk,format=raw,cache=writeback
> libxl: debug: libxl_dm.c:1800:libxl__spawn_local_dm: Spawning 
> device-model /usr/bin/qemu-system-i386 with additional environment:
> libxl: debug: libxl_dm.c:1802:libxl__spawn_local_dm: 
> XEN_QEMU_CONSOLE_LIMIT=1048576
> libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch 
> w=0x1460b00 wpath=/local/domain/0/device-model/85/state token=3/1: 
> register slotnum=3
> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1460b00 
> wpath=/local/domain/0/device-model/85/state token=3/1: event 
> epath=/local/domain/0/device-model/85/state
> libxl: debug: libxl_exec.c:399:spawn_watch_event: domain 85 device 
> model: spawn watch p=(null)
> libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch 
> w=0x1460b00 wpath=/local/domain/0/device-model/85/state token=3/1: 
> deregister slotnum=3
> libxl: error: libxl_dm.c:1872:device_model_spawn_outcome: domain 85 
> device model: spawn failed (rc=-3)
> libxl: error: libxl_create.c:1344:domcreate_devmodel_started: device 
> model did not start: -3
> libxl: error: libxl_dm.c:1987:kill_device_model: Device Model already 
> exited
> libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch 
> w=0x1467550 wpath=/local/domain/0/backend/vbd/85/768/state token=3/2: 
> register slotnum=3
> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1467550 
> wpath=/local/domain/0/backend/vbd/85/768/state token=3/2: event 
> epath=/local/domain/0/backend/vbd/85/768/state
> libxl: debug: libxl_event.c:880:devstate_callback: backend 
> /local/domain/0/backend/vbd/85/768/state wanted state 6 ok
> libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch 
> w=0x1467550 wpath=/local/domain/0/backend/vbd/85/768/state token=3/2: 
> deregister slotnum=3
> libxl: debug: libxl_device.c:991:device_backend_callback: calling 
> device_backend_cleanup
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
> w=0x1467550: deregister unregistered
> libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and 
> environment ready
> libxl: debug: libxl_device.c:1088:device_hotplug: calling hotplug 
> script: /etc/xen/scripts/block-drbd remove
> libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to 
> execute: /etc/xen/scripts/block-drbd remove
> libxl: debug: libxl_event.c:545:watchfd_callback: watch 
> epath=/local/domain/0/backend/vbd/85/768/state token=3/2: empty slot
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
> w=0x1467650: deregister unregistered
> libxl: debug: libxl_linux.c:246:libxl__get_hotplug_script_info: num_exec 
> 1, not running hotplug scripts
> libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to 
> execute
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
> w=0x1467650: deregister unregistered
> libxl: debug: libxl_linux.c:267:libxl__get_hotplug_script_info: 
> backend_kind 6, no need to execute scripts
> libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to 
> execute
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
> w=0x1467ca0: deregister unregistered
> libxl: debug: libxl.c:1738:devices_destroy_cb: forked pid 26093 for 
> destroy of domain 85
> libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x1462fb0: 
> complete, rc=-3
> libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x1462fb0: destroy
> libxl: debug: libxl.c:1477:libxl_domain_destroy: ao 0x1462250: create: 
> how=(nil) callback=(nil) poller=0x1462960
> libxl: error: libxl.c:1610:libxl__destroy_domid: non-existant domain 85
> libxl: error: libxl.c:1568:domain_destroy_callback: unable to destroy 
> guest with domid 85
> libxl: error: libxl.c:1495:domain_destroy_cb: destruction of domain 85 
> failed
> libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x1462250: 
> complete, rc=-21
> libxl: debug: libxl.c:1486:libxl_domain_destroy: ao 0x1462250: 
> inprogress: poller=0x1462960, flags=ic
> libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x1462250: destroy
> xc: debug: hypercall buffer: total allocations:648 total releases:648
> xc: debug: hypercall buffer: current allocations:0 maximum allocations:3
> xc: debug: hypercall buffer: cache current size:3
> xc: debug: hypercall buffer: cache hits:632 misses:3 toobig:13
>
>
>
> > > And the guest is not started
> > >
> > > If i try to attach to dom 0 :
> > >
> > > xl block-list 0
> > > Vdev  BE  handle state evt-ch ring-ref BE-path
> > >
> > > xl block-attach 0 'drbd:disk1-firewall2,hda,w'      (no errors or 
> > > output)
> > > xl block-list 0
> > >
> > > Vdev  BE  handle state evt-ch ring-ref BE-path
> > > 768   0   0      4     102    8        /local/domain/0/backend/vbd/0/768
> > >
> > > xl block-detach 0 768   (no errors or output)
> > >
> > > xl block-list 0
> > >
> > > Vdev  BE  handle state evt-ch ring-ref BE-path
> > >
> > > So it mounts and unmounts fine manually, but not via the hotplug script?
> > When you attach to Dom0 you are also using the hotplug script.
> OK, which is possibly a useless test then, as the hotplug script then 
> works as expected for PV, and when run manually via block-attach, so I 
> think this confirms the script in fact works, just not in the HVM case.
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users