Re: [Xen-devel] block-iscsi with Xen 4.5 / 4.6

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

Hello,

I'm re-adding xen-devel in case someone else also wants to provide feedback.

On Wed, May 04, 2016 at 03:06:23PM +1000, Steven Haigh wrote:
> Hi Roger,
> 
> I've been getting some good progress with iSCSI thanks to your insights.
> 
> I'm now trying to add support for locking via Persistent Reservations to
> ensure that only one Dom0 can attach / use a single iSCSI target at once.

This might be problematic with migrations. IIRC there's a point during the 
migration where both the sending and the receiving side have the disk open 
at the same time. However Xen always makes sure that only one guest is 
actually accessing the disk, either the one on the receiving side (if 
everything has gone OK) or the one on the senders side (if migration has 
failed).

> In a nutshell, my thoughts are to use the following to 'lock' a device:
>       ## Create a hex key for the lock from the systems IP.
>       key=$(gethostip -x $(uname -n))
>       sg_persist -d ${dev} -o -G -S ${key}
>       sg_persist -d ${dev} -o -R -K ${key} -T 6
> 
> This registers the device, and sets an Exclusive Access (-T 6) flag on
> the iSCSI device which means nothing else will be able to open the
> device until the lock is removed.
> 
> To unlock the device, on remove, we should do something like:
>       key=$(gethostip -x $(uname -n))
>         sg_persist -d ${dev} -o -L -K ${key} -T 6
>         sg_persist -d ${dev} -o -G -K ${key} -S 0
> 
> This releases the device for other things to use.
> 
> I've tried putting these in block-iscsi - by using a lock_device and
> unlock_device function and calling it after find_device in both attach()
> and remove().
> 
> My problems:
> 1) -e is set on the script - and maybe elsewhere - so any time something
> returns non-zero, you can't clean up. For example, if you can't get a
> lock, you should make sure all locks are removed from the host in
> question and then detach the iSCSI target.

You can avoid this by adding something like:

sg_persist ... || true

Of course you can replace the "true" command with something else, like a 
fatal message or some cleanup code. You can also place the command inside of 
a conditional if you know it might fail:

if ! sg_persist ...; then
        fatal ...
fi

It is important for us to use the '-e' in order to make sure all the failure 
points are correctly handled, without the '-e' some command might fail and 
the script wouldn't realize.

> 2) I can't find an easy way to clean up by doing an iscsiadm --logout if
> the locking fails.

I'm not really following here, maybe because I don't know that much about 
iSCSI. Can you just put whatever code is needed in order to unlock before 
doing the logout? Or that's not how it works?

> I'm wondering if there is a reason that the script is currently in the
> stucture that it is - or if it just evolved like this? It may be a good
> candidate for a complete re-write :\

TBH, I thought this was one of the most clean and well structured block 
scripts that Xen has ;).

Roger.

> On 15/04/2016 7:01 PM, Roger Pau Monné wrote:
> > On Fri, Apr 15, 2016 at 06:20:56PM +1000, Steven Haigh wrote:
> > [...]
> >> I might have spoken too soon here... I updated this system to 4.6.1 and
> >> created the DomU again - still seems to fail - although it does actually
> >> call the block-iscsi script this time:
> >>
> >> # xl -vvv create /etc/xen/test1.vm
> >> Parsing config from /etc/xen/test1.vm
> >> libxl: debug: libxl_create.c:1560:do_domain_create: ao 0x24ad330: create:
> >> how=(nil) callback=(nil) poller=0x24b7070
> >> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
> >> vdev=xvda spec.backend=unknown
> >> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=xvda, uses
> >> script=... assuming phy backend
> >> libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk
> >> vdev=xvda, using backend phy
> >> libxl: debug: libxl_create.c:945:initiate_domain_create: running bootloader
> >> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
> >> vdev=(null) spec.backend=phy
> >> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=(null), uses
> >> script=... assuming phy backend
> >> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
> >> vdev=xvde spec.backend=phy
> >> libxl: debug: libxl_device.c:207:disk_try_backend: Disk vdev=xvde, uses
> >> script=... assuming phy backend
> >> libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
> >> w=0x24ada00 wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0:
> >> register slotnum=3
> >> libxl: debug: libxl_create.c:1583:do_domain_create: ao 0x24ad330:
> >> inprogress: poller=0x24b7070, flags=i
> >> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x24ada00
> >> wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0: event
> >> epath=/local/domain/0/backend/vbd/0/51776/state
> >> libxl: debug: libxl_event.c:884:devstate_callback: backend
> >> /local/domain/0/backend/vbd/0/51776/state wanted state 2 still waiting 
> >> state
> >> 1
> >> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x24ada00
> >> wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0: event
> >> epath=/local/domain/0/backend/vbd/0/51776/state
> >> libxl: debug: libxl_event.c:880:devstate_callback: backend
> >> /local/domain/0/backend/vbd/0/51776/state wanted state 2 ok
> >> libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch
> >> w=0x24ada00 wpath=/local/domain/0/backend/vbd/0/51776/state token=3/0:
> >> deregister slotnum=3
> >> libxl: debug: libxl_device.c:937:device_backend_callback: calling
> >> device_backend_cleanup
> >> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
> >> w=0x24ada00: deregister unregistered
> >> libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and environment
> >> ready
> >> libxl: debug: libxl_device.c:1034:device_hotplug: calling hotplug script:
> >> /etc/xen/scripts/block-iscsi add
> >> libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to
> >> execute: /etc/xen/scripts/block-iscsi add
> >> libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus:
> >> /etc/xen/scripts/block-iscsi add [2126] exited with error status 1
> >> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
> >> w=0x24adb00: deregister unregistered
> >> libxl: error: libxl_device.c:1084:device_hotplug_child_death_cb: script:
> >> Device already opened
> > 
> > The message indicates that you have this device already opened in this 
> > system, this is detected by the following check, that you can also run from 
> > a shell:
> > 
> > # iscsiadm -m session 2>&1 | grep -q "$iqn" && fatal "Device already opened"
> > 
> > You will have to perform a logout in order for the hotplug script to 
> > correctly attach it.
> > 
> > Roger.
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxx
> > http://lists.xen.org/xen-devel
> > 
> 
> -- 
> Steven Haigh
> 
> Email: netwiz@xxxxxxxxx
> Web: https://www.crc.id.au
> Phone: (03) 9001 6090 - 0412 935 897
> 




_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel