A special thanks goes out to felipef for
all the help today.
Â
History:
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ (4) host pool â one in a
failed state due to hardware failure
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ (1) 3.2T data lun â SR-UUID
= aa15042e-2cdd-5ebc-9f0e-3d189c5cb56a
Â
The issue:
The 3.2T datalun
was presenting as 91% utilized and only 33% virtually
allocated.
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
Work log:
Â
Results were confirmed via the XC GUI and
via the command line as identified below
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ xe sr-list params=all
uuid=aa15042e-2cdd-5ebc-9f0e-3d189c5cb56a
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
physical-utilisation ( RO): 3170843492352
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
physical-size ( RO): 3457918435328
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ virtual
size: 1316940152832
type
( RO): lvmohba
sm-config
(MRO): allocation: thick; use_vhd: true
Â
Further digging found that summing all the
vdis on the SR resulted in the virtual allocation number
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ Commands + results:
xe
vdi-list sr-uuid=aa15042e-2cdd-5ebc-9f0e-3d189c5cb56a
params=physical-utilisation --minimal | sed 's/,/ + /g' | bc
âl
physical
utilization:Â 1,210,564,214,784
xe
vdi-list sr-uuid=aa15042e-2cdd-5ebc-9f0e-3d189c5cb56a
params=virtual-size --minimal | sed 's/,/ + /g' | bc âl
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
virtual size: 1,316,940,152,832
Â
At this point we started looking at the VG
to see if there were some LVs that were taking space but not
known by the xapi
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ Command + result:
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ vgs
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
VGÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
ÂÂÂÂ#PV #LV #SN AttrÂÂ VSizeÂÂÂ VFree
VG_XenStorage-aa15042e-2cdd-5ebc-9f0e-3d189c5cb56aÂÂ
1Â 33ÂÂ 0 wz--n-ÂÂÂ 3.14T 267.36G
Â
(lvs
--units B | grep aa15042e | while read vg lv flags size; do
echo -n "$size +" | sed 's/B//g'; done; echo 0)| bc -l
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
3170843492352
Â
So at this point we have confirmed that
there are in fact lvs not accounted for by xapi. So we look
for them
lvs | grep
aa15042e | grep VHD | cut -c7-42 | while read uuid; do [ "$(xe
vdi-list uuid=$uuid --minimal)" == "" ] && echo $uuid
; done
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
This returned a long list of UUIDs that did not have a
matching entry in xapi
Â
Grabbing one of the UUIDs at random and
searching back in the xensource.log we find something strange
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
[20121113T09:05:32.654Z|debug|xcp-nc-bc1b8|1563388
inet-RPC|SR.scan R:b7ff8ccc6566|dispatcher]
Server_helpers.exec exception_handler: Got exception
SR_BACKEND_FAILURE_181: [ ; Error in Metadata volume operation
for SR. [opterr=VDI delete operation failed for parameters:
/dev/VG_XenStorage-aa15042e-2cdd-5ebc-9f0e-3d189c5cb56a/MGT,
c866d910-f52f-4b16-91be-f7c646c621a5. Error: Failed to read
file with params [3, 0, 512, 512]. Error: Input/output
error];Â ]
Â
After a little googling around and finally
finding a thread on the citrix forums
(http://forums.citrix.com/thread.jspa?threadID=299275) that
pointed me at a process to rebuild the metadata for that
specific SR without having to blow away the SR and start
fresh.
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ Commands
lvrename
/dev/VG_XenStorage-aa15042e-2cdd-5ebc-9f0e-3d189c5cb56a/MGT
/dev/VG_XenStorage-aa15042e-2cdd-5ebc-9f0e-3d189c5cb56a/OLDMGT
xe
sr-scan uuid=aa15042e-2cdd-5ebc-9f0e-3d189c5cb56a
Â
This got rid of the SR_backend errors but
the LVs continued to persist. Started looking in the SMlog
started seeing lines that pointed at the pool not being ready
and exiting
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ <25168> 2012-11-14
12:27:24.195463ÂÂÂÂÂ Pool is not ready, exiting
Â
At this point I manually forced the offline
node out of the pool and the SMlog reported a success in the
purge process.
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ xe host-forget
uuid=<down host>
Â
