[Xen-devel] Shared image files and block script performance

[Mike Latimer <mlatimer@xxxxxxxx>]

Hi,

In an environment with read-only image files being shared across domains, the 
block script becomes exponentially slower with every block attached. While 
this is irritating with a few domains, it becomes very problematic with 
hundreds of domains.

Part of the issue was mentioned in a udev timeout thread started in August. 
[1] In that thread, Roger Pau Monné pointed out a portion of the block script 
that takes an increasingly longer time to execute with every instance of an 
shared image being used. The block of code mentioned is:

shared_list=$(losetup -a |
          sed -n -e "s@^\([^:]\+\)\(:[[:blank:]]\[0*${dev}\]:${inode}
[[:blank:]](.*)\)@\1@p" )
for dev in $shared_list
do
  if [ -n "$dev" ]
  then
    check_file_sharing "$file" "$dev" "$mode"
  fi
done

The code snippet above builds a list of all loopback mounted devices sharing 
the inode of the device being attached. This is not a problem, but the 
subsequent check_file_sharing (which leads to check_sharing) performs the 
following xenstore actions:

for dom in $(xenstore-list "$base_path")
do
  for dev in $(xenstore-list "$base_path/$dom")
  do
    d=$(xenstore_read_default "$base_path/$dom/$dev/physical-device" "")
    ...
    local m=$(xenstore_read_default "$base_path/$dom/$dev/mode" "")

In other words, for every loopback device found in the "shared_list", the 
entire xenstore database is walked - to determine if a domain is using that 
device in write mode. If nothing else is, the 'physical-device' and 'mode' of 
every vbd block device under every domain is checked and cleared. After 
looking through the entire database, the next device is checked - and the tree 
walk starts from the beginning again. The net result is an exponential 
increase in the execution time of the block attach process.

Is there some way the efficiency of this process can be improved? Some options 
I've tried include:

 - Reducing the number of calls to xenstore (combining two reads into one, 
etc.). I saw no real performance gains, but more optimization could be done.

 - Reading all vbd devices from xenstore at the entry of check_sharing, and 
searching this output for matching devices. This works, but does not seem to 
improve things much as the entire database is still read for every shared_list 
device.

 - Passing the entire shared_list to check_sharing (in a single variable), and 
checking every vbd device found in xenstore against this list. This change had 
a dramatic improvement in performance. However, this change seemed like a 
hack. Given enough domains, the size of this list could still be an issue.

With the last option above in place, all of my tests showed a block attach 
time of around 1 second. Without the change, I saw block attach times from 1 
to 1500 (with less than 40 domains sharing one device).

Any better options or ideas?

Thanks,
Mike

[1] http://lists.xen.org/archives/html/xen-devel/2015-08/msg00132.html

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel