Re: [Xen-devel] [PATCH 12 of 14 v4] libxl: set frontend status to 6 on domain destroy

[Roger Pau Monnà <roger.pau@xxxxxxxxxxxxx>]

2011/12/16 Ian Campbell <Ian.Campbell@xxxxxxxxxx>:
> On Thu, 2011-12-15 at 17:44 +0000, Ian Jackson wrote:
>> I have been thinking about this whole area.
>>
>> Originally my opinion was that the block and network setup scripts
>> (eg, setting up loopback devices or iscsi or bridging whatever) should
>> be executed directly by xl. ÂThis naturally gives the best error
>> handling and makes logging easiest.
>>
>> However, if we are serious about supporting driver domains, or indeed
>> any kind of backend not running in the same domain as libxl, then
>> something in the driver domain needs to be responsible for executing
>> these scripts (or equivalent).
>>
>> The obvious way to communicate this information to the driver domain
>> is via xenstore.
>>
>> What we should be discussing is exactly how the driver domain
>> translates that into script execution. ÂCurrently on Linux this is
>> mostly done with udev, and AIUI on BSD using xenbackendd.
>
> xenbackendd is (AFAIK) watching for events (e.g. creation and deletion
> of backend state nodes) in xenstore so at least in concept it is
> portable to all OSes, whereas udev is pretty Linux specific.
>
> One nice effect of the udev approach is that you get an explicit event
> e.g. when a block device is torn down. This makes it trivial to avoid
> problems like racing to teardown a loopback device.

I agree that executing scripts from xl is much more comfortable, and
it would be good to do it that way whenever possible. It allows easier
error detection and control of when hotplug scripts are executed.

> Since xenbackendd is mostly inferring things by watching the backend
> nodes used by the kernel side drivers it struggles due to the model
> where we rm the backend's xenstore directory on destroy which nukes
> state required by xenbackendd.
> The xenbackendd model seems slightly preferable to me, it's simpler to
> setup and more portable. Doing things differently on different OSes
> doesn't help with the confusion here!

xenbackend is a quite simple implementation, it only watches backend
changes and reacts upon them. It is easy to implement, but it is
difficult to debug, since hotplug execution is tied to xenstore
events, and right now there's no way to synchronize the toolstack with
xenbackend (xenbackend is only synchronized with the kernel).

> Perhaps the core functionality could be in libxl such that a toolstack
> can choose to integrate the functionality or run it as a separate daemon
> as necessary?

It would be interesting to implement all hotplug call functions into
libxl, and have a configuration option at xl.conf that specifies if
hotplug scripts should be executed from xl directly or delegate the
execution to xenbackend, that should be rewritten to use the same
libxl functions, to avoid having duplicate code.

> The main issue with the xenbackendd model is the manner in which it has
> to guess what is going on and try to synchronise with what the drivers
> are going. This does also effect the udev driven way of doing things too
> since the scripts often want to look in xenstore for parameters and on
> destroy they are often missing.
>
> We currently get this wrong under Linux at the minute and the network
> teardown hotplug script doesn't work right because the necessary bits
> are gone from xenstore. We mostly get away with this for bridging since
> the device is automatically remove from the bridge but for vswitch we do
> need to actually do some reconfiguration at this point. We also appear
> to leak iptables rules under some circumstances.
>
> Perhaps we would be better off splitting the hotplug stuff into its own
> place in xenstore and have the toolstack explicitly trigger events at
> the right time by writing to it? This could be compatible with existing
> scripts since they use an environment variable to find their xenstore
> base path.

Currently the only possible way to synchronize xenbackend and the
toolstack is to watch the hotplug-status xenbackend entry, I've posted
a patch a long time ago, that tried to synchronize xl and xenbackend
using the hotplug-status xenstore entry.

> We need to be aware that the script models (or at least the sequencing
> wrt the xenstore state transitions) vary slightly with the device type
> since for block devices you typically need to run the hotplug script
> before creating the backend whereas with network devices you create the
> device first and then run the script to configure it. vice versa for
> destroy.

I think that the easier way to deal with hotplug scripts is to execute
them when backend state is 2, that is true for both network and block
scripts (at least on NetBSD).

> We also need to consider Network tap devices. Not all tap devices are
> part of Xen's world which is a wrinkle which needs thought.
>
> I tried to write down my understanding of the events and sequencing
> under Linux but that mostly showed the gaps in my understanding...
>
> AIUI XCP does a lot more of this stuff via xenstored and have been
> (successfully) playing with driver domains -- we should ask for their
> input.
>

After all this, what seems most suitable to me is to have hotplug
script calling functions inside libxl, modify xenbackend to use libxl
API, and give the user the option to execute hotplug scripts directly
from xl (calling libxl hotplug functions) or delegate the work to
xenbackend (writing a value to xenstore that xenbackend can use to
synchronize and triggers execution).

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel