Re: [Xen-devel] [PATCH 12 of 14 v4] libxl: set frontend status to 6 on domain destroy

[Ian Campbell <Ian.Campbell@xxxxxxxxxx>]

On Thu, 2011-12-15 at 17:44 +0000, Ian Jackson wrote:
> I have been thinking about this whole area.
> 
> Originally my opinion was that the block and network setup scripts
> (eg, setting up loopback devices or iscsi or bridging whatever) should
> be executed directly by xl.  This naturally gives the best error
> handling and makes logging easiest.
> 
> However, if we are serious about supporting driver domains, or indeed
> any kind of backend not running in the same domain as libxl, then
> something in the driver domain needs to be responsible for executing
> these scripts (or equivalent).
> 
> The obvious way to communicate this information to the driver domain
> is via xenstore.
> 
> What we should be discussing is exactly how the driver domain
> translates that into script execution.  Currently on Linux this is
> mostly done with udev, and AIUI on BSD using xenbackendd.

xenbackendd is (AFAIK) watching for events (e.g. creation and deletion
of backend state nodes) in xenstore so at least in concept it is
portable to all OSes, whereas udev is pretty Linux specific.

One nice effect of the udev approach is that you get an explicit event
e.g. when a block device is torn down. This makes it trivial to avoid
problems like racing to teardown a loopback device.

Since xenbackendd is mostly inferring things by watching the backend
nodes used by the kernel side drivers it struggles due to the model
where we rm the backend's xenstore directory on destroy which nukes
state required by xenbackendd. 
The xenbackendd model seems slightly preferable to me, it's simpler to
setup and more portable. Doing things differently on different OSes
doesn't help with the confusion here!

Perhaps the core functionality could be in libxl such that a toolstack
can choose to integrate the functionality or run it as a separate daemon
as necessary?

The main issue with the xenbackendd model is the manner in which it has
to guess what is going on and try to synchronise with what the drivers
are going. This does also effect the udev driven way of doing things too
since the scripts often want to look in xenstore for parameters and on
destroy they are often missing.

We currently get this wrong under Linux at the minute and the network
teardown hotplug script doesn't work right because the necessary bits
are gone from xenstore. We mostly get away with this for bridging since
the device is automatically remove from the bridge but for vswitch we do
need to actually do some reconfiguration at this point. We also appear
to leak iptables rules under some circumstances.

Perhaps we would be better off splitting the hotplug stuff into its own
place in xenstore and have the toolstack explicitly trigger events at
the right time by writing to it? This could be compatible with existing
scripts since they use an environment variable to find their xenstore
base path.

We need to be aware that the script models (or at least the sequencing
wrt the xenstore state transitions) vary slightly with the device type
since for block devices you typically need to run the hotplug script
before creating the backend whereas with network devices you create the
device first and then run the script to configure it. vice versa for
destroy.

We also need to consider Network tap devices. Not all tap devices are
part of Xen's world which is a wrinkle which needs thought.

I tried to write down my understanding of the events and sequencing
under Linux but that mostly showed the gaps in my understanding...

AIUI XCP does a lot more of this stuff via xenstored and have been
(successfully) playing with driver domains -- we should ask for their
input.

> So sorry for leading this discussion in what I now think may be a
> wrong direction.
> 
> Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel