Xen Cluster Control Tools
=================================

by Mark Williamson
(C) 2004 Intel Research Cambridge


This document gives an overview of the functionality currently
available and discusses some of the issues involved in the design of a
Xen cluster controller.

Decomposition of functionality
------------------------------

Logically, the control software required for managing Xen servers can
be divided into three components:

Basic control functions:

All Xen nodes, whether they are standalone or part of a cluster will
need a set of basic control functions to be performed locally.

- (de)multiplex console messages
- (de)multiplex other control messages
- any other low-level management functions required
- supervising operations associated with domain migration
- domain reboots
- startup & shutdown domains on request
- suspend / resume domains given a location on disk for the state file
- coredump / reap domains that exit with an error code

Standalone node management:

Standalone Xen servers require some higher-level management
functionality.  This is currently provided by a web interface and by
the xm command-line tool.

This software does:

- automate startup & shutdown domains when the main system starts /
  shuts down
- manage domain memory and disk images within one machine to provide
  an abstraction of a persistent "virtual machine" that the user can
  assign some name to
- recovers from restart, so Xend crashes shouldn't necessarily be fatal to
  domains running on the host

It would be nice to have:
- provide some means to move, copy or remotely access the local disk
  state, etc of a "virtual machine" to another machine, either whilst
  suspended or whilst running, when requested by the user, as well as
  transferring things like memory images using the functionality
  provided by the basic control layer.
  This is to facilitate migration of domains that have state on the local
  disk and could be requested directly by the user, or by the cluster
  controller.

Cluster management:

This software would be used to manage a cluster of heterogeneous Xen
services in a uniform way.

This software should ideally provide:

- global awareness of the cluster (where virtual machines are running, what
  resources are available in the cluster...)
    * resource awareness could also include - free IPs, free MAC addresses
- intelligent application of migration to facilitate load balancing,
  taking down physical hosts for maintenance, etc.
- some means for detecting and recovering from failed cluster nodes.  This
  could sensibly be achieved by polling all running Xends for their current
  state and noting failures to respond.  Once a node has failed a few times,
  the cluster controller would take corrective action or report an error.
- recovery from cluster controller failure, possibly using redundant
  controller nodes
- provide a separate layer of indirection for domain consoles so that migration
  can occur transparently without breaking console TCP connections
- a concept of a virtual machine class and a virtual machine instance
  (i.e. "Fedora dynamic web server" vs. "Fred's server").  Instances can be
  created based on classes.

The last point makes virtual machine management nicer but could be
quite complex in itself.  It is also largely orthogonal to the other
points, which are concerned with managing already-defined virtual
machines.


Thoughts on the implementation
------------------------------

The cluster management software will likely utilise a full-featured
database server to maintain global state for the cluster.  A control
daemon would run on the cluster's "master" machine and would monitor
the cluster and issue commands to nodes on behalf of the user.

The cluster management software in particular is a very complex
problem and so its feature set should be carefully chosen for the
initial implementation in order to ensure that a product is eventually
completed!  Not all of the above is required for it to be useful.