Architecture for Split Drivers Within Xen
=========================================

The block, net and TPM Xen drivers are "split" drivers: they have a portion in
a privileged domain handling the physical device -- the backend -- and a
frontend in the unprivileged domain acting as a proxy upon the backend.  The
backend and frontend communicate using shared event channels and ring buffers,
in an architecture known as the Xenbus.

For concreteness, this section shall discuss the block driver, with
differences between that and the net driver highlighted where necessary.  The
block driver is referred to in various places by the shorthands "blk" or
"vbd".  I shall also refer to the privileged domain as domain 0, and the
unprivileged domain as domain U.

The basic architecture can be considered to be a chain of four protagonists, a
pair of devices communicating across the Xenbus, and then a pair of devices
using that bus, handling the device-specific interaction with the kernel
device layer.


priv.                                                                 unpriv.
domain                  Xenbus  interdomain   Xenbus                  domain
kernel -- blkback  --  backend -------------- frontend -- blkfront -- kernel
device    instance    instance  interconnect  instance    instance    device
layer                                                                 layer


In order to establish communication across this chain, a number of parameters
need to be passed from privileged to unprivileged domain, and vice versa.
Some of these parameters are specific to the blkback/blkfront pair, and some
are more general, applying to all split drivers.  All of these parameters are
passed using the Xen Store.


Device Initialisation
---------------------

To trigger the creation of a device connection, Xend (or another tool) writes
frontend and backend details to the store.  These new details are seen by the
Xenbus driver instances, and initialisation begins.

The details to be written are:

/local/domain/0/backend/vbd/U/<deviceID>/...
    frontend       /local/domain/U/device/vbd/<deviceID>
    frontend-id    U
    state          XenbusStateInitialising
    ...            <device-specific details>

/local/domain/U/device/vbd/<deviceID>/...
    backend        /local/domain/0/backend/vbd/U/<deviceID>
    backend-id     0
    state          XenbusStateInitialising
    ...            <device-specific details>


The Xenbus backend instance has a watch on /local/domain/0/backend/vbd and
the frontend instance has a watch on /local/domain/U/device/vbd.  When the
device details above are written, these two watches fire, and the Xenbus
instances begin negotiation.

The backend reads the frontend and frontend-id nodes, and then places a watch
on <frontend>/state.

The frontend reads the backend and backend-id nodes, and then places a watch
on <backend>/state.

These two watches are handled symmetrically inside
xenbus_probe:read_otherend_details and the details are made available in
xenbus_device.otherend, xenbus_device.otherend_id, and
xenbus_device.otherend_watch.

For the backend, the hotplug subsystem is triggered, in order to bring the
physical device online.


Initialisation proceeds by calling the blkfront/blkback probe functions, in
order that they may perform device-specific initialisation, and when this is
complete, each driver will switch to a different state.  blkback creates a
watch on the store, waiting for the hotplug scripts to complete, and switches
to XenbusStateInitWait.  blkfront creates the ring buffer and event channel 
for sharing with the backend, advertises those details in the store, and
switches to XenbusStateInitialised.

When blkback has received the physical device details from the hotplug
scripts, then it creates the necessary connection to the kernel device layer.
When it has received the ring-buffer details from the frontend (indicated by
the frontend state change) then it maps that connection.  When both these
things have happened (in either order) then it writes the physical device
details to the store, for use by the frontend, and then switches to the
Connected state.

When blkfront sees the switch to the Connected state, it can read those
physical device details, connect to the kernel device layer itself, and also
switch to the Connected state.


Event diagram:

              Xenbus                       Xenbus
Hotplug       Backend                     Frontend
-------       -------                     --------

            Initialising                Initialising
                 |                           |
   |<---start----+                           |
   |             |                           |
   |          InitWait                       |
   |             |                         write
   |             |                         ring/
 write           |                        channel
physdets-------->|                        details
                 |                           |
                 |<---------------------Initialised
                 |                           |
               write                         |
              physdets                       |
                 |                           |
             Connected---------------------->|
                 |                           |
                 |                       Connected
                 |                           |

The netfront driver does not need to wait for details from its backend, and so
can skip immediately to the Connected state.


Device Closedown
----------------

Orderly closedown can be requested by the user, as a device hotplug request to
Xend or other tools, or by the drivers when they encounter an error.

An orderly closedown can be accomplished by changing the backend state to
Closing.  This will trigger the frontend to tear down it's kernel connection,
flushing through any requests that it has in flight, and then to change to
state Closed.  The backend will respond to the frontend's change to Closed by 
deregistering itself and switching to state Closed also.  Frontends may tear
down immediately on error, without requiring the backend state to change to
Closing first.


              Xenbus                       Xenbus
Hotplug       Backend                     Frontend
-------       -------                     --------

           (Written by control               |
            tools, e.g. Xend,                |
            or by backend on                 |
            error)                           |
                 |                           |
              Closing-------------------->Closing
                 |                           |
                 |                           |
                 |                         flush
                 |                           |
                 |                           |
               Closed<--------------------Closed
                 |                           |
   |<--------unregister                      |
   |           device                        |
   |             |                           |
 remove
 store
 directories
   |


or

              Xenbus                       Xenbus
Hotplug       Backend                     Frontend
-------       -------                     --------

                 |                    (Written by frontend
                 |                         on error)
                 |                           |
              Closing<--------------------Closing
                 |  +----------------------->|
                 |                           |
                 |                         flush
                 |                           |
                 |                           |
               Closed<--------------------Closed
                 |                           |
   <---------unregister                      |
   |           device                        |
   |             |                           |
 remove
 store
 directories
   |


Migration
---------

Migration differs from closedown in that the connection from frontend driver
to the kernel device layer is not disturbed; only the Xenbus connection is
torn down.  When a driver is disconnected for this reason, it receives a call
from the lower layers.  On resumption, the new backend details are read and
new watches established.


Device reconfiguration
----------------------

If live reconfiguration is required between backend and frontend, this is
handled with device-specific watches on the store.  Each driver stays in the
Connected state throughout this.