[Xen-devel] [RFC] Shared coprocessor framework

[Andrii Anisov <andrii.anisov@xxxxxxxxx>]

Dear All,

At this moment we are designing a shared coprocessor framework to be
introduced into XEN hypervisor.
Please review an initial draft of the requirements and design overview.

=======================================================================

Shared coprocessor framework
============================

The main idea of the coprocessor sharing is to let different domains use
coprocessor concurrently and independently within some time share.

It is targeted capability of different domains to run concurrently different
firmware on the coproc. This target implies that there is a way to switch
coproc execution context externally from the main CPU which is running a
hypervisor.

Сomplexity and predictability of a coprocessor externally issued context
switching process defines the accuracy of the coproc scheduling and resources
allocation for domains as well as scheduling overhead.

Shared coprocessor framework requirements
=========================================

Functional
----------
The framework shall:
* provide a coprocessor resources sharing between different domains
* support a number of coprocessor instances shared between domains in runtime
* support configurable on system startup shareable coprocessors configuration
  (number of instances, type of instances, scheduling algorithm per
  coprocessor instance, etc)
* support configurable on domain startup list of coprocessors shared to this
  particular domain
* support runtime configuration of scheduling algorithm per instance of shared
  coprocessor

Non-functional
--------------
* The framework will provide an interface to integrate different coprocessor
  support implementation
* The framework will provide an interface to integrate different scheduling
  algorithms implementation.
* (Optional) The coprocessor firmware and drivers should not be changed in
  order to support the sharing.

Operation environment
---------------------
* XEN 4.8+

Assumptions and dependencies
----------------------------
* MMU-enabled SoC with MMU-protected coprocessor(s)

Design overview
===============

The Shared coprocessor framework will be implemented as a part of the XEN
hypervisor. The framework design and development will be based on master
branch not earlier than 4.8.0-rc2 targeting upstreaming before XEN 4.9 release.

Following are described design overview addressing requirements:

Coprocessors resources sharing mechanism
----------------------------------------

The coprocessor resources sharing will be done in a way of switching context
(per domain instance of firmware running) on the coprocessor issued by the
main processor. It is done by a hypervisor running on the main cpu and aware
about domains, their capabilities and needs.

The hypervisor is aware about coprocessors to be shared. For each shared
coprocessor there are: an assigned timer, scheduler and per-domain vcoproc
instances list.

It will be implemented a vcoproc scheduling mechanism independent of XEN VCPU
scheduling mechanism. It is caused by the difference in vcoproc and vcpu
nature:
* it is not considered pool of equal vcoproc running on a pool of equal
  coprocessors (SMP like behavior)
* due to fact that scheduler is not running on the coprocessor itself:
    * a vcoproc scheduler does not preempt the coprocessor context
    * coprocessor could deny context switching at the moment scheduler decide
    so, scheduling mechanism should adopt that
* there could be number of different coprocessors instances with scheduler and
  vcoproc queue associated

Due to the fact that some domain assigned a vcoproc could access coproc when
it is running another domain context, framework will implement iomem access
emulation for domains which are not provided coproc at the moment of access.

Interrupts from coprocessor will be effectively delivered to the domain which
vcoproc context is being running at the moment of the interrupt.

While the context switching is coprocessor and SoC specific procedure, the
framework would not encapsulate such procedure, but provides interface to
register platform implementation of context switching procedure.

It is assumed that the coprocessor has an iommu associated or corresponding
system level iommu (i.e. arm smmu) owned by hypervisor. The iommu setup is
also platform specific code and part of context switching.

In case of only having coprocessor associated iommu, platform code is also
responsible to handle that iommu setup (i.e. in a way of iomem access
emulation).

Support a number of coprocessor instances shared between domains in runtime
---------------------------------------------------------------------------

The coprocessor framework will have a list of coprocessors available for
sharing during system runtime. Any running domain could be using none, one or
several shared coprocessors.

System startup configuration
----------------------------

The compilation time configuration will provide separate config for the shared
coprocessor framework support, configs enabling scheduling algorithms, configs
enabling support of the coprocessor platform support.

The system startup configuration will be described in the system device tree
in the xen "choosen" node. The configuration should at least describe which
coprocessors within the system will be shared to the domains, a scheduling
algorithm assigned to each shared coprocessor, and an identificator of this
particular coprocessor instance. Hypervisor will read corresponding nodes and
configure the coprocessor framework with platform implementations on startup.

Domain startup configuration
----------------------------

In a domain configuration file it will be specified which particular instance
of the coprocessor should be shared to this domain.

The shared coprocessor domain configuration should be read by tools/xen and on
domain creation a vcoproc should be created and assigned to vcoproc instances
list of the particular coprocessor. Starting from here it is assumed that
vcoproc is ready to be scheduled as soon as domain starts using it.

In case of failure of vcoproc creation, the domain creation should be aborted.

Runtime configuration of scheduling algorithm per instance of shared
--------------------------------------------------------------------
coprocessor
-----------

There will be a initial domain tool implemented which will provide a CLI for
runtime monitoring of shared coprocessors instances available, per-domain/
per-coproc instance list of vcoproc and runtime setup of the scheduler per
each coprocessor instance.

Integration of different coprocessor support implementation
-----------------------------------------------------------

The coprocessor sharing framework is a platform independent entity which
implements generic flows and actions. It does need platform specific hooks
which implement exact coprocessor management operations.
The coprocessor platform specific implementations will provide:
* domain create/destroy hooks
* context switch hooks (coprocessor start, stop, context load/save)
* iomem handler(s)
* interrupt handler(s)

Integration of different scheduling algorithms implementation
-------------------------------------------------------------

There are number of possible coprocessor sharing scenarios with different
needs of coprocessor resources allocation for different domains. Within
current design the coprocessor resource allocation to domains is a matter of
scheduling algorithm and its configuration. The framework will use the
scheduling algorithm to determine if context switching should be done and what
is the next vcoproc to be ran on coprocessor.

The algorithm impelementation will provide:
* initialization/deinitialization hooks
* scheduling operations hooks

(Optional) The coprocessor firmware and drivers should not be changed in order
------------------------------------------------------------------------------
to support the sharing.
-----------------------

This requirement could be compromised for implementations which support
coprocessors not designed for externally issued context switching. Such
context switch function support would be introduced into the firmware and the
driver of the coprocessor.

Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel