Re: [Embedded-pv-devel] [Xen-devel] [RFC] Shared coprocessor framework

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 28/10/16 18:53, Andrii Anisov wrote:
> Dear All,
>
> At this moment we are designing a shared coprocessor framework to be
> introduced into XEN hypervisor.
> Please review an initial draft of the requirements and design overview.

Thankyou for the design doc.  An immediate +1 from me, simply for the
doc existing :)

>
> =======================================================================
>
> Shared coprocessor framework
> ============================
>
> The main idea of the coprocessor sharing is to let different domains use
> coprocessor concurrently and independently within some time share.

Forgive my ignorance (I am an x86 person, and given the CC list, I guess
this is talking about ARM systems), but what are coprocessors and what
might I do with one?

I think you might want a brief introduction sentence or two describing
what kind of systems this is applicable to, and an example of the kind
of thing which might want to be shared.

>
> It is targeted capability of different domains to run concurrently different
> firmware on the coproc.

I cant parse this sentence.  I presume you mean that the purpose of this
framework is to provide a mechanism for Xen to share a coprocessors
resource between multiple domains?

> This target implies that there is a way to switch
> coproc execution context externally from the main CPU which is running a
> hypervisor.
>
> Сomplexity and predictability of a coprocessor externally issued context
> switching process defines the accuracy of the coproc scheduling and resources
> allocation for domains as well as scheduling overhead.
>
> Shared coprocessor framework requirements
> =========================================
>
> Functional
> ----------
> The framework shall:
> * provide a coprocessor resources sharing between different domains

Grammar nit.  Either "Provide coprocessor resource sharing between..."
or "Provide sharing of coprocessor resources between..."

> * support a number of coprocessor instances shared between domains in runtime
> * support configurable on system startup shareable coprocessors configuration
>   (number of instances, type of instances, scheduling algorithm per
>   coprocessor instance, etc)

Does it need to only be configurable at system startup?  There is often
more flexibility by having a default configuration at system start (so
dom0 can use the resources), which can later be altered by toolstack policy.

Considering the latter option, even if you don't implement support at
first, tends to lead to a cleaner design, but of course it does depend
heavily on the details of the situation.

> * support configurable on domain startup list of coprocessors shared to this
>   particular domain
> * support runtime configuration of scheduling algorithm per instance of shared
>   coprocessor
>
> Non-functional
> --------------
> * The framework will provide an interface to integrate different coprocessor
>   support implementation
> * The framework will provide an interface to integrate different scheduling
>   algorithms implementation.
> * (Optional) The coprocessor firmware and drivers should not be changed in
>   order to support the sharing.
>
> Operation environment
> ---------------------
> * XEN 4.8+
>
> Assumptions and dependencies
> ----------------------------
> * MMU-enabled SoC with MMU-protected coprocessor(s)

Right - definitely ARM then, but it took me until half way through the
document to work this out.

>
> Design overview
> ===============
>
> The Shared coprocessor framework will be implemented as a part of the XEN
> hypervisor. The framework design and development will be based on master
> branch not earlier than 4.8.0-rc2 targeting upstreaming before XEN 4.9 
> release.
>
> Following are described design overview addressing requirements:
>
> Coprocessors resources sharing mechanism
> ----------------------------------------
>
> The coprocessor resources sharing will be done in a way of switching context
> (per domain instance of firmware running) on the coprocessor issued by the
> main processor. It is done by a hypervisor running on the main cpu and aware
> about domains, their capabilities and needs.
>
> The hypervisor is aware about coprocessors to be shared. For each shared
> coprocessor there are: an assigned timer, scheduler and per-domain vcoproc
> instances list.
>
> It will be implemented a vcoproc scheduling mechanism independent of XEN VCPU
> scheduling mechanism. It is caused by the difference in vcoproc and vcpu
> nature:
> * it is not considered pool of equal vcoproc running on a pool of equal
>   coprocessors (SMP like behavior)
> * due to fact that scheduler is not running on the coprocessor itself:
>     * a vcoproc scheduler does not preempt the coprocessor context
>     * coprocessor could deny context switching at the moment scheduler decide
>     so, scheduling mechanism should adopt that
> * there could be number of different coprocessors instances with scheduler and
>   vcoproc queue associated
>
> Due to the fact that some domain assigned a vcoproc could access coproc when
> it is running another domain context, framework will implement iomem access
> emulation for domains which are not provided coproc at the moment of access.
>
> Interrupts from coprocessor will be effectively delivered to the domain which
> vcoproc context is being running at the moment of the interrupt.
>
> While the context switching is coprocessor and SoC specific procedure, the
> framework would not encapsulate such procedure, but provides interface to
> register platform implementation of context switching procedure.
>
> It is assumed that the coprocessor has an iommu associated or corresponding
> system level iommu (i.e. arm smmu) owned by hypervisor. The iommu setup is
> also platform specific code and part of context switching.
>
> In case of only having coprocessor associated iommu, platform code is also
> responsible to handle that iommu setup (i.e. in a way of iomem access
> emulation).
>
> Support a number of coprocessor instances shared between domains in runtime
> ---------------------------------------------------------------------------
>
> The coprocessor framework will have a list of coprocessors available for
> sharing during system runtime. Any running domain could be using none, one or
> several shared coprocessors.

I would be tempted to extend this slightly, and specify that there
should be a mechanism for the toolstack to query all of this information
at arbitrary points in time.

>
> System startup configuration
> ----------------------------
>
> The compilation time configuration will provide separate config for the shared
> coprocessor framework support, configs enabling scheduling algorithms, configs
> enabling support of the coprocessor platform support.
>
> The system startup configuration will be described in the system device tree
> in the xen "choosen" node. The configuration should at least describe which
> coprocessors within the system will be shared to the domains, a scheduling
> algorithm assigned to each shared coprocessor, and an identificator of this
> particular coprocessor instance. Hypervisor will read corresponding nodes and
> configure the coprocessor framework with platform implementations on startup.
>
> Domain startup configuration
> ----------------------------
>
> In a domain configuration file it will be specified which particular instance
> of the coprocessor should be shared to this domain.
>
> The shared coprocessor domain configuration should be read by tools/xen and on
> domain creation a vcoproc should be created and assigned to vcoproc instances
> list of the particular coprocessor. Starting from here it is assumed that
> vcoproc is ready to be scheduled as soon as domain starts using it.
>
> In case of failure of vcoproc creation, the domain creation should be aborted.
>
> Runtime configuration of scheduling algorithm per instance of shared
> --------------------------------------------------------------------
> coprocessor
> -----------
>
> There will be a initial domain tool implemented which will provide a CLI for
> runtime monitoring of shared coprocessors instances available, per-domain/
> per-coproc instance list of vcoproc and runtime setup of the scheduler per
> each coprocessor instance.
>
> Integration of different coprocessor support implementation
> -----------------------------------------------------------
>
> The coprocessor sharing framework is a platform independent entity which
> implements generic flows and actions. It does need platform specific hooks
> which implement exact coprocessor management operations.
> The coprocessor platform specific implementations will provide:
> * domain create/destroy hooks
> * context switch hooks (coprocessor start, stop, context load/save)
> * iomem handler(s)
> * interrupt handler(s)
>
> Integration of different scheduling algorithms implementation
> -------------------------------------------------------------
>
> There are number of possible coprocessor sharing scenarios with different
> needs of coprocessor resources allocation for different domains. Within
> current design the coprocessor resource allocation to domains is a matter of
> scheduling algorithm and its configuration. The framework will use the
> scheduling algorithm to determine if context switching should be done and what
> is the next vcoproc to be ran on coprocessor.
>
> The algorithm impelementation will provide:
> * initialization/deinitialization hooks
> * scheduling operations hooks
>
> (Optional) The coprocessor firmware and drivers should not be changed in order
> ------------------------------------------------------------------------------
> to support the sharing.
> -----------------------
>
> This requirement could be compromised for implementations which support
> coprocessors not designed for externally issued context switching. Such
> context switch function support would be introduced into the firmware and the
> driver of the coprocessor.

Do you mean that, ideally, Xen can fully context switch a coprocessor
behind the back of domU, and the domU driver need not know or care about
the difference?

And, where that isn't possible, some virtual hooks could be introduced
to the domU driver so domU can opt into sharing when it has a compatible
driver?

~Andrew

_______________________________________________
Embedded-pv-devel mailing list
Embedded-pv-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/cgi-bin/mailman/listinfo/embedded-pv-devel