[Xen-devel] SCF configuration followup

[Andrii Anisov <andrii_anisov@xxxxxxxx>]

Dear All,

During the developers summit a Shared Coprocessor Framework (SCF) 
concept was presented. One of the topics discussed was a shared 
coprocessor configuration.
The SCF itself is intended to share coprocessor (i.e. GPU, DPS, FPGA, 
whatever) resources to guest domains [1][2].
We are trying to make the SCF as common as possible. So we have to 
define a common and clear approach of its configuration. Under 
“coprocessor configuration” we mean description of hardware resources: 
MMIO ranges, interrupt requests, clocks, etc. We are keeping in mind not 
only DT but ACPI as well (despite the fact that ACPI mostly terra 
incognita to us). Yet we have the future plans to spread to x86. Also 
service of sharing coprocessor resources for VM in ARM server racks 
could be another ACPI system use-case.
Currently the shared coprocessors configuration description problem 
could be splitted into three parts:
1. configuring which coprocessors should be shared (for hypervisor itself)
2. configuring which shared coprocessors instances should be available 
for Dom0 (and their details)
3. configuring which shared coprocessors instances should be available 
for DomU (and their details)
Points 1 and 3 are relevant both for embedded and server use-cases.
Point 2 seems to be relevant only for embedded systems. For server 
systems a servicing OS in Dom0 likely don't need a shared coprocessor 
resources.
The coprocessor configuration for guests is used in two different ways:
* configuring a virtual coprocessor for specific domain in hypervisor
* building a hardware description for that domain (e.g. a device tree 
passed to domU)
Notes for following reading:
* by the current design, a coprocessor to be shared will not be assigned 
by default to any domain.
* the complex coprocessor means a coprocessor which has several 
separated MMIO ranges and several IRQs
* coprocessor remapping means coprocessor’s MMIO ranges can not be used 
by their base addresses, so we have to find and configure different bases.
Let us take a closer look at selection of possible implementation 
approaches for each point listed above:
== Configuring which coprocessors should be shared ==
* List coprocessors to be shared in the hypervisor command line.
    + this should be pretty universal and suitable both for DT and ACPI 
systems.
        - hypervisor command line could be pretty long
* Mark coprocessors to be shared with specific properties in DT [3].
    + such configuration format seems to be more convenient for developers
    - will not work for ACPI systems

== Configuring which shared coprocessors should be available for Dom0 
and their parameters ==
* List shared coprocessors instances assigned to Dom0 in the hypervisor 
command line.
    + this should be pretty universal and suitable both for DT and ACPI 
systems.
       - hypervisor command line could be pretty long
    - remapping of a shared coprocessor (especially complex) would make 
a command line even longer
* Describe shared coprocessors instances assigned to Dom0 as a special 
nodes in DT [3].
    + seems to be more clear for understanding
    + description of shared coprocessor remapping will be easy and clear
    - will not work for ACPI systems

== Сonfiguring which shared coprocessors instances should be available 
for DomU ==
* Passing the shared coprocessor configuration using partial device tree 
[3]. Device tree blob to be passed to XEN hypervisor in order to let SCF 
pick all required data, in libxl the same blob will be processed in 
order to prepare dtb for DomU.
    + seems to be more clear for understanding
    + allows clear description of complex coprocessors sharing 
(including remapping)
    - will not work for ACPI systems
    - only trusted device trees must be used due to risk of exploiting 
through libfdt [4]
* Describe shared coprocessor instance resources solely in domain 
configuration file. Passing each coprocessor property by individual 
domctl from tools to hypervisor.
+ this should be pretty universal and suitable both for DT and ACPI systems
- description format design would be repeating DT if targeting 
flexibility in complex coprocessors sharing configurations (i.e. remapping)
    - high development efforts are expected

* Describe shared coprocessor instances by its path only in a domain 
configuration file.  Use hypercalls to query SCF driver about resources 
needed for coprocessor.
    + this should be pretty universal and suitable both for DT and ACPI 
systems
    + description format expected to be lean
    - high development efforts are expected


In order to take some decision we need some advices from beyond of our 
current experience on following topics:
* How is a coprocessor described in ACPI, is it somehow equivalent to DT?
* How complex could be a coprocessor description (f.e. DSP description 
here [5] has three MMIO regions), how common are such coprocessors?
* Is coprocessors remapping scenario really needed?

We are looking forward to any kind of feedback and will be glad to 
collaborate on the above.
[1] 
https://xendeveloperanddesignsummit2017.sched.com/event/Aj8l/keynote-shared-coprocessor-framework-on-arm-oleksandr-andrushchenko-epam-systems
[2] 
https://lists.xenproject.org/archives/html/xen-devel/2016-10/msg01966.html
[3] 
https://lists.xenproject.org/archives/html/xen-devel/2017-05/msg00348.html
[4] 
https://lists.xenproject.org/archives/html/xen-devel/2017-05/msg00365.html
[5] 
http://git.omapzoom.org/?p=kernel/omap.git;a=blob;f=arch/arm/boot/dts/dra7.dtsi;h=a05300c64bf381f6bc1431e9181a8bd63616267b;hb=862436d35b0d5d88167c4abb6cf8746f6274f3ce#l1032


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel