Re: [Xen-devel] [RFC] Shared coprocessor framework

[Artem Mygaiev <joculator@xxxxxxxxx>]

Let me explain a bit background of this work.

We see growing amount of use cases for different "co-processors" like
 - GPUs (inside of most modern SoCs)
 - low-power side CPU cores (like ARM Cortex M or R on board with
Cortex A cores to handle PM or other tasks)
 - DSPs (for example, TI C6000 family DSP core inside of Jacinto 6 SoC)
 - FPGAs (Altera or Xilinx SoCs = ARM+FPGA)

These cores most often used standalone for some specific function, but
quite often there is a need to "virtualize" such co-processor together
with the main CPU cores. For example, they may be used in some
virtualized heterogeneous computing environment so there shall be some
sort of an independent "context" of a co-processor associated with a
VM which interacts with it. In some cases, addressing security and
stability requirements, "context" means not only "data" but also
"code" (firmware); i.e. when VMs switch on main CPU, both code and
data memory shall switch on co-processor.

Couple examples when VMs run on same SoC, both want to use some
co-processor in data-intense tasks with different data sets and with
different firmware images and ensure isolation (no data is leaked or
code corrupted through co-processor's memory access) and stability
(restart of one system does not lead to crash of another):
1. use GPU for GL rendering in one VM, another for NN state computing
2. use DSP for HW-accelerated media decoding, another for video image
processing (object recognition, etc.)

We already see a need for enabling such cases in Embedded/Automotive
space (mostly dominated by ARM now), but also this might fit generic
computing in heterogeneous environments - different co-processors are
now deployed alongside generic CPUs in server environments (Google use
own tensor processors for NN computing acceleration, Microsoft used
Altera's FPGAs in project Catapult, there are deployments of GPU
computing nodes in some clouds, etc.)

Hope that makes sense

Best regards,
Artem Mygaiev


On Mon, Oct 31, 2016 at 9:31 PM, Andrii Anisov <andrii.anisov@xxxxxxxxx> wrote:
>> Thankyou for the design doc.  An immediate +1 from me, simply for the
>> doc existing :)
>
> Thank you for you interest and comments.
>
>> Forgive my ignorance (I am an x86 person, and given the CC list, I guess
>> this is talking about ARM systems), but what are coprocessors and what
>> might I do with one?
>
> Well coprocessor could be a some processing unit inside a SoC which is
> running some firmware and supplementing primary processor functions.
> F.e. GPU, DSP, some FPGA inside a SoC.
> The living example is a GPU sharing implemented for the ARM based SoC.
> BTW, the xengt implements pretty close approach and is a pure x86
> world solution.
>
>> > It is targeted capability of different domains to run concurrently 
>> > different
>> > firmware on the coproc.
>> I cant parse this sentence.  I presume you mean that the purpose of this
>> framework is to provide a mechanism for Xen to share a coprocessors
>> resource between multiple domains?
>
> Maybe it should be reworded. I mean that coprocessors are entities
> which are running some firmware to perform their tasks. So different
> domains in their time slice could run different versions of firmware
> on the same coprocessor.
> It is mentioned here to stress that domains contexts are totally
> independent (both for processed data and for firmware code).
>
>> Grammar nit.  Either "Provide coprocessor resource sharing between..."
>> or "Provide sharing of coprocessor resources between..."
>
> Will take "Provide sharing of coprocessor resources between...".
>
>
>> Does it need to only be configurable at system startup?  There is often
>> more flexibility by having a default configuration at system start (so
>> dom0 can use the resources), which can later be altered by toolstack policy.
>
> I did mean that hypervisor, what starts first, checks what
> coprocessors within the system would be shared, own them and provide
> to a framework. Providing some of those resources to dom0 would not a
> big deal: just assign a vcoproc to the domain. And yes, it could be a
> default configuration.
>
>>
>> Considering the latter option, even if you don't implement support at
>> first, tends to lead to a cleaner design, but of course it does depend
>> heavily on the details of the situation.
>
> Definitely we would tailor the design along with an implementation.
>
>> > * MMU-enabled SoC with MMU-protected coprocessor(s)
>> Right - definitely ARM then, but it took me until half way through the
>> document to work this out.
>
> You know, it was specified ARM based SoC here. At some point it was
> removed such a dependency. Inspired by the already mentioned xengt.
>
>> I would be tempted to extend this slightly, and specify that there
>> should be a mechanism for the toolstack to query all of this information
>> at arbitrary points in time.
>
> It should be covered here:
>>
>> > Runtime configuration of scheduling algorithm per instance of shared
>> > --------------------------------------------------------------------
>> > coprocessor
>> > -----------
>> >
>> > There will be a initial domain tool implemented which will provide a CLI 
>> > for
>> > runtime monitoring of shared coprocessors instances available, per-domain/
>> > per-coproc instance list of vcoproc and runtime setup of the scheduler per
>> > each coprocessor instance.
>
> Maybe it needs some rewording.
>
>> Do you mean that, ideally, Xen can fully context switch a coprocessor
>> behind the back of domU, and the domU driver need not know or care about
>> the difference?
> You've got the point.
>
>> And, where that isn't possible, some virtual hooks could be introduced
>> to the domU driver so domU can opt into sharing when it has a compatible
>> driver?
> Yes, due to specifics of SoC implementation pure solution could be
> inefficient, not secure or even impossible. In this case
> drivers/firmware could be modified to cooperate with coprocessor
> sharing framework in XEN.
> This part could be architecture (coprocessor) specific, or machine
> (SoC) specific.
>
> Sincerely,
> Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel