[Xen-devel] [ARM] Native application design and discussion (I hope)

[Volodymyr Babchuk <vlad.babchuk@xxxxxxxxx>]

Hello all,

I want to discuss EL0 (native) applications for XEN. This will be relatively
long e-mail with requirements, proposed design and my PoC results.

So, why we want XEN native applications in the first place? I see the following
reasons:

1. Isolation. I see XEN as a sort of micro-kernel, so there are no place for
device drivers, emulators, specific SMC handlers, hypervisor extension, etc..

2. Modularity. Just look at Linux kernel. Obviously, for different devices we
can load different drivers.

3. Performance. Native application should be faster than stub domain, or there
will be no sense in it.

4. Ease of use. I want to make call to EL0 app as easy as possible.
Ideally - as a function call.

Actually, no one wants extra code in hypervisor, so reasons (1) and (2) are most
important. I know that there was tries to do such thing in x86 but with
different approach. I want describe my idea for arm64.

Native application is an another domain type. It has own vCPU (only one at this
moment) Native app is loaded as any other kernel, using ELF loader.
It looks like another stub-domain such as MiniOS, but there are two big
differences:

1. MiniOS has event loop that serves requests from hypervisor. Native
application does not has such loop. It has one entry point where you jump every
time when you need something from it.

2. Native application runs in EL0 mode, so it does not have access to MMU,
it can't handle vIQRs, exceptions and so on. XEN does all this for it.

You can find example native application at [1]. I used exactly this one to
benchmark my implementation. Mostly it is inspired by approach used in TEE.
Actually, I took some code directly from OP-TEE Trusted Application library.
In app_entry.c you can find entry point - __app_entry(). It takes function
number and some parameters that will be passed to a function. I probably going
to change ABI a bit, but basic idea will be the same.

Function number will be something like APP_INIT, APP_HANDLE_SMC
or APP_HANDLE_MMIO... I think you got the idea. I also implemented two syscalls
(via old plain SVC instruction). app_log() writes to XEN log, app_return() exits
from application back to hypervisor. We will need other syscalls like
app_call_smc(), app_map_guest_page(), app_map_io(), etc.

Now, back to XEN. Classic way to handle something with stubdomain is to
write request to a ring buffer, fire an event through event channel, that will
trigger vIRQ in stubdomain and stubdomain's vCPU will be scheduled to handle
a request. Problem it that you can't control scheduler, so you don't know
when your request will be really handled, which in not fine in some
embedded use cases.

There is how I see handling requests with native application:

0. Hypervisor pauses requester vCPU
1. Hypervisor either passes parameters via registers or writes request to a
shared page/ring buffer.
2. Then in sets PC of native app vCPU to entry point and initializes r0-r7
with function code and other parameters.
3. Hypervisor switches context to native app vCPU
4. When native app finishes request handling it calls special syscall app_exit()
5. Hypervisor analyses return code, updates requester vCPU state (if needed),
switches back to that vCPU, unpauses it.

Most of that was done at [2]. Most interesting part is in arch/arm/domain.c
There are functions call_el0_app() and return_from_el0_app() that do most
of the work. Also I have added syscalls handlers (in the same way,
as hypercalls are handled). You can find them at xen/arch/arm/syscall.c

At this moment entry point is hardcoded and you need to update it every time
you rebuild native application. Also there are no actual parameters passed.
Also, whole code is a piece of gosa, because it was first time I hacked XEN.

I don't want to repeat benchmark results, because they already was posted in ML.
You can find them at [3].

I understand that I have missed many things:

1. How to ship and load native app, because some of them will be needed even
before dom0 is created.

2. How to distinguish multiple native apps

3. Concurrency in native apps

4. How to restart misbehaved apps.

But at this moment I want to discuss basic approach. If there are will be no
objections against basic concept, then we can develop details.

[1] https://github.com/lorc/xen_app_stub - native app
[2] https://github.com/lorc/xen/tree/el0_app - my branch with PoC
[3] http://marc.info/?l=xen-devel&m=149088856116797&w=2 - benchmark results


-- 
WBR Volodymyr Babchuk aka lorc [+380976646013]
mailto: vlad.babchuk@xxxxxxxxx

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel