[Xen-devel] [RFD] OP-TEE (and probably other TEEs) support

[Volodymyr Babchuk <vlad.babchuk@xxxxxxxxx>]

Hello all,

My name is Volodymyr Babchuk, I'm working on EPAM Systems with bunch
of other guys like Artem Mygaiev or Andrii Anisov. My responsibility
there is security in embedded systems.

I would like to discuss approaches to OP-TEE support in XEN.
But first small introduction for those who is not familiar with topic:

There are such thing as Security Extensions for ARM cores. It is part
of bigger thing - ARM TrustZone. Security extensions allows CPU to
work in two states: Normal (Non-Secure) and Secure.
Other parts of TrustZone provide hardware protection for peripherals,
memory regions, etc. So, when CPU is running in secure state, it have
access to all peripherals and memories, also it can forbid access to
some of those resources from Normal state. Secure state have own
Supervisor and User modes, so we can run OS there.
Just to be clear, CPU can run in following modes; user, supervisor,
hypervisor, secure monitor, secure user and secure supervisor. Last
two basically are the same as normal USR and SVC, but in secure state.
And Secure Monitor is special mode, that can switch Secure state to
Non-secure state and back. SMC instruction takes into this mode.
So, we can run one OS in secure state and one in non-secure state.
They will be mostly independent from each other. In normal
(non-secure) mode we run, say, Linux. And in Secure mode we run some
Trusted OS (or Secure Os). Secure OS provides services like
cryptography, secure storage, DRM, virtual SIM, etc. to Normal world
OS (it also called "Rich OS". It can be Linux for example.).
There are standard created by GlobalPlatform that is called "Trusted
Execution Environment" (TEE). It defines requirements and APIs for the
Secure OS. There are number number of Secure OSes that tries to be
conform with TEE specification and OP-TEE is one of them.

CPU is normally running in Non-Secure state. When normal world OS
needs some services from Secure OS, it issues SMC instruction with
command data. This instruction puts processor into Secure Monitor
Mode, where secure monitor code examines request. If it is request to
Secure OS, it switches processor to Secure Supervisor mode and passes
request to Secure OS. When Secure OS finishes work, it also issues SMC
instruction and Secure Monitor switches processor back to non-secure
state, returning control to Rich OS.
SMC instruction can be used not only to communicate with SecureOS, but
also for other services like power control, control of processor
cores, etc. ARM wrote document "SMC Calling Convention" which in
detail describes how this instruction can be used.  By the way, SMCCC
requires to pass Virtual Machine ID in one of the registers during SMC
call.

Now let's get back to XEN. We want XEN to provide TEE services to Dom0
and guests. I can see different approaches to this:

 - One of them I call "Emulated TEE". Guests does not have access to
real TEE OS. Instead somewhere (in Dom0) we run instance of TEE for
each of the Guests. This provides perfect isolation, as TEE instances
does not anything about each one. But we will miss all hardware
benefits like cryptographic acceleration.
 - Another way is to allow guests to work with real TEE running Secure
state. In this case TEE should be aware about guests to track shared
memories, opened sessions, etc. It requires some careful programming
to ensure that guest belongings are isolated from each other. But as
reward we are much closer to original TrustZone desing.

Personally I prefer second approach. I, even, did small PoC that
allows different guests to work with OP-TEE (but not simultaneously!).
You can find patches at [1] if you are interested.
During working on this PoC I have identified main questions that
should be answered:

On XEN side:
1. SMC handling in XEN. There are many different SMCs and only portion
of them belong to TEE. We need some SMC dispatcher that will route
calls to different subsystems. Like PSCI calls to PSCI subsystem, TEE
calls to TEE subsystem.

2. Support for different TEEs. There are OP-TEE, Google Trusty, TI
M-Shield... They all work thru SMC, but have different protocols.
Currently, we are aimed only to OP-TEE. But we need some generic API
in XEN, so support for new TEE can be easily added.

3. TEE services. Hypervisor should inform TEE when new guest is
created or destroyed, it should tag SMCs to TEE with GuestID, so TEE
can isolate guest data on its side.

4. SMC mangling. RichOS communicates with TEE using shared buffers, by
providing physical memory addresses. Hypervisor should convert IPAs to
PAs.
Currently I'm rewriting parts of OP-TEE to make it support arbitrary
buffers originated from RichOS.

5. Events from TEE. This is hard topic. Sometimes OP-TEE needs some
services from RichOS. For example it wants Linux to service pending
IRQ request, or allocate portion of shared memory, or lock calling
thread, etc. This is called "RPC request". To do RPC request OP-TEE
initiates return to Normal World, but it sets special return code to
indicate that Linux should do some job for OP-TEE. When Linux finishes
work, it initiates another SMC with code like "I have finished RPC
request" and OP-TEE resumes its execution.
OP-TEE mutexes create problem there. We don't  want to sleep in secure
state, so when OP-TEE thread gets blocked on a mutex, it issues RPC
request that asks calling thread to wait on wait queue. When mutex
owner unlocks it, that another thread also issues RPC to wake up first
thread.
This works perfectly when there are one OS (or one guest). But when
there are many, it is possible that request from one guest blocks
another guest. That another guest will wait on wait queue, but there
will be no one, who can wake it up. So we need another mechanism to
wake up sleeping threads. Obvious candidate is IPI. There are 8
non-secure IPIs that all are used by linux kernel. By there are also 8
secure IPIs. I think OP-TEE can use one of those to deliver events to
Normal World. But this will require changes to OP-TEE, XEN and Linux
kernel.

6. Some mechanism to control which guests can work with TEE. At this
time I have no idea how this should work.

On OP-TEE side:
1. Shared memory redesign which is almost complete.
2. Implement new SMCs  from hypervsiror to TEE to track VM lifecycle.
3. Track VM IDs to isolated VM data.
4. RPCs to sleeping guests.

That is basically all what I currently have on my mind. It will be
great if all interested sides will share their opinions, requirements,
thoughts on this.

[1] https://github.com/lorc/xen/commits/staging-4.7 - Xen + OP-TEE PoC

-- 
WBR Volodymyr Babchuk aka lorc [+380976646013]
mailto: vlad.babchuk@xxxxxxxxx

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel