[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [RFC PATCH 00/15] RFC: SGX virtualization design and draft patches



On 7/17/2017 6:08 PM, Huang, Kai wrote:

Hi Andrew,
Thank you very much for comments. Sorry for late reply, and please see my reply below. 

On 7/12/2017 2:13 AM, Andrew Cooper wrote:

On 09/07/17 09:03, Kai Huang wrote:

Hi all,
This series is RFC Xen SGX virtualization support design and RFC draft patches. 

Thankyou very much for this design doc.


2. SGX Virtualization Design

2.1 High Level Toolstack Changes:

2.1.1 New 'epc' parameter
EPC is limited resource. In order to use EPC efficiently among all domains, when creating guest, administrator should be able to specify domain's virtual 
EPC size. And admin
alao should be able to get all domain's virtual EPC size.
For this purpose, a new 'epc = <size>' parameter is added to XL configuration file. This parameter specifies guest's virtual EPC size. The EPC base address will be calculated by toolstack internally, according to guest's memory size, MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be accepted.
How will this interact with multi-package servers? Even though its fine to implement the single-package support first, the design should be extensible to the multi-package case. 

First of all, what are the implications of multi-package SGX?
(Somewhere) you mention changes to scheduling. I presume this is because a guest with EPC mappings in EPT must be scheduled on the same package, or ENCLU[EENTER] will fail. I presume also that each package will have separate, unrelated private keys?
The ENCLU[EENTE] will continue to work on multi-package server. Actually I was told all ISA existing behavior documented in SDM won't change for server, as otherwise this would be a bad design :)
Unfortunately I was told I cannot talk about MP server SGX a lot now. Basically I can only talk about staff already documented in SDM (sorry :( ). But I guess multiple EPC in CPUID is designed to cover MP server, at lease mainly (we can do reasonable guess).
In terms of the design, I think we can follow XL config file parameters for memory. 'epc' parameter will always specify totol EPC size that the domain has. And we can use existing NUMA related parameters, such as setting cpus='...' to physically pin vcpu to specific pCPUs, so that EPC will be mostly allocated from related node. If that node runs out of EPC, we can decide whether to allocate EPC from other node, or fail to create domain. I know Linux supports NUMA policy which can specify whether to allow allocating memory from other nodes, does Xen has such policy? Sorry I haven't checked this. If Xen has such policy, we need to choose whether to use memory policy, or introduce new policy for EPC.
If we are going to support vNUAM EPC in the future. We can also use similar way to config vNUMA EPC in XL config.
Sorry I mentioned scheduling. I should say *potentially* :). My thinking was as SGX is per-thread, then SGX info reported by different CPU package may be different (ex, whether SGX2 is supported), then we may need scheduler to be aware of SGX. But I think we don't have to consider this now. 

What's your comments?
I presume there is no sensible way (even on native) for a single logical process to use multiple different enclaves? By extension, does it make sense to try and offer parts of multiple enclaves to a single VM?
The native machine allows running multiple enclaves, even signed by multiple authors. SGX only has limit that before launching any other enclave, Launch Enclave (LE) must be launched. LE is the only enclave that doesn't require EINITTOKEN in EINIT. For LE, its signer (SHA256(sigstruct->modulus)) must be equal to the value in IA32_SGXLEPUBKEYHASHn MSRs. LE will generates EINITTOKEN for other enclaves (EINIT for other enclaves requires EINITTOKEN). For other enclaves, there's no such limitation that enclave's signer must match IA32_SGXLEPUBKEYHASHn so the signer can be anybody. But for other enclaves, before running EINIT, the LE's signer (which is equal to IA32_SGXLEPUBKEYHASHn as explained above) needs to be updated to IA32_SGXLEPUBKEYHASHn (MSRs can be changed, for example, when there's multiple LEs running in OS). This is because EINIT needs to perform EINITTOKEN integrity check (EINITTOKEN contains MAC info that calculated by LE, and EINIT needs LE's IA32_SGXLEPUBKEYHASHn to derive the key to verify MAC).
SGX in VM doesn't change those behaviors, so in VM, the enclaves can also be signed by anyone, but Xen needs to emulate IA32_SGXLEPUBKEYHASHn so that when one VM is running, the correct IA32_SGXLEPUBKEYHASHn are already in physical MSRs. 




2.1.3 Notify domain's virtual EPC base and size to Xen
Xen needs to know guest's EPC base and size in order to populate EPC pages for it. Toolstack notifies EPC base and size to Xen via XEN_DOMCTL_set_cpuid.
I am currently in the process of reworking the Xen/Toolstack interface when it comes to CPUID handling. The latest design is available here: https://lists.xenproject.org/archives/html/xen-devel/2017-07/msg00378.html but the end result will be the toolstack expressing its CPUID policy in terms of the architectural layout.
Therefore, I would expect that, however the setting is represented in the configuration file, xl/libxl would configure it with the hypervisor by setting CPUID.0x12[2] with the appropriate base and size.
I agree. I saw you are planning to introduce new XEN_DOMCTL_get{set}_cpuid_policy, which will allow toolstack to query/set cpuid policy in single hypercall (if I understand correctly), so I think we should definitely use the new hypercalls.
I also saw you are planning to introduce new hypercall to query raw/host/pv_max/hvm_max cpuid policy (not just featureset), so I think 'xl sgxinfo' (or xl info -sgx) can certainly use that to get physical SGX info (EPC info). And 'xl sgxlist' (or xl list -sgx) can use XEN_DOMCTL_get{set}_cpuid_policy to display domain's SGX info (EPC info).
Btw, do you think we need 'xl sgxinfo' and 'xl sgxlist'? If we do, which is better? New 'xl sgxinfo' and 'xl sgxlist', or extending existing 'xl info' and 'xl list' to support SGX, such as 'xl info -sgx' and 'xl list -sgx' above? 





2.1.4 Launch Control Support (?)
Xen Launch Control Support is about to support running multiple domains with each running its own LE signed by different owners (if HW allows, explained below). As explained in 1.4 SGX Launch Control, EINIT for LE (Launch Enclave) only succeeds when SHA256(SIGSTRUCT.modulus) matches IA32_SGXLEPUBKEYHASHn, 
and EINIT for other enclaves will derive EINITTOKEN key according to
IA32_SGXLEPUBKEYHASHn. Therefore, to support this, guest's virtual
IA32_SGXLEPUBKEYHASHn must be updated to phyiscal MSRs before EINIT (which also means the physical IA32_SGXLEPUBKEYHASHn need to be *unlocked* in BIOS 
before booting to OS).
For physical machine, it is BIOS's writer's decision that whether BIOS would provide interface for user to specify customerized IA32_SGXLEPUBKEYHASHn (it is default to digest of Intel's signing key after reset). In reality, OS's SGX driver may require BIOS to make MSRs *unlocked* and actively write the hash value to MSRs in order to run EINIT successfully, as in this case, the driver will not depend on BIOS's capability (whether it allows user to customerize 
IA32_SGXLEPUBKEYHASHn value).
The problem is for Xen, do we need a new parameter, such as 'lehash=<SHA256>' to specify the default value of guset's virtual IA32_SGXLEPUBKEYHASHn? And do we need a new parameter, such as 'lewr' to specify whether guest's virtual MSRs 
are locked or not before handling to guest's OS?
I tends to not introduce 'lehash', as it seems SGX driver would actively update 
the MSRs. And new parameter would add additional changes for upper layer
software (such as openstack). And 'lewr' is not needed either as Xen can always 
*unlock* the MSRs to guest.

Please give comments?

Currently in my RFC patches above two parameters are not implemented.
Xen hypervisor will always *unlock* the MSRs. Whether there is 'lehash'
parameter or not doesn't impact Xen hypervisor's emulation of
IA32_SGXLEPUBKEYHASHn. See below Xen hypervisor changes for details.


Reading around, am I correct with the following?
1) Some processors have no launch control. There is no restriction on which enclaves can boot.
Yes that some processors have no launch control. However it doesn't mean there's no restriction on which enclaves can boot. Contrary, on those machines only Intel's Launch Enclave (LE) can run, as on those machine, IA32_SGXLEPUBKEYHASHn either doesn't exist, or equal to digest of Intel's signing RSA pubkey. However although only Intel's LE can be run, we can still run other enclaves from other signers. Please see my reply above.
2) Some Skylake client processors claim to have launch control, but the MSRs are unavailable (is this an erratum?). These are limited to booting enclaves matching the Intel public key.
Sorry I don't know whether this is an erratum. I will get back to you after confirming internally. 

Hi Andrew,
I raised this internally, and it turns out that in the latest SDM Intel has fixed the statement, so that IA32_SGXLEPUBKEYHASHn MSRs are only available when both SGX and SGX_LC is present in CPUID. When I was writing the design and patches, I was referring to old SDM, and the old one doesn't mention SGX_LC in CPUID as condition. So it is my fault and this statement has been fixed in latest SDM (41.2.2 Intel SGX Launch Control Configuration): 

https://software.intel.com/sites/default/files/managed/7c/f1/332831-sdm-vol-3d.pdf

However in latest SDM volume 4: Model-Specific Registers:

https://software.intel.com/sites/default/files/managed/22/0d/335592-sdm-vol-4.pdf
You can still see that for IA32_SGXLEPUBKEYHASHn (table 2-2, register address 8CH): "Read permitted If CPUID.(EAX=12H,ECX=0H):EAX[0]=1". So there's still error in SDM.
I don't think this will be an erratum. Intel will fix the error in vol 4 in next version SDM. We should refer to 41.2.2 as it has accurate description.
3) Launch control may be locked by the BIOS. There may be a custom hash, or it might be the Intel default. Xen can't adjust it at all, but can support running any number of VMs with matching enclaves.
Yes Launch control may be locked by BIOS, although this depends on whether BIOS provides interface for user to configure. I was told that typically BIOS will unlock Launch Control, as SGX driver is expecting such behavior. But I am not sure we can always assume this.
Whether there will be custom hash also depends on BIOS. BIOS may or may not provide interface for user to configure custom hash. So on physical machine, I think we need to consider all the cases. On machine that with Launch control *unlocked*, Xen is able to dynamically change IA32_SGXLEKEYHASHn so that Xen is able to run multiple VM with each running LE from different signer. However if launch control is *locked* in BIOS, then Xen is still able to run multiple VM, but all VM can only run LE from the signer that matches the IA32_SGXLEPUBKEYHASHn (which in most case should be Intel default, but can be custom hash if BIOS allows user to configure).
Sorry I am not quite sure the typical implementation of BIOS. I think I can reach out internally and get back to you if I have something.
I also reached out internally to find the typical BIOS implementation in terms of SGX LC. Typically BIOS will neither provide configuration options for user to set custom hash, nor select whether MSRs are locked or not. Typically for client machine, MSRs are locked with Intel default, and for server machine, MSRs are unlocked. But we cannot rule out 3rd party to provide different BIOS that may provide options for user to choose locked/unlocked mode, and/or for user to specify custom hash. Custom hash + locked mode may be useful for some special purpose (ex, IT management) as it provides most secure option -- that even kernel/VMM can only launch LE signed with particular signer. In case of VM, custom hash + locked mode may be even more useful than bare-metal as VM is usually supposed to run some particular purpose appliance.
So I think it is better to keep 'lehash' and 'lewr' XL parameters. They both are optional -- the former provides custom hash, and the latter set VM to be in unlocked mode. If neither is specified, then VM will be in locked mode, and VM's virtual IA32_SGXLEPUBKEYHASHn either have Intel's default value (when physical machine is unlocked), or have machine's MSR values (when machine is in locked mode). And when physical machine is in locked mode, specifying either 'lehash' or 'lewr' will result in creating VM failure.
So we have 3 XL parameters for SGX: 'epc', 'lehash' and 'lewr', probably we should consolidate them into one XL parameter, such sgx=['epc=<size>', 'lehash=<sha256>', 'lewr=[on|off]'] ? 

Thanks,
-Kai
4) Launch control may be unlocked by the BIOS. In this case, Xen can context switch a hash per domain, and run all enclaves. 

Yes. With enclave == LE I think you meant.
The eventual plans for CPUID and MSR levelling should allow all of these to be expressed in sensible ways, and I don't forsee any issues with supporting all of these scenarios.
So do you think we should have 'lehash' and 'lewr' parameters in XL config file? The former provides custom hash, and the latter provides whether unlock guest's Launch control.
My thinking is SGX driver needs to *actively* write LE's pubkey hash to IA32_SGXLEPUBKEYHASHn in *unlocked* mode, so 'lehash' alone is not needed. 'lehash' only has meaning when 'lewr' is needed to provide a default hash value in locked mode, as if we always use *unlocked* mode for guest, 'lehash' is not necessary. 






2.2 High Level Xen Hypervisor Changes:

2.2.1 EPC Management (?)

Xen hypervisor needs to detect SGX, discover EPC, and manage EPC before
supporting SGX to guest. EPC is detected via SGX CPUID 0x12.0x2. It's possible that there are multiple EPC sections (enumerated via sub-leaves 0x3 and so on, until invaid EPC is reported), but this is only true on multiple-socket server machines. For server machines there are additional things also needs to be done, such as NUMA EPC, scheduling, etc. We will support server machine in the future 
but currently we only support one EPC.
EPC is reported as reserved memory (so it is not reported as normal memory). EPC must be managed in 4K pages. CPU hardware uses EPCM to track status of each EPC pages. Xen needs to manage EPC and provide functions to, ie, alloc and free 
EPC pages for guest.
There are two ways to manage EPC: Manage EPC separately; or Integrate it to 
existing memory management framework.
It is easy to manage EPC separately, as currently EPC is pretty small (~100MB), and we can even put them in a single list. However it is not flexible, for example, you will have to write new algorithms when EPC becomes larger, ex, GB. And you have to write new code to support NUMA EPC (although this will not come 
in short time).
Integrating EPC to existing memory management framework seems more reasonable, as in this way we can resume memory management data structures/algorithms, and it will be more flexible to support larger EPC and potentially NUMA EPC. But modifying MM framework has a higher risk to break existing memory management 
code (potentially more bugs).

In my RFC patches currently we choose to manage EPC separately. A new
structure epc_page is added to represent a single 4K EPC page. A whole array of struct epc_page will be allocated during EPC initialization, so that given the other, one of PFN of EPC page and 'struct epc_page' can be got by adding 
offset.

But maybe integrating EPC to MM framework is more reasonable. Comments?

2.2.2 EPC Virtualization (?)
It looks like managing the EPC is very similar to managing the NVDIMM ranges. We have a (set of) physical address ranges which need 4k ownership granularity to different domains. 

I think integrating this into struct page_struct is the better way to go.
Will do. So I assume we will introduce new MEMF_epc, and use existing alloc_domheap/xenheap_pages to allocate EPC? MEMF_epc can also be used if we need to support ballooning in the future (using existing XENMEM_{decrease/increase}_reservation. 





This part is how to populate EPC for guests. We have 3 choices:
     - Static Partitioning
     - Oversubscription
     - Ballooning
Static Partitioning means all EPC pages will be allocated and mapped to guest when it is created, and there's no runtime change of page table mappings for EPC pages. Oversubscription means Xen hypervisor supports EPC page swapping between domains, meaning Xen is able to evict EPC page from another domain and assign it to the domain that needs the EPC. With oversubscription, EPC can be assigned to domain on demand, when EPT violation happens. Ballooning is similar to memory ballooning. It is basically "Static Partitioning" + "Balloon driver" in guest.
Static Partitioning is the easiest way in terms of implementation, and there will be no hypervisor overhead (except EPT overhead of course), because in "Static partitioning", there is no EPT violation for EPC, and Xen doesn't need to turn on ENCLS VMEXIT for guest as ENCLS runs perfectly in non-root mode.
Ballooning is "Static Partitioning" + "Balloon driver" in guest. Like "Static Paratitioning", ballooning doesn't need to turn on ENCLS VMEXIT, and doesn't have EPT violation for EPC either. To support ballooning, we need ballooning driver in guest to issue hypercall to give up or reclaim EPC pages. In terms of hypercall, we have two choices: 1) Add new hypercall for EPC ballooning; 2) Using existing XENMEM_{increase/decrease}_reservation with new memory flag, ie, XENMEMF_epc. I'll discuss more regarding to adding dedicated hypercall or not 
later.
Oversubscription looks nice but it requires more complicated implemetation. Firstly, as explained in 1.3.3 EPC Eviction & Reload, we need to follow specific steps to evict EPC pages, and in order to do that, basically Xen needs to trap ENCLS from guest and keep track of EPC page status and enclave info from all 
guest. This is because:
     - To evict regular EPC page, Xen needs to know SECS location
- Xen needs to know EPC page type: evicting regular EPC and evicting SECS, 
       VA page have different steps.
- Xen needs to know EPC page status: whether the page is blocked or not. 

Those info can only be got by trapping ENCLS from guest, and parsing its
parameters (to identify SECS page, etc). Parsing ENCLS parameters means we need to know which ENCLS leaf is being trapped, and we need to translate guest's virtual address to get physical address in order to locate EPC page. And once ENCLS is trapped, we have to emulate ENCLS in Xen, which means we need to reconstruct ENCLS parameters by remapping all guest's virtual address to Xen's virtual address (gva->gpa->pa->xen_va), as ENCLS always use *effective address* 
which is able to be traslated by processor when running ENCLS.

     --------------------------------------------------------------
                 |   ENCLS   |
     --------------------------------------------------------------
                 |          /|\
     ENCLS VMEXIT|           | VMENTRY
                 |           |
                \|/          |

        1) parse ENCLS parameters
        2) reconstruct(remap) guest's ENCLS parameters
        3) run ENCLS on behalf of guest (and skip ENCLS)
        4) on success, update EPC/enclave info, or inject error
And Xen needs to maintain each EPC page's status (type, blocked or not, in enclave or not, etc). Xen also needs to maintain all Enclave's info from all guests, in order to find the correct SECS for regular EPC page, and enclave's 
linear address as well.
So in general, "Static Partitioning" has simplest implementation, but obviously not the best way to use EPC efficiently; "Ballooning" has all pros of Static Partitioning but requies guest balloon driver; "Oversubscription" is best in 
terms of flexibility but requires complicated hypervisor implemetation.

We have implemented "Static Partitioning" in RFC patches, but needs your
feedback on whether it is enough. If not, which one should we do at next stage -- Ballooning or Oversubscription. IMO Ballooning may be good enough, given fact 
that currently memory is also "Static Partitioning" + "Ballooning".

Comments?
Definitely go for static partitioning to begin with. This is far simpler to implement.
I can't see a pressing usecase for oversubscription or ballooning. Any datacenter work will be using exclusively static, and I expect static will fine for all (or at least, most) client usecases. 

Thanks. So for the first stage I will focus on static partitioning.





2.2.3 Populate EPC for Guest
Toolstack notifies Xen about domain's EPC base and size by XEN_DOMCTL_set_cpuid, so currently Xen populates all EPC pages for guest in XEN_DOMCTL_set_cpuid, particularly, in handling XEN_DOMCTL_set_cpuid for CPUID.0x12.0x2. Once Xen checks the values passed from toolstack is valid, Xen will allocate all EPC 
pages and setup EPT mappings for guest.

2.2.4 New Dedicated Hypercall (?)
All this information should (eventually) be available via the appropriate SYSCTL_get_{cpuid,msr}_policy hypercalls. I don't see any need for dedicated hypercalls.
Yes I agree. Originally I had concern that without dedicated hypercall, it is hard to implement 'xl sgxinfo' and 'xl sgxlist', but according to your new CPUID enhancement plan, the two can be done via the new hypercalls to query Xen's and domain's cpuid policy. See my reply above regarding to "Notify Xen about guest's EPC info". 




2.2.9 Guest Suspend & Resume
On hardware, EPC is destroyed when power goes to S3-S5. So Xen will destroy guest's EPC when guest's power goes into S3-S5. Currently Xen is notified by Qemu in terms of S State change via HVM_PARAM_ACPI_S_STATE, where Xen will 
destroy EPC if S State is S3-S5.
Specifically, Xen will run EREMOVE for guest's each EPC page, as guest may not handle EPC suspend & resume correctly, in which case physically guest's EPC pages may still be valid, so Xen needs to run EREMOVE to make sure all EPC pages are becoming invalid. Otherwise further operation in guest on EPC may 
fault as it assumes all EPC pages are invalid after guest is resumed.
For SECS page, EREMOVE may fault with SGX_CHILD_PRESENT, in which case Xen will keep this SECS page into a list, and call EREMOVE for them again after all EPC pages have been called with EREMOVE. This time the EREMOVE on SECS will succeed 
as all children (regular EPC pages) have already been removed.

2.2.10 Destroying Domain
Normally Xen just frees all EPC pages for domain when it is destroyed. But Xen will also do EREMOVE on all guest's EPC pages (described in above 2.2.7) before free them, as guest may shutdown unexpected (ex, user kills guest), and in this 
case, guest's EPC may still be valid.

2.3 Additional Point: Live Migration, Snapshot Support (?)
How big is the EPC? If we are talking MB rather than GB, movement of the EPC could be after the pause, which would add some latency to live migration but should work. I expect that people would prefer to have the flexibility of migration even at the cost of extra latency.
The EPC is typically ~100MB at maximum (as I observed). The EPC is typically reserved with EPCM (EPC map, which is invisible to SW) together by BIOS as processor reserved memory (RPM). On real machine, for both our internal develop machines, and some machines that from Dell, HP, Lenovo (that you can buy from market now), BIOS always provides 3 choices in terms RPM: 32M, 64M, and 128M. And with 128M RPM, EPC is slightly less than 100M.
The problem is EPC cannot be moved. I think you were saying moving EPC by evicting EPC out at last stage and copy evicted content to remote, and then reload. However I don't think this will work, as EPC eviction itself needs to use a VA slot (which itslef is EPC), so you can image that the VA slots cannot be moved to remote. Even if they can, they cannot be used to reload EPC in remote, as info in VA slot is bound to platform and cannot be used on remote.
To support live migration, we can only choose to ignore EPC during live migration and let guest SGX driver/user SW stack to handle restoring enclave (which is actually a lot simpler in hypervisor/toolstack's implementation) . Guest SGX driver needs to handle lose EPC anyway, as EPC is destroyed in S3-S5. The only difference is to support live migration, guest SGX driver needs to support *sudden* lose of EPC, which is not HW behavior, and I was told that currently both Windows & Linux SGX driver already support *sudden* lose of EPC, which leaves us a question whether we need to support SGX live migration (and snapshot). 


~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel

 

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.

Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.