[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] Design session report: LivePatch improvements and features

LivePatch improvements and features

Development plans for LivePatch on Xen:
  • Support for module parameters
  • Additional hooks support
  • Concept of expectations
  • Inline assembly patching
  • Replaceable apply/revert actions
  • Fixes and improvements for stacked modules

Use live patch for newer Xen version.
Implemented these live patch changes based on challenges faced during security events.

Patches sent upstream till July 11th 2019:
Started work on live patch stacked modules that were not supported in upstream Xen.

Live patch build tools copy sections, but some rela entries appear as undefined.
  • The solution is to walk through result object and check undefined in rela sections.
  • Patches are sent upstream.

Read-only data sections
  • Content keep accumulating for stacked modules e.g. the last one contains all data from the previous hotpatches.
  • Looked through them to not have duplicate data from previous patches.
  • Patches are sent upstream.

Independent module loading
  • Currently a chain of build ids is enforced.
  • Would want to load an arbitrary module. It is not possible now.
  • Setup flag, only xen build is checked, not the previous hotpatch build id.
  • Now there are 2 flags.
  • A bunch of changes in live patch.

Hooks
  • Separate layers of hooks – pre/post/middle.
  • Currently apply / revert available in upstream codebase.
  • Want to really clean up is something goes wrong during hotpatching.
  • Introduce pre hook which has access to the payload struct e.g. allocate memory; if anything goes wrong, the post does cleanup.
  • The developer needs to write the hooks.

Replace operation
  • Hooks can fail.
  • The current implementation prevents executing replace ops on payloads with hooks.

Apply / Revert hooks
  • By default use standard apply / revert.
  • Would want to have only hooks used, for dev testing.
  • Functions have access to payload struct, have enhanced capabilities to operate on payload.

Expectations
  • Addressing stacked modules and inline assembly patches.
  • Embed an array to each hotpatch to say which is the expected content of the memory to be replaced.
  • The developer needs to write the expectations.
  • There are changes in the ABI.
  • Quite intrusive.

Python bindings for livepatch tools
  • The existing implementation was not so easy to extend.

How is assembly hotpatching working?
  • Manually create the live patch; have a container and jump back to the replaced code.
  • Get ret annotations for assembly in entry.s. Need to have correct elf annotations. It works with the newest version. Patch the build and put the extra elf metadata.

The concept of hotpatch metadata
  • It is added in the hotpatch, together with the payload.
  • A key-value data struct.
  • There is also an interface available to extract that info.
  • It helps e.g. for a stack of modules.
  • Can obtain info about of the past.
  • The codebase change is relatively big.

Expectations, using the build is was simpler. Why it was easier to do that way, with an array of expected content?
  • For defense in depth.
  • The build id works, but it is a hidden assumption.
  • There can be errors when creating the hotpatches, would prevent mistakes.

Is it not a problem with alternative patches?
  • It can be an issue, but expectations feature is gated by a flag.

A build id should identify the hot patch uniquely.
Have you run into issues with using build id?
  • Yes, but not in production.
  • The expectations are only for inline assembly, for normal ones the build id is ok.

Would like to have assembly moved into C pieces of code. Or get proper elf annotations. To change an entry point, no need to change existing code. For exit, it needs e.g. when placing a jump.


What’s regarding the one patch only solution?
  • We need to have a stacked solution e.g. for legacy reasons.
  • The replacement approach has an assumption that the hotpatch can be unloaded safely, but there can be cases when it is not.

Expectations are a reasonable thing to have as long they are optional.


What are middle hooks?
  • They are replacement for apply / revert functions.
  • Introduced a flag for tracking if the patch is applied or not.
  • Apply / Revert run from the quiesce zone; there were situations when replacing them was useful.
  • Dynamically add functions for apply / revert.
  • One example is the hotpach for the Spectre mitigation. Hotpatch the code on the CPU return after quiesce; w/o custom apply it would have been impossible.

Hotpatches that have safety checks is a good thing. One example to have apply/revert is when it is needed to apply and update an MSR / CPU registers (CR4). Would need a different apply / revert per CPU.


Pre / post hooks for apply / revert
  • Checks and cleanup within pre / post.
  • Would it be useful? Yes.
  • For apply would need that.
  • Need to have a safe way to do that with replace operation. For example, when we do not use stacked patches. At least start discussion on that.
  • We have chosen to not have a patch loaded because we could not get it safely unloaded. It would have been useful to have hooks like this.

Independent modules
  • How does it interact with stacked modules?
  • For production it is not the case to have arbitrary patches.
  • For dev testing it is useful to have an arbitrary patch.
  • The feature is optional, mainly for debugging.
  • Mainly use it as a kprobe, so that would not need a reboot.
  • The things can go wrong in this case, for arbitrary patches.

Extra metadata
  • What was it about? There is name available until now.
  • There can be commit of the patch, build tools commit. Need to understand the source of the hotpatch, name, version.
  • Data is generated after the hotpatch is created, in a separate elf section. Can easily strip off the elf section afterwards.
  • It would be useful to have it in upstream.
  • There is also the ability to dump the metadata from the xen debug keys.


Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel

 

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.

Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.