Xen project Mailing List

On Wed, Apr 13, 2016 at 9:59 AM, Lars Kurth <lars.kurth.xen@xxxxxxxxx> wrote:

Draft blog at https://blog.xenproject.org/?p=11282&preview=true

@Tamas
Please do a quick sanity check and let me know whether to publish. I also updated your bio: if you want extra mods, let me know
Your account is linked to the above email address: you may want to reset the password

Lars

On 13 Apr 2016, at 16:24, Lars Kurth <lars.kurth.xen@xxxxxxxxx> wrote:

Tamas,

apologies. I missed this in my inbox ...

I made a few minor suggestions: mainly shortening some sentences.
There are also a couple of questions

Do you want me to upload this to the blog (under your username)? Where do you want to insert the png's?

On 25 Jan 2016, at 20:51, Lengyel, Tamas <tlengyel@xxxxxxxxxxx> wrote:

Hi all,
here is an updated version. Also, a couple pictures that could be used as illustrations. Feel free to punch it up as needed ;)

Tamas

Stealthy monitoring with Xen altp2m

One of the core features that differentiates Xen from other open-source hypervisors is its native support for stealthy and secure monitoring of guest internals (aka. virtual machine introspection [1]). In the latest release of Xen last summer

"In Xen 4.6 which was was released last autumn"

several new features have been introduced that make this subsystem better; a cleaned-up, optimized API and ARM support being just some of the biggest items on this list. As part of this release of Xen, a new and unique feature was also successfully added by a team from Intel that make stealthy monitoring even better on Xen: altp2m. In this blog entry we will take a look at what it's all about.

In Xen's terminology, p2m stands for the memory management layer that handles the translation from guest [p]hysical memory to [m]achine physical. This translation is critical for safely partitioning the real memory of machine between Xen and the various VMs running as to ensure a VM can't simply access the memory of another. There are several implementations of this

There are several implementations of this mechanism,

including one with hardware support via Intel Extended Page Tables (EPT) available to HVM (and PVH) guests

and PVH <guests>.

, called Hardware Assisted Paging (hap) in Xen's terminology.

In Xen's terminology, this is called Hardware Assisted Paging (hap).

In this implementation the hypervisor maintains a second pagetable, similar to the one in 64-bit operating systems use, dedicated to running the p2m translation. All (open-source) hypervisors that use this hardware assisted paging method use a single EPT per virtual machine to handle this translation, as most of the time the memory of the guest is assigned at VM creation an doesn't change much afterwards.

Xen altp2m is the first implementation which changes this setup by allowing Xen to create more then one EPT for each guest. Interestingly, the Intel hardware has been capable of maintaining up to 512 EPT pointers in the VMCS since the first introduction of EPT; however, noone made use of this capability until now. This changed in Xen 4.6, where we can now create of up to 10 EPTs per guest. The primary reason for this extension is of course the new #VE and VMFUNC extensions that were released in the Skylake generation of CPUs (which is worth a whole blog-entry on its own), but it can also be used by external monitoring applications via the Xen vm_event system.

It can also be used by external monitoring applications via the Xen vm_event system.

Add a sub-headline: Why alt2pm is a game-changer

Why this feature is a game-changer for applications performing purely external monitoring is because it simplifies the monitoring process of multi-vCPU guests.

Alt2pm is a game-changer for applications performing purely external monitoring is because it simplifies the monitoring process of multi-vCPU guests.

The EPT layer has been successfully used in stealthy monitoring applications to track the memory accesses made by the VM from a safe vantage point by restricting the type of access the VM may perform on various memory pages. Since EPT permission violations trap into the hypervisor, the VM would receive no indication that anything out of the ordinary has happened. While the method allowed for stealthy tracing of R/W/X memory accesses of the guest, the memory permission needs to be relaxed in order to allow the guest to continue execution. When a single EPT is shared across multiple running vCPUs, relaxing the permissions to allow one vCPU to continue may inadvertantly allow another one to perform the memory access we would otherwise want to track. While under normal circumstances such race-condition may rarely occur, malicious code could easily use this to hide some of its actions from a monitoring application.

Solutions to this problem exist already. For example we can pause all vCPUs while the one violating the access is singlestepped. This approach however introduces heavy overhead just to avoid a race-condition that may rarely occur in practice. Alternatively, one could emulate the instruction that was violating the EPT permission without relaxing the EPT access permissions, as Xen's built-in emulator doesn't use EPT to access the guest memory. This solution, while supported in Xen, is not particularly ideal either as Xen's emulator is incomplete and is known to have issues that can lead to guest instability [2]. Furthermore, over the years emulation has been a hotbed of various security issues in many hypervisors (including Xen [3]), thus building security tools based on emulation is simply asking for trouble. It can be handy but should be used only when no other option is available.

Xen's altp2m system changes this problem quite significantly. By having multiple EPTs we can have differing access permissions defined in each table, which can be easily swapped around by changing the active EPT index in the VMCS. When the guest makes a memory access that is monitored, instead of having to relax the access permission, Xen can simply switch to an EPT (called a view) that allows the operation to continue. Afterwards the permissive view can be switched back to the restriced view to continue monitoring. Since each vCPU has its own VMCS where this switching is performed, this monitoring can be performed specific to each vCPU, without having to pause any of the other ones, or having to emulate the access. All without the guest noticing any of this switching at all. A truly simple and elegant solution.

Add a sub-headline: Other introspection methods for stealthy monitoring

Of course, EPT based monitoring is not the only introspecting technique used for stealthy monitoring. For example, the Xen based DRAKVUF Dynamic Malware Analysis [4] uses it in combination with an additional technique to maximum effect.

The main motivation for that is because EPT based monitoring is known to introduce significant overhead, even with altp2m: the granularity of the monitoring is that of a memory page (4KB).

This sentence is a bit convoluted and I lost you here. Not quite sure what you are trying to say

If the monitoring application is really just interested, for example, when a function-entry point is called, EPT based monitoring creates a lot of "false" events when that page is accessed for the rest of the function's code.

Again, something seems to be missing here and I am struggling to understand what you are trying to say

Fortunately, this can be avoided by enabling the trapping of debug instructions into the hypervisor, a built-in feature of Intel CPUs that Xen exposes to third-party applications. This method is used in DRAKVUF, which writes breakpoint instructions into the guests' memory at code-locations of interest. Since we will only get an event for precisely the code-location we are interested in this method effectively reduces the overhead. However, the trade-off is that unlike EPT permissions the breakpoints are now visible to the guest. Thus, to hide the presence of the breakpoints from the guest, these pages need to get further protected by restricting the pages to be execute-only in the EPT. This allows DRAKVUF to remove the breakpoints before in-guest code-integrity checking mechanisms (like Windows Patchguard) can access the page. While with altp2m the EPT permissions can be safely used with multi-vCPU systems, using breakpoints similarly presents a race-condition: the breakpoint hit by one vCPU has to be removed to allow the guest to execute the instruction that was originally overwritten, potentially allowing another vCPU to do so as well without notice.

Fortunately, altp2m has another neat feature that can be used to solve this problem. Beside allowing for changing the memory permissions in the different altp2m views, it also allows to change the mapping itself! The same guest physical memory can be setup to be backed by different pages in the different views. With this feature we can really thing of guest physical

assuming: thing=think

memory as "virtual": where it is mapped really depends on which view the vCPU is running on. Using this feature allows us to hide the presence of the breakpoints in a brand new way. To do this, first we create a complete shadow copy of the memory page where a breakpoint is going to be written and only write the breakpoint into this shadow copy. Now, using altp2m, we setup a view where the guest physical memory of the page gets mapped to our shadow copy. The guest continues to access its physical memory as before, but underneath it is now using the trapped shadow copy. When the breakpoint is hit, or if something is trying to scan the code, we simply switch the view to the un-altered view for the duration of a singlestep, then switch back to the trapped view. This allows us to hide the presence of the breakpoints specific to each vCPU! All without having pause any of the other vCPUs or having to emulate. The first open-source implementation of this tracing has been already merged into the DRAKVUF Malware Analysis System and is available as a reference implementation for those interested in more details.

As we can see, Xen continues to be on the forefront of advancing the development of virtualization based security application and allowing third-party tools to create some very exotic setups. This flexibility is what's so great about Xen and why it will continue to be a trend-setter for the foreseeable future.

Add a sub-headline: References

[1] http://wiki.xenproject.org/wiki/Virtual_Machine_Introspection
[2] http://lists.xen.org/archives/html/xen-devel/2016-01/msg00285.html
[3] https://blog.xenproject.org/2015/05/14/hardening-hypervisors-against-venom-style-attacks/
[4] http://drakvuf.com
<altp2m.png><altp2m-shadow.png><p2m.png>

Re: [Publicity] Stealthy monitoring with Xen altp2m