Re: [Publicity] Stealthy monitoring with Xen altp2m

["Lengyel, Tamas" <tlengyel@xxxxxxxxxxx>]

Hi Razvan,
thanks for chiming in!

On Jan 4, 2016 5:43 PM, "Razvan Cojocaru" <rcojocaru@xxxxxxxxxxxxxxx> wrote:
>
> On 01/04/2016 04:26 PM, Lengyel, Tamas wrote:
> > However, this solution (while supported) is not particularly ideal as it
> > still creates significant performance overhead. Moreover, emulation in
> > Xen is known to be buggy, so creating security tools that ultimately
> > rely on an emulator is just asking for trouble.
>
> That's debatable. The emulator is not as much buggy as it is incomplete,

Yes, that's a nicer way to state it.

> and there are ways around that - see the emulator bypass patch in our
> initial series back in 2014. I do concede that the Xen emulator does
> have a way to go.

By default the only option is Xen's emulator so that's what I think should be discussed. Using another emulator, while probably possible, would certainly not be trivial.

>
> As for the performance overhead, if the security tool chooses the
> restricted pages wisely then it is more than acceptable. Of course, if
> all the guest's pages have restricted access rights then the guest will
> be noticeably impaired.
>
> And still on the performance issue, with emulation we have:
>
> 1. receive page fault event;
> 2. reply to the hypervisor with emulate flags on;
> 3. get a new page fault within the hypervisor (page restrictions have
> not been lifted) - but not hypervisor <-> userspace context switch);
> 4. emulate.

That's six context switches + emulation.

>
> With altp2m we have:
>
> 1. receive page fault event;
> 2. switch altp2m view in the vm_event reply + set singlestepping on
> (hypercalls, thus context switching);

It's done in the same hypercall you have to issue anyway for vm_event to signal the request has been processed so this adds no extra.

> 3. get singlestep event from the hypervisor / guest (more context
> switching);

Here indeed there is an extra step of delivering the event to the subscriber.

> 4. switch altp2m view again + set singlestepping off (again context
> switching / hypercalls).

This is again done in the reply directly so no extra hypercalls. So overall this is eight context switches but no emulation.

In a nutshell, performance will of course vary and I have no in-depth analysis showing one to be better then the other one. Depending on how complex the emulation is one may be faster in some cases and slower in others. The argument for the altp2m solution is that it avoids emulation and reduces complexity, without pausing all vCPUs on the system.

>
> This is definitely not a very detailed analysis of what's happening, but
> I would think that the latter process takes at least as much time as the
> former. In other words, I don't see how the altp2m case improves
> performance.

I don't think it does either.

>
> Further on you say "By having multiple EPTs we can have differing access
> permissions defined in each table, which can be swapped around at will.
> When the guest makes a memory access that is monitored, instead of
> having to relax the access permission, Xen can simply switch to an EPT
> (called a view) that allows the operation to continue."
>
> However, that is now an all-or-none operating procedure - either allow
> or deny the operation that triggered the event. You can't, for example,
> allow the operation but disallow its actual write, so that you can
> continue to observe malicious activity without the malware being able to
> actually do what it wants to do.

Certainly, that's a use-case for which the emulation based solution is still suitable for. That however is beyond the point of simple monitoring and would be more like an IPS solution.

>
> There might also be issues with guests where the number of VCPUs is
> bigger than 10, which is the maximum number of allowed EPTs.

You don't need a separate altp2m view for each vCPU. All you need is two views that can be interchanged in each vCPU independently.

>
> Both solutions have advantages and disadvantages, and having them both
> makes Xen a great platform for the discerning developer.

Certainly, having access to the emulator does have its usecases.

While the
> altp2m solution might be simpler for your use-case, perhaps neither is
> intrinsically superior.

For the usecase of monitoring arbitrary memory accesses and code execution on multi-vCPU systems it is the only readily available solution at the moment. Using the emulator in this fashion unfortunately crashes the guest (that's the case you verified too, we should probably report that issue on xen-devel). But even if the emulator worked properly, provided the long list of bugs we have encountered in emulators in recent years I would not be surprised to find more issues. So if it can be avoided, it should be. And in this case it can be quite nicely.

>
> Cheers,
> Razvan

Thanks!
Tamas

_______________________________________________
Publicity mailing list
Publicity@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/publicity