Re: [Xen-devel] PML (Page Modification Logging) design for Xen

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 11/02/15 13:13, Jan Beulich wrote:
>>>> On 11.02.15 at 12:52, <andrew.cooper3@xxxxxxxxxx> wrote:
>> On 11/02/15 08:28, Kai Huang wrote:
>>> With PML, we don't have to use write protection but just clear D-bit
>>> of EPT entry of guest memory to do dirty logging, with an additional
>>> PML buffer full VMEXIT for 512 dirty GPAs. Theoretically, this can
>>> reduce hypervisor overhead when guest is in dirty logging mode, and
>>> therefore more CPU cycles can be allocated to guest, so it's expected
>>> benchmarks in guest will have better performance comparing to non-PML.
>> One issue with basic EPT A/D tracking was the scan of the EPT tables. 
>> Here, hardware will give us a list of affected gfns, but how is Xen
>> supposed to efficiently clear the dirty bits again?  Using EPT
>> misconfiguration is no better than the existing fault path.
> Why not? The misconfiguration exit ought to clear the D bit for all
> 511 entries in the L1 table (and set it for the one entry that is
> currently serving the access). All further D bit handling will then
> be PML based.
>
>>> - PML buffer flush
>>>
>>> There are two places we need to flush PML buffer. The first place is
>>> PML buffer full VMEXIT handler (apparently), and the second place is
>>> in paging_log_dirty_op (either peek or clean), as vcpus are running
>>> asynchronously along with paging_log_dirty_op is called from userspace
>>> via hypercall, and it's possible there are dirty GPAs logged in vcpus'
>>> PML buffers but not full. Therefore we'd better to flush all vcpus'
>>> PML buffers before reporting dirty GPAs to userspace.
>> Why apparently?  It would be quite easy for a guest to dirty 512 frames
>> without otherwise taking a vmexit.
> I silently replaced apparently with obviously while reading...
>
>>> We handle above two cases by flushing PML buffer at the beginning of
>>> all VMEXITs. This solves the first case above, and it also solves the
>>> second case, as prior to paging_log_dirty_op, domain_pause is called,
>>> which kicks vcpus (that are in guest mode) out of guest mode via
>>> sending IPI, which cause VMEXIT, to them.
>>>
>>> This also makes log-dirty radix tree more updated as PML buffer is
>>> flushed on basis of all VMEXITs but not only PML buffer full VMEXIT.
>> My gut feeling is that this is substantial overhead on a common path,
>> but this largely depends on how the dirty bits can be cleared efficiently.
> I agree on the overhead part, but I don't see what relation this has
> to the dirty bit clearing - a PML buffer flush doesn't involve any
> alterations of D bits.

I admit that I was off-by-one level when considering the
misconfiguration overhead.  It would be inefficient (but not unsafe as
far as I can tell) to clear all D bits at once; the PML could end up
with repeated gfns in it, or different vcpus could end up with the same
gfn, depending on the exact access pattern, which will add to the flush
overhead.

~Andrew


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel