Re: [Xen-devel] [RFC PATCHv2 0/3]: x86/ept: reduce translation invalidation impact

["Tian, Kevin" <kevin.tian@xxxxxxxxx>]

> From: David Vrabel [mailto:david.vrabel@xxxxxxxxxx]
> Sent: Saturday, November 14, 2015 2:50 AM
> 
> This RFC series improves the performance of EPT by reducing the impact
> of the translation invalidations (ept_sync_domain()).  Two approaches
> are used:
> 
> a) Removing unnecessary invalidations after fixing misconfigured
>    entries (after a type change).
> 
> b) Deferring invalidations until the p2m write lock is released.

Do you have a sense which one above incurs more overhead?

> 
> Prior to this change a 16 VCPU guest could not be successfully
> migrated on an (admittedly slow) 160 PCPU box because the p2m write
> lock was held for such extended periods of time.  This starved the
> read lock needed (by the toolstack) to map the domain's memory,
> triggering the watchdog.
> 
> After this change a 64 VCPU guest could be successfully migrated.
> 
> ept_sync_domain() is very expensive because:
> 
> a) it uses on_selected_cpus() and the IPI cost can be particularly
>    high for a multi-socket machine.
> 
> b) on_selected_cpus() is serialized by its own spin lock.
> 
> On this particular box, ept_sync_domain() could take ~3-5 ms.
> 
> Simply using a fair rw lock was not sufficient to resolve this (but it
> was an improvement) as the cost of the ept_sync_domain calls() was
> still delaying the read locks enough for the watchdog to trigger (the
> toolstack maps a batch of 1024 GFNs at a time, which means trying to
> acquire the p2m read lock 1024 times).
> 
> Changes in v2:
> 
> - Use a per-p2m (not per-CPU) list for page table pages to be freed.
> - Hold the write lock while updating the synced_mask.
> 
> David


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel