Re: [Xen-devel] xen/arm: Alternative start of day cache coherency

[Ian Campbell <Ian.Campbell@xxxxxxxxxx>]

On Thu, 2014-01-23 at 13:55 +0000, Stefano Stabellini wrote:
> On Thu, 23 Jan 2014, Ian Campbell wrote:
> > On Wed, 2014-01-22 at 17:32 +0000, Stefano Stabellini wrote:
> > > > +int xc_domain_cacheflush(xc_interface *xch, uint32_t domid)
> > > > +{
> > > > +    DECLARE_DOMCTL;
> > > > +    domctl.cmd = XEN_DOMCTL_cacheflush;
> > > > +    domctl.domain = (domid_t)domid;
> > > > +    domctl.u.cacheflush.start_mfn = 0;
> > > > +    return do_domctl(xch, &domctl);
> > > > +}
> > > 
> > > Do we really need to flush the entire p2m, or just things we have
> > > written to?
> > 
> > I think we need to flush everything (well, all RAM backed pages, the
> > patch skips everything else).
> > 
> > Even things which we haven't explicitly written to will have been
> > scrubbed and therefore have scrubbed data in the cache but data
> > belonging to the previous owner in actual RAM. So we would really want
> > to clean in that case too.
> > 
> > We could do the clean at scrub time which arguably be better anyway and
> > would potentially allows us to only invalidate instead of clean
> > +invalidate some subset of pages, but we would need to track which sort
> > of page was which -- e.g. with a special p2m type for a page which had
> > been foreign mapped or some other bit of metadata.
> 
> That seems like the way to go.

I'm not convinced actually, and in any case, not for 4.4...

> > > > diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> > > > index 85ca330..f35ed57 100644
> > > > --- a/xen/arch/arm/p2m.c
> > > > +++ b/xen/arch/arm/p2m.c
> > > > @@ -228,15 +228,26 @@ enum p2m_operation {
> > > >      ALLOCATE,
> > > >      REMOVE,
> > > >      RELINQUISH,
> > > > +    CACHEFLUSH,
> > > >  };
> > > >  
> > > > +static void do_one_cacheflush(paddr_t mfn)
> > > > +{
> > > > +    void *v = map_domain_page(mfn);
> > > > +
> > > > +    flush_xen_dcache_va_range(v, PAGE_SIZE);
> > > > +
> > > > +    unmap_domain_page(v);
> > > > +}
> > > 
> > > A pity that we need to map a page just to flush the dcache.  It could be
> > > expensive, especially if we really have to map every single guest mfn.
> > 
> > Remember that this is basically free for arm64 and for arm32 we actually
> > map 2MB regions and cache, so it is only actually one map per 2MB
> > region.
> 
> Even with 2MB at a time it is easy for this to become really slow. A 4GB
> guest would need 2048 iterations of map/flush/unmap. I don't have any
> numbers but I bet they won't look good.

It happens once during domain build and the delay it isn't noticeable to
me compared with the rest of the build process.

> At least if it was combined with the RAM scrub we would save the 2048
> map/unmap.

The scrub has to map it in exactly the same way.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel