Re: [Xen-devel] [PATCH] x86/mm/p2m: don't needlessly limit MMIO mapping order to 4k

[Paul Durrant <Paul.Durrant@xxxxxxxxxx>]

> -----Original Message-----
> From: Andrew Cooper
> Sent: 17 October 2018 14:24
> To: Paul Durrant <Paul.Durrant@xxxxxxxxxx>; xen-devel@xxxxxxxxxxxxxxxxxxxx
> Cc: George Dunlap <George.Dunlap@xxxxxxxxxx>; Jan Beulich
> <jbeulich@xxxxxxxx>; Wei Liu <wei.liu2@xxxxxxxxxx>
> Subject: Re: [PATCH] x86/mm/p2m: don't needlessly limit MMIO mapping order
> to 4k
> 
> On 16/10/18 15:41, Paul Durrant wrote:
> > The P2M common code currently restricts the MMIO mapping order of any
> > domain with IOMMU mappings, but that is not using shared tables, to 4k.
> > This has been shown to have a huge performance cost when passing through
> > a PCI device with a very large BAR (e.g. NVIDIA P40), increasing the
> guest
> > boot time from ~20s to several minutes when iommu=no-sharept is
> specified
> > on the Xen command line.
> >
> > The limitation was added by commit c3c756bd "x86/p2m: use large pages
> > for MMIO mappings" however the underlying implementations of p2m-
> >set_entry
> > for Intel and AMD are coded to cope with mapping orders higher than 4k,
> > even though the IOMMU mapping function is itself currently limited to
> 4k,
> > so there is no real need to limit the order passed into the method.
> >
> > With this patch applied the VM boot time is restored to something
> > reasonable. Not as fast as without iommu=no-sharept, but within a few
> > seconds of it.
> >
> > NOTE: This patch takes the opportunity to shorten a couple of > 80
> >       character lines in the code.
> >
> > Signed-off-by: Paul Durrant <paul.durrant@xxxxxxxxxx>
> 
> I'm afraid that it isn't needless.
> 
> The fact that the underlying IOMMU functions can only work with 4k
> pages, in combination with no -ERestart support, is precisely why you
> must only ever pass a 4k order here.  Otherwise we will genuinely wait
> for the IOMMU to map/unmap 1G worth of 4k pages at a time, which the
> security team has previously deemed is too long to wait for a single
> operation.
> 

Ok, I agree that there may be an issue with 1G but not for 2M. I have tested 
the patch and, as I said in my comment, the total cost of the all the domctls 
was barely different from when page sharing was turned on.

> I'm afraid that the only safe option I see here is to make the p2m
> operations support properly preemptible.
> 

That's a lot of re-work. I agree it is the correct solution but without a patch 
to allow at least 2M mapping, a system without shared tables is unusable so I 
think such a patch is reasonable stop-gap.

  Paul

> ~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel