Re: [Xen-devel] PCI passthrough performance loss with Skylake-SP

[Paul Durrant <Paul.Durrant@xxxxxxxxxx>]

> -----Original Message-----
> From: Roger Pau Monne
> Sent: 26 June 2018 07:57
> To: Andreas Kinzler <hfp@xxxxxxxxx>
> Cc: xen-devel@xxxxxxxxxxxxx; Paul Durrant <Paul.Durrant@xxxxxxxxxx>
> Subject: Re: [Xen-devel] PCI passthrough performance loss with Skylake-SP
> 
> On Mon, Jun 25, 2018 at 05:32:58PM +0200, Andreas Kinzler wrote:
> > I am currently testing PCI passthrough on the Skylake-SP platform using a
> > Supermicro X11SPi-TF mainboard. Using PCI passthrough (an LSI SAS HBA)
> > causes severe performance loss on the Skylake-SP platform while Xeon E3
> v5
> > is not affected at all. The test uses standard iperf3 as a client - the
> > passed PCI device is not used in the test - so that just the presence of the
> > passed device will cause the iperf3 performance to drop from 6.5 gbit/sec
> > (no passthrough) to 4.5 gbit/sec.
> 
> I assume that the network interface that you are testing is a PV
> network interface?
> 
> > Any explanation/fixes for that?
> 
> Are both systems using the same version of Xen and Linux?
> 
> Paul discovered that (recent?) Linux versions the grant table is
> allocated in a MMIO area, which has the memory cache attribute set to
> uncached. This has a noticeable impact on performance when the guest
> also has a passthrough device.

I can't necessarily claim credit for the discovery but that is indeed the case, 
and the sort of performance drop seen is exactly what I'd expect. I recently 
put a change into the Windows PV drivers to use a ballooned-out region of the 
guest RAM to host the grant tables instead, which avoids this problem.

> 
> Note that this is not an issue without passthrough because Xen forces
> the cache attributes of memory to WB in that case, regardless of the
> guest choice.
> 

We run with this little hack in XenServer, which also 'fixes' things for guests 
OS that have not been modified:

diff --git a/xen/arch/x86/hvm/mtrr.c b/xen/arch/x86/hvm/mtrr.c
index 3c51244..d0126b4 100644
--- a/xen/arch/x86/hvm/mtrr.c
+++ b/xen/arch/x86/hvm/mtrr.c
@@ -811,7 +811,8 @@ int epte_get_entry_emt(struct domain *d, unsigned long gfn, 
mfn_t mfn,
         return MTRR_TYPE_UNCACHABLE;
     }
 
-    if ( !need_iommu(d) && !cache_flush_permitted(d) )
+    if ( (!need_iommu(d) && !cache_flush_permitted(d)) ||
+         is_xen_heap_mfn(mfn_x(mfn)) )
     {
         *ipat = 1;
         return MTRR_TYPE_WRBACK;

So you may want to give that a try.

  Paul

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel