Re: [Xen-devel] RE: Intel GPU pass-through with > 3G

[Daniel De Graaf <dgdegra@xxxxxxxxxxxxx>]

I have also noticed this issue (9ms IOMMU flush), although I not during
domain creation. The path in which I observed it is page remapping when
using map_grant_ref. I haven't tested a DomU with over 3G of memory,
however; the delay may also be present in that case on my platform.

I have done some work to try to add an 'order' parameter to iommu_map_page,
but it isn't stable yet; if this is the only way to get around the slow
flush, I will look at finishing it.

Would it be possible to add a flag to delay IOMMU flushing until after a
batch update is finished? A single flush at the end, even if expensive,
would be faster than 10ms per page on mappings of a significant size. This
is also likely to be a less intrusive patch.

In case you're interested, my platform is a Dell Optiplex 755, 4G RAM:

# lspci
00:00.0 Host bridge: Intel Corporation 82Q35 Express DRAM Controller (rev 02)
00:01.0 PCI bridge: Intel Corporation 82Q35 Express PCI Express Root Port (rev 
02)
00:02.0 VGA compatible controller: Intel Corporation 82Q35 Express Integrated 
Graphics Controller (rev 02)
00:02.1 Display controller: Intel Corporation 82Q35 Express Integrated Graphics 
Controller (rev 02)
00:19.0 Ethernet controller: Intel Corporation 82566DM-2 Gigabit Network 
Connection (rev 02)
00:1a.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
Controller #4 (rev 02)
00:1a.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
Controller #5 (rev 02)
00:1a.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI 
Controller #2 (rev 02)
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio 
Controller (rev 02)
00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 
(rev 02)
00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
Controller #3 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI 
Controller #1 (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IO (ICH9DO) LPC Interface Controller 
(rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port 
SATA AHCI Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)

-- 
Daniel De Graaf
National Security Agency

On 11/10/2010 07:04 PM, Kay, Allen M wrote:
> Jean,
> 
> Do you see any boot time difference between passing through integrated 
> graphics for the very first time and the subsequent times?  Which platform 
> are you using?
> 
> Allen
> 
> -----Original Message-----
> From: Jean Guyader [mailto:jean.guyader@xxxxxxxxx] 
> Sent: Wednesday, November 10, 2010 1:50 PM
> To: xen-devel@xxxxxxxxxxxxxxxxxxx
> Cc: Kay, Allen M
> Subject: Intel GPU pass-through with > 3G
> 
> Hello,
> 
> I'm passing through a graphic card to a guest that has more than 3G of
> RAM (4G to be precise in my case).
> 
> What happen is that the VM creation is stuck in the process, so I put
> some tracing in the Xen code to see what
> was taking the time. I discovered that the guest was stuck in
> hvmloader inside this loop:
> 
>    while ( (pci_mem_start >> PAGE_SHIFT) < hvm_info->low_mem_pgend )
>     {
>         struct xen_add_to_physmap xatp;
>         if ( hvm_info->high_mem_pgend == 0 )
>             hvm_info->high_mem_pgend = 1ull << (32 - PAGE_SHIFT);
>         xatp.domid = DOMID_SELF;
>         xatp.space = XENMAPSPACE_gmfn;
>         xatp.idx   = --hvm_info->low_mem_pgend;
>         xatp.gpfn  = hvm_info->high_mem_pgend++;
>         if ( hypercall_memory_op(XENMEM_add_to_physmap, &xatp) != 0 )
>             BUG();
>     }
> 
> This loop relocate the RAM on the top to leave so space for the PCI BARs.
> It's loop on each page so in my case it's quite a big loop because the
> GPU has a BAR of 256M.
> 
> So the interesting is that the function add_to_physmap takes most of
> the time. I believe
> that what takes most part of it is the iommu iotlb flush that come
> with the iommu_map_pages
> or the iommu_unmap_page which are called when we manipulate the p2m table.
> 
> In my case the iommu flush take a very long time (because of the intel
> gpu ?), about 10
> milliseconds. So if I'm patient enough my domain will start, about 10 minutes.
> 
> A way to go will be to create a range interface to iommu_map_page
> iommu_unmap_page
> since iommu_flush are so expensive. Then some work need to be done to
> add a range interface
> to all the function between add_to_physmap and the p2m_set_entry which
> would be a big
> patch. I hope there is another way out of this problem.
> 
> Thanks,
> Jean

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel