RE: [Xen-devel] RE: Intel GPU pass-through with > 3G

["Kay, Allen M" <allen.m.kay@xxxxxxxxx>]

Another thing to try is to set add a flag to XENMEM_add_physmap hypercall (or 
create another hypercall) to set a no_iotlb_flag.  Hvmloader will then make 
this hypercall with this flag set to 1.  When this flag is set, then 
intel_iommu_map_page will not do iotlb_flush.

Given this mapping is done at domain creation time, there is really no need to 
do iotlb flushes, especially when it is doing page mapping.

Allen

-----Original Message-----
From: Jean Guyader [mailto:jean.guyader@xxxxxxxxx] 
Sent: Friday, November 12, 2010 6:23 AM
To: Daniel De Graaf
Cc: Kay, Allen M; xen-devel@xxxxxxxxxxxxxxxxxxx; Tim.Deegan@xxxxxxxxxx
Subject: Re: [Xen-devel] RE: Intel GPU pass-through with > 3G

I've done a patch like this where I set a flag d->iommu_dont_flush at
the beginning of my batched function then
an explicit call to flush the iotlb at the end.

It's not a very nice way of solving this problem maybe it would be
better to have a range/batch interface at the p2m_set_entry level.

Jean

On 12 November 2010 14:18, Daniel De Graaf <dgdegra@xxxxxxxxxxxxx> wrote:
> I have also noticed this issue (9ms IOMMU flush), although I not during
> domain creation. The path in which I observed it is page remapping when
> using map_grant_ref. I haven't tested a DomU with over 3G of memory,
> however; the delay may also be present in that case on my platform.
>
> I have done some work to try to add an 'order' parameter to iommu_map_page,
> but it isn't stable yet; if this is the only way to get around the slow
> flush, I will look at finishing it.
>
> Would it be possible to add a flag to delay IOMMU flushing until after a
> batch update is finished? A single flush at the end, even if expensive,
> would be faster than 10ms per page on mappings of a significant size. This
> is also likely to be a less intrusive patch.
>
> In case you're interested, my platform is a Dell Optiplex 755, 4G RAM:
>
> # lspci
> 00:00.0 Host bridge: Intel Corporation 82Q35 Express DRAM Controller (rev 02)
> 00:01.0 PCI bridge: Intel Corporation 82Q35 Express PCI Express Root Port 
> (rev 02)
> 00:02.0 VGA compatible controller: Intel Corporation 82Q35 Express Integrated 
> Graphics Controller (rev 02)
> 00:02.1 Display controller: Intel Corporation 82Q35 Express Integrated 
> Graphics Controller (rev 02)
> 00:19.0 Ethernet controller: Intel Corporation 82566DM-2 Gigabit Network 
> Connection (rev 02)
> 00:1a.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
> Controller #4 (rev 02)
> 00:1a.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
> Controller #5 (rev 02)
> 00:1a.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI 
> Controller #2 (rev 02)
> 00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio 
> Controller (rev 02)
> 00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 
> (rev 02)
> 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
> Controller #1 (rev 02)
> 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
> Controller #2 (rev 02)
> 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
> Controller #3 (rev 02)
> 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI 
> Controller #1 (rev 02)
> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
> 00:1f.0 ISA bridge: Intel Corporation 82801IO (ICH9DO) LPC Interface 
> Controller (rev 02)
> 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port 
> SATA AHCI Controller (rev 02)
> 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 
> 02)
>
> --
> Daniel De Graaf
> National Security Agency
>
> On 11/10/2010 07:04 PM, Kay, Allen M wrote:
>> Jean,
>>
>> Do you see any boot time difference between passing through integrated 
>> graphics for the very first time and the subsequent times?  Which platform 
>> are you using?
>>
>> Allen
>>
>> -----Original Message-----
>> From: Jean Guyader [mailto:jean.guyader@xxxxxxxxx]
>> Sent: Wednesday, November 10, 2010 1:50 PM
>> To: xen-devel@xxxxxxxxxxxxxxxxxxx
>> Cc: Kay, Allen M
>> Subject: Intel GPU pass-through with > 3G
>>
>> Hello,
>>
>> I'm passing through a graphic card to a guest that has more than 3G of
>> RAM (4G to be precise in my case).
>>
>> What happen is that the VM creation is stuck in the process, so I put
>> some tracing in the Xen code to see what
>> was taking the time. I discovered that the guest was stuck in
>> hvmloader inside this loop:
>>
>>    while ( (pci_mem_start >> PAGE_SHIFT) < hvm_info->low_mem_pgend )
>>     {
>>         struct xen_add_to_physmap xatp;
>>         if ( hvm_info->high_mem_pgend == 0 )
>>             hvm_info->high_mem_pgend = 1ull << (32 - PAGE_SHIFT);
>>         xatp.domid = DOMID_SELF;
>>         xatp.space = XENMAPSPACE_gmfn;
>>         xatp.idx   = --hvm_info->low_mem_pgend;
>>         xatp.gpfn  = hvm_info->high_mem_pgend++;
>>         if ( hypercall_memory_op(XENMEM_add_to_physmap, &xatp) != 0 )
>>             BUG();
>>     }
>>
>> This loop relocate the RAM on the top to leave so space for the PCI BARs.
>> It's loop on each page so in my case it's quite a big loop because the
>> GPU has a BAR of 256M.
>>
>> So the interesting is that the function add_to_physmap takes most of
>> the time. I believe
>> that what takes most part of it is the iommu iotlb flush that come
>> with the iommu_map_pages
>> or the iommu_unmap_page which are called when we manipulate the p2m table.
>>
>> In my case the iommu flush take a very long time (because of the intel
>> gpu ?), about 10
>> milliseconds. So if I'm patient enough my domain will start, about 10 
>> minutes.
>>
>> A way to go will be to create a range interface to iommu_map_page
>> iommu_unmap_page
>> since iommu_flush are so expensive. Then some work need to be done to
>> add a range interface
>> to all the function between add_to_physmap and the p2m_set_entry which
>> would be a big
>> patch. I hope there is another way out of this problem.
>>
>> Thanks,
>> Jean
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel