Re: [Xen-devel] PCI Passthrough and PV-OPS

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

On Thu, Mar 03, 2011 at 07:07:48PM +0000, Kinsella, Ray wrote:
> Hi Konrad,
> 
> Thanks for your response.
> My understanding of this solution is that we are essentially switching off 
> VT-d and using a software solution instead (bounce buffers?).

No.
> Am I correct?.

There are _two_ swiotlb implementation in the pvops kernel.

The baremetal is to be used under baremetal and it won't be turned
on when running DomU/Dom0.

The other, xen-swiotlb is turned on by default when dom0 boots. And
it is turned on for DomU if the user specified 'iommu=soft'.

The xen-swiotlb has two modes: bounce buffer for devices that need it (which 
nowadays is USB), and
translate the PFNs to MFNs (and vice-versa). We want to use the second 
functionality
which hooks up in the PCI DMA. You can turn on the bounce buffer for everything
if you specify 'swiotlb=force', but there is no point of doing that.

So in essence you have Xen hypervisor doing VT-D, dom0 and domU both using
the Linux DMA API wherein the Xen-SWIOTLB is hooked up. The Xen-SWIOTLB runs
in translation mode and when a device is setup using the DMA API it provides
the MFNs instead of the PFNs to the driver.

> 
> Ray Kinsella
> 
> -----Original Message-----
> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@xxxxxxxxxx] 
> Sent: Thursday, March 03, 2011 3:03 PM
> To: Kinsella, Ray
> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
> Subject: Re: [Xen-devel] PCI Passthrough and PV-OPS
> 
> On Thu, Mar 03, 2011 at 02:25:47PM +0000, Kinsella, Ray wrote:
> > Hi all,
> > 
> > I am having a problem with Xen 4.0.1, PCI Passthrough with VT-d and Linux 
> > PV Guests, and I was wondering if anyone else had seen it.
> > In both Dom 0 and Dom U's I am using 64bit PV-OPS 2.6.32.26 Kernels from 
> > Jeremy Fitzhartridge's Git repo.
> 
> Do you have 'iommu=soft' in your guest configuration?
> 
> > 
> > I am using dual port SR-IOV NIC's to test, passing through physical 
> > functions only.
> > Passthrough appears to work, I can passthrough the NIC and it will appear 
> > in the guest with lspci.
> > The guest detects the new device and loads the driver to service the NIC, 
> > as you would expect.
> > 
> > The guest is complaining about the NIC being hung, the message "Detected Tx 
> > Unit Hang..." is appearing in the system log on the guest.
> > In the Xen log, VT-d is producing errors similar to this one;
> > 
> > (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
> > (XEN) [VT-D]iommu.c:799: DMAR:[DMA Read] Request device [03:00.0] fault 
> > addr 7c584000, iommu reg = ffff82c3fff57000
> > (XEN) DMAR:[fault reason 06h] PTE Read access is not set
> > (XEN) print_vtd_entries: iommu = ffff83016fffa5f0 bdf = 3:0.0 gmfn = 7c584
> > (XEN)     root_entry = ffff83016ff38000
> > (XEN)     root_entry[3] = 14d90a001
> > (XEN)     context = ffff83014d90a000
> > (XEN)     context[0] = 102_152209005
> > (XEN)     l4 = ffff830152209000
> > (XEN)     l4_index = 0
> > (XEN)     l4[0] = 152208003
> > (XEN)     l3 = ffff830152208000
> > (XEN)     l3_index = 1
> > (XEN)     l3[1] = 0
> > (XEN)     l3[1] not present
> > 
> > My understanding of what is happening here is that page table mappings are 
> > missing from VT-d's page table, mapping machine physical addresses to my PV 
> > guest's physical addresses. I have had a look at iommu_populate_page_table 
> > and I dumped the mappings as they are being setup but couldn't see a 
> > mapping for the GPA 0x7c584000 above?
> > 
> > Has anyone else encountered this issue?
> 
> Yes. And it was fixed by running a PV guest with 'iommu=soft' as Linux kernel 
> command line argument.
> --------------------------------------------------------------
> Intel Shannon Limited
> Registered in Ireland
> Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
> Registered Number: 308263
> Business address: Dromore House, East Park, Shannon, Co. Clare
> 
> This e-mail and any attachments may contain confidential material for the 
> sole use of the intended recipient(s). Any review or distribution by others 
> is strictly prohibited. If you are not the intended recipient, please contact 
> the sender and delete all copies.
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel