Re: [Xen-devel] swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU kernel with PCI passthrough

[Dante Cinco <dantecinco@xxxxxxxxx>]

On Fri, Nov 12, 2010 at 7:56 AM, Konrad Rzeszutek Wilk
<konrad.wilk@xxxxxxxxxx> wrote:
> On Thu, Nov 11, 2010 at 12:42:03PM -0700, Lin, Ray wrote:
>>
>> Konrad,
>>
>>    See my response in red.
>
> Please don't top post.
>>
>>
>> -Ray
>>
>> -----Original Message-----
>> From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx 
>> [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Konrad Rzeszutek 
>> Wilk
>> Sent: Thursday, November 11, 2010 11:04 AM
>> To: Dante Cinco
>> Cc: Xen-devel
>> Subject: Re: [Xen-devel] swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops 
>> domU kernel with PCI passthrough
>>
>> On Thu, Nov 11, 2010 at 10:31:48AM -0800, Dante Cinco wrote:
>> > Konrad,
>> >
>> > Without swiotlb=force, I don't see "PCI-DMA: Using software bounce
>> > buffering for IO" in /var/log/kern.log.
>> >
>> > With iommu=soft and without swiotlb=force, I see the "software bounce
>> > buffering" in /var/log/kern.log and an NMI (see below) when I load the
>> > kernel module drivers. I made sure the NMI is reproducible and not a
>>
>> What is the kernel module doing to cause this? DMA?
>
> ??? What did it do?
>
>> > one-time event.
>>
>> So doing 64-bit DMA causes an NMI. Do you have the Hypervisor's IOMMU VT-d 
>> enabled or disabled? (iommu=off,verbose) If you turn it off does this work?
>>
>> We have IOMMU VT-d enabled. If we turn it off (iommu=off,verbose), the DMA 
>> doesn't work properly and the driver code is unable to detect the source of 
>> interrupt. The interrupts of our device would be disabled by kernel 
>> eventually due to nobody services the interrupts for more than 100000 times.
>
> That does not sound right. You should be able to use the PCI passthrough 
> without the IOMMU. Since it
> is an interrupt issue it sounds like that you are using x2APIC and that is 
> enabled without the IOMMU.
> Had you tried disabling IOMMU and x2apic? (this is all on the hypervisor 
> line?)
>
>

I set the hypervisor boot options to iommu=0 x2apic=0. I booted the
pvops domU with swiotlb=force initially since that's the option that
worked in the past. Not long after loading the kernel module drivers,
domU hung/froze but dom0 stayed up. I checked the Xen interrupt
bindings and I see the PCI-passthrough devices have either (PS--) or
(-S--).

(XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:72
type=PCI-MSI         status=00000010 in-flight=0
domain-list=1:127(-S--),
(XEN)    IRQ:  67 affinity:00000000,00000000,00000000,00000200 vec:3b
type=PCI-MSI         status=00000010 in-flight=0
domain-list=1:126(-S--),
(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:8a
type=PCI-MSI         status=00000010 in-flight=0
domain-list=1:125(-S--),
(XEN)    IRQ:  69 affinity:00000000,00000000,00000000,00000800 vec:43
type=PCI-MSI         status=00000010 in-flight=0
domain-list=1:124(PS--),
(XEN)    IRQ:  70 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:9a
type=PCI-MSI         status=00000010 in-flight=0
domain-list=1:123(-S--),
(XEN)    IRQ:  71 affinity:fffffff,ffffffff,ffffffff,ffffffff vec:a2
type=PCI-MSI         status=00000010 in-flight=0
domain-list=1:122(PS--),
(XEN)    IRQ:  72 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:aa
type=PCI-MSI         status=00000010 in-flight=0
domain-list=1:121(-S--),
(XEN)    IRQ:  73 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b2
type=PCI-MSI         status=00000010 in-flight=0
domain-list=1:120(PS--),
(XEN)    IRQ:  74 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:ba
type=PCI-MSI         status=00000010 in-flight=0
domain-list=1:119(-S--),
(XEN)    IRQ:  75 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:c2
type=PCI-MSI         status=00000010 in-flight=0
domain-list=1:118(PS--),
(XEN)    IRQ:  76 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:ca
type=PCI-MSI         status=00000010 in-flight=0
domain-list=1:117(-S--),
(XEN)    IRQ:  77 affinity:00000000,00000000,00000000,00080000 vec:4b
type=PCI-MSI         status=00000010 in-flight=0
domain-list=1:116(PS--),
(XEN)    IRQ:  78 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:da
type=PCI-MSI         status=00000010 in-flight=0
domain-list=1:115(-S--),
(XEN)    IRQ:  79 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:23
type=PCI-MSI         status=00000010 in-flight=0
domain-list=1:114(PS--),
(XEN)    IRQ:  80 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:2b
type=PCI-MSI         status=00000010 in-flight=0
domain-list=1:113(-S--),
(XEN)    IRQ:  81 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:33
type=PCI-MSI         status=00000010 in-flight=0
domain-list=1:112(PS--),

I will reboot the pvops domU with iommu=soft and without swiotlb=force next.

- Dante

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel