Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

[Bruce Edge <bruce.edge@xxxxxxxxx>]

On Wed, Oct 13, 2010 at 2:46 PM, Konrad Rzeszutek Wilk
<konrad.wilk@xxxxxxxxxx> wrote:
> On Wed, Oct 13, 2010 at 02:36:47PM -0700, Bruce Edge wrote:
>> On Mon, Oct 11, 2010 at 2:46 PM, Konrad Rzeszutek Wilk
>> <konrad.wilk@xxxxxxxxxx> wrote:
>> > On Mon, Oct 11, 2010 at 02:12:22PM -0700, Bruce Edge wrote:
>> >> On Fri, Oct 1, 2010 at 2:11 PM, Konrad Rzeszutek Wilk
>> >> <konrad.wilk@xxxxxxxxxx> wrote:
>> >> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
>> >> >> One of our developers who is working on a tachyon driver is
>> >> >> complaining that the pvops domU kernel is not working for these MSI
>> >> >> interrupts.
>> >> >> This is using the current head of xen/2.6.32.x on both a single
>> >> >> Nahelam 920 and a dual E5540. This behavior is consistent with Xen
>> >> >> 4.0.1, 4.0.2.rc1-pre and 4.1.
>> >> >
>> >> >
>> >> > I just checked on my SuperMicro X8DTN, this combination
>> >> >  - For Dom0, git commit fe999249 (2.6.32.18)
>> >> >  - For DomU, devel/xen-pcifront-0.6 or devel/xen-pcifront-0.7
>> >> >  - For Hypervisor I used cs 21976, but found that the latest (22155) 
>> >> > works too
>> >> >
>> >> > with which where I passed in PCI devices with legacy IRQ, MSI, and 
>> >> > MSI-X. I tried
>> >> > a combination of doing this with IOMMU (VT-d) and without - both cases 
>> >> > these devices:
>> >> >
>> >> > 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
>> >> > Controller #1 (rev 02)
>> >> > 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
>> >> > Controller #2 (rev 02)
>> >> > 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI 
>> >> > Controller #3 (rev 02)
>> >> > 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 
>> >> > EHCI Controller #1 (rev 02)
>> >> > 03:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet 
>> >> > Controller (Copper) (rev 06)
>> >> > 0a:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network 
>> >> > Connection (rev 02)
>> >> >
>> >> > worked just fine (either defining pci=["..."] or just using pci-attach).
>> >> >
>> >> > But if I use the latest xen/next or xen/stable-2.6.32.x it does not look
>> >> > that happy :-(
>> >> >
>> >> >
>> >>
>> >> Konrad,
>> >> To try eliminate the remaining differences here, could you post your
>> >> dom0/domU config files?
>> >
>> > Sure. See attached
>>
>> Konrad,
>> That made a big difference. Looks much better now. It's been kicked
>> over to several developers who have each got our tachyon driver
>> working a little bit better.
>
> Hmm, I didn't really try to do anything fancy with the configs. Any
> inklings of what config option might have caused all this headache?

I had pruned out a lot of stuff I didn't need from the kernel. I may
have been a bit overzealous. Also, we were using the same kernel for
dom0 and domU. It may be that having all the dom0 baggage in the domU
has some unexpected consequences. Very vague I know, sorry. If I have
time I'll try narrow down the diffs that break it.

>
>>
>> Now the sticking point is an apparent limitation on the amount of
>> memory one can request using pci_map_single. It appears that we can
>> only ask for 256K or less.  We need a 2MB DMA buffer.
>
> You can't use SG buffers? And chain them together and provide them
> to the device? A lot of other drivers do this ...

Perhaps. I don't know. I'm not the driver author. I think that it
worked using 1 giant buffer before so there was no need to use SG
buffers.

>
>> Is there some alternate mechanism for getting a larger physically
>> contiguous buffer under pvops?
>
> Why the contiguous requirement?

It may not be a  hard requirement and was just a matter of
convenience. I'll kick that over the the tachyon guys and see what
they say. I think it's a non-trivial exercise to switch over. All the
internal buffer mgmt would have to change. If that's the only option,
well, that's it. ...is it?

Thanks again for your help. You've been instrumental it getting this up.

-Bruce

>>
>> Thanks
>>
>> -Bruce
>>
>> >> I'd like to build the same kernels to get an apples->apples comparison.
>> >>
>> >> Also, could you include your grub info and domU cfg file?
>> >>
>> >> These may eliminate some of the remaining diffs in the configs and
>> >> show why your's works while mine does not.
>> >>
>> >> Thanks
>> >>
>> >> -Bruce
>> >
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel