Re: PCI pass-through problem for SN570 NVME SSD

[Jan Beulich <jbeulich@xxxxxxxx>]

On 07.07.2022 17:24, G.R. wrote:
> On Wed, Jul 6, 2022 at 2:33 PM Jan Beulich <jbeulich@xxxxxxxx> wrote:
>>
>> On 06.07.2022 08:25, G.R. wrote:
>>> On Tue, Jul 5, 2022 at 7:59 PM Jan Beulich <jbeulich@xxxxxxxx> wrote:
>>>> Nothing useful in there. Yet independent of that I guess we need to
>>>> separate the issues you're seeing. Otherwise it'll be impossible to
>>>> know what piece of data belongs where.
>>> Yep, I think I'm seeing several different issues here:
>>> 1. The FLR related DPC / AER message seen on the 1st attempt only when
>>> pciback tries to seize and release the SN570
>>>     - Later-on pciback operations appear just fine.
>>> 2. MSI-X preparation failure message that shows up each time the SN570
>>> is seized by pciback or when it's passed to domU.
>>> 3. XEN tries to map BAR from two devices to the same page
>>> 4. The "write-back to unknown field" message in QEMU log that goes
>>> away with permissive=1 passthrough config.
>>> 5. The "irq 16: nobody cared" message shows up *sometimes* in a
>>> pattern that I haven't figured out  (See attached)
>>> 6. The FreeBSD domU sees the device but fails to use it because low
>>> level commands sent to it are aborted.
>>> 7. The device does not return to the pci-assignable-list when the domU
>>> it was assigned shuts-down. (See attached)
>>>
>>> #3 appears to be a known issue that could be worked around with
>>> patches from the list.
>>> I suspect #1 may have something to do with the device itself. It's
>>> still not clear if it's deadly or just annoying.
>>> I was able to update the firmware to the latest version and confirmed
>>> that the new firmware didn't make any noticeable difference.
>>>
>>> I suspect issue #2, #4, #5, #6, #7 may be related, and the
>>> pass-through was not completely successful...
>>>
>>> Should I expect a debug build of XEN hypervisor to give better
>>> diagnose messages, without the debug patch that Roger mentioned?
>>
>> Well, "expect" is perhaps too much to say, but with problems like
>> yours (and even more so with multiple ones) using a debug
>> hypervisor (or kernel, if there such a build mode existed) is imo
>> always a good idea. As is using as up-to-date a version as
>> possible.
> 
> I built both 4.14.3 debug version and 4.16.1 release version for
> testing purposes.
> Unfortunately they gave me absolutely zero information, since both of
> them are not able to get through issue #1
> the FlR related DPC / AER issue.
> With 4.16.1 release, it actually can survive the 'xl
> pci-assignable-add' which triggers the first AER failure.

Then that's what needs debugging first. Yet from all I've seen so
far I'm not sure who one the Xen side could be doing that, the more
without themselves being able to repro - this seems more like a
Linux side issue (and even outside of the pciback driver).

> But the 'xl pci-assignable-remove' will lead to xl segmentation fault...
>> [  655.041442] xl[975]: segfault at 0 ip 00007f2cccdaf71f sp 
>> 00007ffd73a3d4d0 error 4 in libxenlight.so.4.16.0[7f2cccd92000+7c000]
>> [  655.041460] Code: 61 06 00 eb 13 66 0f 1f 44 00 00 83 c3 01 39 5c 24 2c 
>> 0f 86 1b 01 00 00 48 8b 34 24 89 d8 4d 89 f9 4d 89 f0 4c 89 e9 4c 89 e2 <48> 
>> 8b 3c c6 31 c0 48 89 ee e8 53 44 fe ff 83 f8 04 75 ce 48 8b 44

That'll need debugging. Cc-ing Anthony for awareness, but I'm sure
he'll need more data to actually stand a chance of doing something
about it.

Is there any chance you could be doing some debugging work yourself,
at the very least to figure out where this (apparent) NULL deref is
happening?

Jan