Re: PCI pass-through problem for SN570 NVME SSD

["G.R." <firemeteor@xxxxxxxxxxxxxxxxxxxxx>]

On Fri, Jul 8, 2022 at 10:28 AM G.R. <firemeteor@xxxxxxxxxxxxxxxxxxxxx> wrote:
>
> On Fri, Jul 8, 2022 at 12:38 AM Jan Beulich <jbeulich@xxxxxxxx> wrote:
> >
> > > I built both 4.14.3 debug version and 4.16.1 release version for
> > > testing purposes.
> > > Unfortunately they gave me absolutely zero information, since both of
> > > them are not able to get through issue #1
> > > the FlR related DPC / AER issue.
> > > With 4.16.1 release, it actually can survive the 'xl
> > > pci-assignable-add' which triggers the first AER failure.
> >
> > Then that's what needs debugging first. Yet from all I've seen so
> > far I'm not sure who one the Xen side could be doing that, the more
> > without themselves being able to repro - this seems more like a
> > Linux side issue (and even outside of the pciback driver).
> >
> Yep, this one is likely not XEN related, as I've seen some discussions
> ([1],[2]) on similar syndrome (not necessarily same root cause
> though).
> The question is why this only shows up during the FLR attempt and if
> following pci-assignable-adds that do not trigger the error are
> actually reliable.
> BTW, I'm under the impression that the device is still usable in dom0
> afterwards, I'll have to double check though...
I think I'm finally making progress here.
Today I verified that the SSD does not survive FLR in Linux as long as
AER / DPC is enabled.
Irrelevant to pciback or whatever driver it attaches to, the same
syndrome can be observed.
And after the unsuccessful FLR, I can't even use the device on dom0 or
the Linux itself.

Forcing disabling ASPM does not fix the issue, as long as AER / DPC
are left enabled.
However, as long as AER / DPC is disabled (through pcie-aspm=off
kernel command-line, which ironically doesn't turn off ASPM),
the SSD device appears to FLR just fine and I'm still able to use it
on the Linux host afterwards.

And what's even better, the same pcie-aspm=off kernel command works in
XEN env too.
The FreeBSD12 domU is now able to access the SSD without issue now!

Thanks,
G.R.

>
> [1] 
> https://patchwork.kernel.org/project/linux-pci/patch/20220408153159.106741-1-kai.heng.feng@xxxxxxxxxxxxx/
> [2] 
> https://patchwork.kernel.org/project/linux-pci/patch/20220127025418.1989642-1-kai.heng.feng@xxxxxxxxxxxxx/#24713767
>