Re: PCI pass-through problem for SN570 NVME SSD

["G.R." <firemeteor@xxxxxxxxxxxxxxxxxxxxx>]

On Tue, Jul 5, 2022 at 12:21 AM Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
>
> On Mon, Jul 04, 2022 at 11:37:13PM +0800, G.R. wrote:
> > On Mon, Jul 4, 2022 at 11:15 PM G.R. <firemeteor@xxxxxxxxxxxxxxxxxxxxx> 
> > wrote:
> > >
> > > On Mon, Jul 4, 2022 at 10:51 PM G.R. <firemeteor@xxxxxxxxxxxxxxxxxxxxx> 
> > > wrote:
> > > >
> > > > On Mon, Jul 4, 2022 at 9:09 PM Roger Pau Monné <roger.pau@xxxxxxxxxx> 
> > > > wrote:
> > > > > Can you paste the lspci -vvv output for any other device you are also
> > > > > passing through to this guest?
> > > > >
> > >
> > > As reminded by this request, I tried to assign this nvme device to
> > > another FreeBSD12 domU.
> > Just to clarify, this time this NVME SSD is the only device I passed to 
> > this VM.
> >
> > > This time it does not fail at the VM setup stage, but the device is
> > > still not usable at the domU.
> > > The nvmecontrol command is not able to talk to the device at all:
> > > nvme0: IDENTIFY (06) sqid:0 cid:0 nsid:0 cdw10:00000001 cdw11:00000000
> > > nvme0: ABORTED - BY REQUEST (00/07) sqid:0 cid:0 cdw0:0
> > > nvme0: IDENTIFY (06) sqid:0 cid:0 nsid:0 cdw10:00000001 cdw11:00000000
> > > nvme0: ABORTED - BY REQUEST (00/07) sqid:0 cid:0 cdw0:0
> > >
> > > The QEMU log says the following:
> > > 00:05.0] Write-back to unknown field 0x09 (partially) inhibited (0x00)
> > > [00:05.0] If the device doesn't work, try enabling permissive mode
> > > [00:05.0] (unsafe) and if it helps report the problem to xen-devel
> > > [00:05.0] msi_msix_setup: Error: Mapping of MSI-X (err: 61, vec: 0x30, 
> > > entry 0)
> >
> > I retried with the following:
> > pci=['05:00.0,permissive=1,msitranslate=1']
> > Those extra options suppressed some error logging, but still didn't
> > make the device usable to the domU.
> > The nvmecontrol command still get ABORTED result from the kernel...
> >
> > The only thing remained in the QEMU file is this one:
> > [00:05.0] msi_msix_setup: Error: Mapping of MSI-X (err: 61, vec: 0x30, 
> > entry 0)
>
> Hm it seems like Xen doesn't find the position of the MSI-X table
> correctly, given there's only one error path from msi.c returning
> -ENODATA (61).
>
> Are there errors from pciback when this happens?  I would expect the
> call to pci_prepare_msix() from pciback to fail and thus also report
> some error?
>
> I think it's likely I will have to provide an additional debug patch
> to Xen, maybe Jan has an idea of what could be going on.
>
pciback reports the same MSI-x related error.
But even with DEBUG enabled, I didn't see more context reported.
Please find details from the attachment.

> Roger.

Attachment: pciback_dbg_xl-pci_assignable_XXX.log
Description: Text Data