Re: PCI pass-through problem for SN570 NVME SSD

[Jan Beulich <jbeulich@xxxxxxxx>]

On 04.07.2022 18:31, G.R. wrote:
> On Tue, Jul 5, 2022 at 12:21 AM Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
>>
>> On Mon, Jul 04, 2022 at 11:37:13PM +0800, G.R. wrote:
>>> On Mon, Jul 4, 2022 at 11:15 PM G.R. <firemeteor@xxxxxxxxxxxxxxxxxxxxx> 
>>> wrote:
>>>>
>>>> On Mon, Jul 4, 2022 at 10:51 PM G.R. <firemeteor@xxxxxxxxxxxxxxxxxxxxx> 
>>>> wrote:
>>>>>
>>>>> On Mon, Jul 4, 2022 at 9:09 PM Roger Pau Monné <roger.pau@xxxxxxxxxx> 
>>>>> wrote:
>>>>>> Can you paste the lspci -vvv output for any other device you are also
>>>>>> passing through to this guest?
>>>>>>
>>>>
>>>> As reminded by this request, I tried to assign this nvme device to
>>>> another FreeBSD12 domU.
>>> Just to clarify, this time this NVME SSD is the only device I passed to 
>>> this VM.
>>>
>>>> This time it does not fail at the VM setup stage, but the device is
>>>> still not usable at the domU.
>>>> The nvmecontrol command is not able to talk to the device at all:
>>>> nvme0: IDENTIFY (06) sqid:0 cid:0 nsid:0 cdw10:00000001 cdw11:00000000
>>>> nvme0: ABORTED - BY REQUEST (00/07) sqid:0 cid:0 cdw0:0
>>>> nvme0: IDENTIFY (06) sqid:0 cid:0 nsid:0 cdw10:00000001 cdw11:00000000
>>>> nvme0: ABORTED - BY REQUEST (00/07) sqid:0 cid:0 cdw0:0
>>>>
>>>> The QEMU log says the following:
>>>> 00:05.0] Write-back to unknown field 0x09 (partially) inhibited (0x00)
>>>> [00:05.0] If the device doesn't work, try enabling permissive mode
>>>> [00:05.0] (unsafe) and if it helps report the problem to xen-devel
>>>> [00:05.0] msi_msix_setup: Error: Mapping of MSI-X (err: 61, vec: 0x30, 
>>>> entry 0)
>>>
>>> I retried with the following:
>>> pci=['05:00.0,permissive=1,msitranslate=1']
>>> Those extra options suppressed some error logging, but still didn't
>>> make the device usable to the domU.
>>> The nvmecontrol command still get ABORTED result from the kernel...
>>>
>>> The only thing remained in the QEMU file is this one:
>>> [00:05.0] msi_msix_setup: Error: Mapping of MSI-X (err: 61, vec: 0x30, 
>>> entry 0)
>>
>> Hm it seems like Xen doesn't find the position of the MSI-X table
>> correctly, given there's only one error path from msi.c returning
>> -ENODATA (61).
>>
>> Are there errors from pciback when this happens?  I would expect the
>> call to pci_prepare_msix() from pciback to fail and thus also report
>> some error?
>>
>> I think it's likely I will have to provide an additional debug patch
>> to Xen, maybe Jan has an idea of what could be going on.
>>
> pciback reports the same MSI-x related error.
> But even with DEBUG enabled, I didn't see more context reported.
> Please find details from the attachment.

And nothing pertinent in "xl dmesg"? Looking back through the thread I
couldn't spot a complete hypervisor log (i.e. from boot to assignment
attempt). An issue with MSI-X table determination, as Roger suspects,
would typically be associated with a prominent warning emitted to the
log. But there are also further possible sources of -ENXIO, which
would go silently.

Jan