Re: PCI pass-through problem for SN570 NVME SSD

[Jan Beulich <jbeulich@xxxxxxxx>]

On 04.07.2022 18:05, Roger Pau Monné wrote:
> On Mon, Jul 04, 2022 at 11:37:13PM +0800, G.R. wrote:
>> On Mon, Jul 4, 2022 at 11:15 PM G.R. <firemeteor@xxxxxxxxxxxxxxxxxxxxx> 
>> wrote:
>>>
>>> On Mon, Jul 4, 2022 at 10:51 PM G.R. <firemeteor@xxxxxxxxxxxxxxxxxxxxx> 
>>> wrote:
>>>>
>>>> On Mon, Jul 4, 2022 at 9:09 PM Roger Pau Monné <roger.pau@xxxxxxxxxx> 
>>>> wrote:
>>>>> Can you paste the lspci -vvv output for any other device you are also
>>>>> passing through to this guest?
>>>>>
>>>
>>> As reminded by this request, I tried to assign this nvme device to
>>> another FreeBSD12 domU.
>> Just to clarify, this time this NVME SSD is the only device I passed to this 
>> VM.
>>
>>> This time it does not fail at the VM setup stage, but the device is
>>> still not usable at the domU.
>>> The nvmecontrol command is not able to talk to the device at all:
>>> nvme0: IDENTIFY (06) sqid:0 cid:0 nsid:0 cdw10:00000001 cdw11:00000000
>>> nvme0: ABORTED - BY REQUEST (00/07) sqid:0 cid:0 cdw0:0
>>> nvme0: IDENTIFY (06) sqid:0 cid:0 nsid:0 cdw10:00000001 cdw11:00000000
>>> nvme0: ABORTED - BY REQUEST (00/07) sqid:0 cid:0 cdw0:0
>>>
>>> The QEMU log says the following:
>>> 00:05.0] Write-back to unknown field 0x09 (partially) inhibited (0x00)
>>> [00:05.0] If the device doesn't work, try enabling permissive mode
>>> [00:05.0] (unsafe) and if it helps report the problem to xen-devel
>>> [00:05.0] msi_msix_setup: Error: Mapping of MSI-X (err: 61, vec: 0x30, 
>>> entry 0)
>>
>> I retried with the following:
>> pci=['05:00.0,permissive=1,msitranslate=1']
>> Those extra options suppressed some error logging, but still didn't
>> make the device usable to the domU.
>> The nvmecontrol command still get ABORTED result from the kernel...
>>
>> The only thing remained in the QEMU file is this one:
>> [00:05.0] msi_msix_setup: Error: Mapping of MSI-X (err: 61, vec: 0x30, entry 
>> 0)
> 
> Hm it seems like Xen doesn't find the position of the MSI-X table
> correctly, given there's only one error path from msi.c returning
> -ENODATA (61).
> 
> Are there errors from pciback when this happens?  I would expect the
> call to pci_prepare_msix() from pciback to fail and thus also report
> some error?
> 
> I think it's likely I will have to provide an additional debug patch
> to Xen, maybe Jan has an idea of what could be going on.

No, sorry, not without - as you say - further debugging output added.

Jan