Re: [EXT] Re: DOMU: Virtual Function FLR in PCI passthrough is crashing

[Jan Beulich <jbeulich@xxxxxxxx>]

On 25.04.2022 13:01, Naresh Bhat wrote:
> From: Jan Beulich <jbeulich@xxxxxxxx>
> Sent: 20 April 2022 18:37
> 
> On 20.04.2022 14:48, Naresh Bhat wrote:
>> I have the following setup and try to test the Function Level Reset feature. 
>>  Any suggestions or pointers will be very much helpful.
>>
>> DOM0
>> Distribution: Ubuntu-20.04.3 (kernel 5.8.0-43)
>> Xen version : 4.11.4-pre
>>
>> DOMU
>> Distribution: Ubuntu-18.04.6 LTS (kernel 5.8.0)
>> PCIe device with SRIOV support, VF (Virtual Function) interface connected to 
>> DOMU via PCI pass-through
>>
>> Issue on DOMU: 
>> 1. Enable MSIX on DOMU (We have used the following kernel APIs 
>> pci_enable_msix_range, pci_alloc_irq_vectors)
>> 2. Execute FLR (Function Level Reset) via sysfs interface on the PCIe 
>> passthrough device in DOMU
>>     # echo "1" > /sys/bus/pci/devices/<ID>/reset
>>
>> The following crash observed 
>>
>> [ 4126.391455] BUG: unable to handle page fault for address: ffffc90040029000
>> [ 4126.391489] #PF: supervisor write access in kernel mode
>> [ 4126.391503] #PF: error_code(0x0003) - permissions violation
>> [ 4126.391516] PGD 94980067 P4D 94980067 PUD 16a155067 PMD 16a156067 PTE 
>> 80100000a000c075
>> [ 4126.391537] Oops: 0003 [#1] SMP NOPTI
>> [ 4126.391550] CPU: 0 PID: 971 Comm: bash Tainted: G           OE     5.8.0 
>> #1
>> [ 4126.391570] RIP: e030:__pci_write_msi_msg+0x59/0x150
>> [ 4126.391580] Code: 8b 50 d8 85 d2 75 31 83 78 fc 03 74 2b f6 47 54 01 74 
>> 6e f6 47 55 02 75 1f 0f b7 47 56 c1 e0 04 48 98 48 03 47 60 74 10 8b 16 <89> 
>> 10 8b 56 04 89 50 04 8b 56 08 89 50 08 48 8b 03 49 89 44 24 20
>> [ 4126.391606] RSP: e02b:ffffc90040407cc0 EFLAGS: 00010286
> 
> The RSP related selector value suggests you're talking about a PV DomU.
> Such a DomU cannot write the MSI-X table directly, yet at a guess (from
> the PTE displayed) that's what the insn does where the crash occurred. I
> would guess you've hit yet another place in the kernel where proper PV
> abstraction is missing. You may want to check with newer kernels.
> 
> [Naresh]: We have tested with latest kernel i.e. 5.17.0 kernel, issue 
> persists.

Thanks for checking.

> As to FLR - I guess this operation as a whole needs passing through
> pcifront to pciback, such that the operation can be carried out safely
> (e.g. to save and restore active MSIs, which is what I infer is being
> attempted here, as per the stack trace).
> 
> [Naresh]: Any idea when the support will be added to Xen ?

I'm not sure there's anything in need to be added to Xen, except for
an addition to the related public header (io/pciif.h). It's pcifront
and pciback (part of the kernel) which would need extending. I'm
unaware of anyone having plans in that direction. Maybe you want to
make an attempt?

Jan