Re: PCI pass-through problem for SN570 NVME SSD

[Jan Beulich <jbeulich@xxxxxxxx>]

(Note: please don't cross-post; removing xen-users@.)

On 04.07.2022 08:37, G.R. wrote:
> Update some findings with extra triage effort...
> Detailed log could be found in the attachments.
> 1. Confirm stock Debian 11.2 kernel (5.10) shares the same syndrome..
> 2. With loglvl=all, it reveals why the mapping failure happens, looks
> like it comes from some duplicated mapping..
> (XEN) memory_map:add: dom1 gfn=f3074 mfn=a2610 nr=2
> (XEN) memory_map:add: dom1 gfn=f3077 mfn=a2615 nr=1
> (XEN) memory_map:add: dom1 gfn=f3078 mfn=a2616 nr=1 <===========Here
> (XEN) ioport_map:add: dom1 gport=c140 mport=4060 nr=20
> (XEN) ioport_map:add: dom1 gport=c170 mport=4090 nr=8
> (XEN) ioport_map:add: dom1 gport=c178 mport=4080 nr=4
> (XEN) memory_map:add: dom1 gfn=f3070 mfn=a2500 nr=2
> (XEN) memory_map:add: dom1 gfn=f3073 mfn=a2503 nr=1
> (XEN) memory_map:add: dom1 gfn=f3078 mfn=a2504 nr=1 <===========Here
> (XEN) d1: GFN 0xf3078 (0xa2616,0,5,7) -> (0xa2504,0,5,7) not permitted
> (XEN) domain_crash called from p2m.c:1301
> (XEN) Domain 1 reported crashed by domain 0 on cpu#2:
> (XEN) memory_map:fail: dom1 gfn=f3078 mfn=a2504 nr=1 ret:-1
> (XEN) memory_map:remove: dom1 gfn=f3078 mfn=a2504 nr=1

Neither here nor in your initial mail I've spotted information
on the BARs the device has. The above makes me wonder whether it
has two BARs each covering less than 4k and both sharing a page.
Or wait - the hvmloader output actually has some useful data:

(d1) pci dev 05:0 bar 24 size 000000800: 0f3077000
...
(d1) pci dev 05:0 bar 14 size 000000100: 0f3077800

The sharing is apparently introduced in hvmloader, but might not
have been deemed a problem because it's generally advisable (for
security reasons) or even necessary (for functionality) for BARs
of devices to be passed through to all live in distinct (4k) pages.
However - while hvmloader has no knowledge of host addresses
occupied by the BARs (so it has no indication to place them in
separate pages), it should still not put any two BARs in the same
(guest) page. Even then, as the P2M mapping occurs at 4k
granularity, it would further need to know the host's offset-into-
page value to correctly calculate the guest address. IOW this will
in addition require the host to put all BARs at the beginning of
4k pages (which may well be the case already for you).

(d1) pci dev 06:0 bar 20 size 000000100: 0f3077904

would cause the same issue (afaict), unless the host BAR shared a
page with the earlier BAR of 05:0.

> 3. Recompiled kernel with DEBUG enabled for xen_pciback driver and
> play with xl pci-assignable-XXX with it
> 3.1 It's confirmed that the DPC / AER error log happens only when
> xen_pciback attempts to seize && release the device
> 3.1.1 It only happens on each of the first add / remove operations.
> 3.2 There is still a 'MSI-X preparation failed' message later-on, but
> otherwise it appears to be successful to add / remove the device after
> the 1st attempt.
> 3.3 Not necessarily related, but the DPC / AER log looks similar to
> this report [1]

The only thing I can say here is that quite likely pciback needs
work to become up-to-date again with advanced feature handling
elsewhere in the kernel.

Jan

> [1]: 
> https://patchwork.kernel.org/project/linux-pci/patch/20220127025418.1989642-1-kai.heng.feng@xxxxxxxxxxxxx/#24713767