Re: Segment truncation in multi-segment PCI handling?

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Mon, Jun 10, 2024 at 09:58:11AM +0200, Jan Beulich wrote:
> On 07.06.2024 21:52, Andrew Cooper wrote:
> > On 07/06/2024 8:46 pm, Marek Marczykowski-Górecki wrote:
> >> Hi,
> >>
> >> I've got a new system, and it has two PCI segments:
> >>
> >>     0000:00:00.0 Host bridge: Intel Corporation Device 7d14 (rev 04)
> >>     0000:00:02.0 VGA compatible controller: Intel Corporation Meteor 
> >> Lake-P [Intel Graphics] (rev 08)
> >>     ...
> >>     10000:e0:06.0 System peripheral: Intel Corporation RST VMD Managed 
> >> Controller
> >>     10000:e0:06.2 PCI bridge: Intel Corporation Device 7ecb (rev 10)
> >>     10000:e1:00.0 Non-Volatile memory controller: Phison Electronics 
> >> Corporation PS5021-E21 PCIe4 NVMe Controller (DRAM-less) (rev 01)
> >>
> >> But looks like Xen doesn't handle it correctly:

In the meantime you can probably disable VMD from the firmware and the
NVMe devices should appear on the regular PCI bus.

> >>     (XEN) 0000:e0:06.0: unknown type 0
> >>     (XEN) 0000:e0:06.2: unknown type 0
> >>     (XEN) 0000:e1:00.0: unknown type 0
> >>     ...
> >>     (XEN) ==== PCI devices ====
> >>     (XEN) ==== segment 0000 ====
> >>     (XEN) 0000:e1:00.0 - NULL - node -1 
> >>     (XEN) 0000:e0:06.2 - NULL - node -1 
> >>     (XEN) 0000:e0:06.0 - NULL - node -1 
> >>     (XEN) 0000:2b:00.0 - d0 - node -1  - MSIs < 161 >
> >>     (XEN) 0000:00:1f.6 - d0 - node -1  - MSIs < 148 >
> >>     ...
> >>
> >> This isn't exactly surprising, since pci_sbdf_t.seg is uint16_t, so
> >> 0x10000 doesn't fit. OSDev wiki says PCI Express can have 65536 PCI
> >> Segment Groups, each with 256 bus segments.
> >>
> >> Fortunately, I don't need this to work, if I disable VMD in the
> >> firmware, I get a single segment and everything works fine.
> >>
> > 
> > This is a known issue.  Works is being done, albeit slowly.
> 
> Is work being done? After the design session in Prague I put it on my
> todo list, but at low priority. I'd be happy to take it off there if I
> knew someone else is looking into this.

We had a design session about VMD?  If so I'm afraid I've missed it.

> > 0x10000 is indeed not a spec-compliant PCI segment.  It's something
> > model specific the Linux VMD driver is doing.
> 
> I wouldn't call this "model specific" - this numbering is purely a
> software one (and would need coordinating between Dom0 and Xen).

Hm, TBH I'm not sure whether Xen needs to be aware of VMD devices.
The resources used by the VMD devices are all assigned to the VMD
root.  My current hypothesis is that it might be possible to manage
such devices without Xen being aware of their existence.

Regards, Roger.