Re: [Xen-devel] RFC: [PATCH 1/3] Enhance platform support for PCI

["Jan Beulich" <JBeulich@xxxxxxxx>]

>>> On 23.02.15 at 13:45, <ian.campbell@xxxxxxxxxx> wrote:
> On Mon, 2015-02-23 at 08:43 +0000, Jan Beulich wrote:
>> >>> On 20.02.15 at 18:33, <ian.campbell@xxxxxxxxxx> wrote:
>> > On Fri, 2015-02-20 at 15:15 +0000, Jan Beulich wrote:
>> >> > That's the issue we are trying to resolve, with device tree there is no
>> >> > explicit segment ID, so we have an essentially unindexed set of PCI
>> >> > buses in both Xen and dom0.
>> >> 
>> >> How that? What if two bus numbers are equal? There ought to be
>> >> some kind of topology information. Or if all buses are distinct, then
>> >> you don't need a segment number.
>> > 
>> > It's very possible that I simply don't have the PCI terminology straight
>> > in my head, leading to me talking nonsense.
>> > 
>> > I'll explain how I'm using it and perhaps you can put me straight...
>> > 
>> > My understanding was that a PCI segment equates to a PCI host
>> > controller, i.e. a specific instance of some PCI host IP on an SoC.
>> 
>> No - there can be multiple roots (i.e. host bridges)
> 
> Where a "host bridge" == what I've been calling "PCI host controller"?

Some problems here may originate in this naming: I'm not aware of
anything named "host controller" in PCI. The root of a PCI hierarchy
(single or multiple buses) connects to the system bus via a host
bridge. Whether one or more of them sit in a single chip is of no
interest (on the x86 and ia64 sides at least).

> I suppose in principal a single controller might expose multiple host
> bridges, but I think we can logically treat such things as being
> multiple controllers (e.g. with multiple CFG spaces etc).

Perhaps.

>>  on a single
>> segment. Segments are - afaict - purely a scalability extension
>> allowing to overcome the 256 bus limit.
> 
> Is the converse true -- i.e. can a single host bridge span multiple
> segments?

Not that I'm aware of.

> IOW is the mapping from segment->host bridge many->one or
> many->many?

Each segment may have multiple host bridges, each host bridge
connect devices on multiple buses. Any such hierarchy is entirely
separate from any other such hierarchy (both physically and in
terms of the SBDFs used to identify them).

> Maybe what I should read into what you are saying is that segments are
> purely a software and/or firmware concept with no real basis in the
> hardware?

Right - they just represent separation in hardware, but they have
no real equivalent there.

> In which case might we be at liberty to specify that on ARM+Device Tree
> systems (i.e. those where the f/w tables don't give an enumeration)
> there is a 1:1 mapping from segments to host bridges?

This again can only be answered knowing how bus number
assignment gets done on ARM (see also below). If all bus numbers
are distinct, there's no need for using segments numbers other
then zero. In the end, if you used segment numbers now, you may
end up closing the path to using them for much larger future setups.
But if that's not seen as a problem then yes, I think you could go
that route.

>> > So given a system with two PCI host controllers we end up with two
>> > segments (lets say A and B, but choosing those is the topic of this
>> > thread) and it is acceptable for both to contain a bus 0 with a device 1
>> > on it, i.e. (A:0:0.0) and (B:0:0.0) are distinct and can coexist.
>> > 
>> > It sounds like you are saying that this is not actually acceptable and
>> > that 0:0.0 must be unique in the system irrespective of the associated
>> > segment? iow (B:0:0.0) must be e.g. (B:1:0.0) instead?
>> 
>> No, there can be multiple buses numbered zero. And at the same
>> time a root bus doesn't need to be bus zero on its segment.
> 
> 0:0.0 was just an example I pulled out of thin air, it wasn't supposed
> to imply some special property of bus 0 e.g. being the root or anything
> like that.
> 
> If there are multiple buses numbered 0 then are they distinguished via
> segment or something else?

Just by segment.

>> What I don't get from this is where the BDF is being represented.
> 
> It isn't, since this is representing the host controller not any given
> PCI devices which it contains.
> 
> I thought in general BDFs were probed (or even configured) by the
> firmware and/or OS by walking over the CFG space and so aren't
> necessarily described anywhere in the firmware tables.

They're effectively getting assigned by firmware, yes. This mainly
affects bridges, which get stored (in their config space) the
secondary and subordinate bus numbers (bounding the bus range
they're responsible for when it comes to routing requests). If on
ARM firmware doesn't assign bus numbers, is bridge setup then a
job of the OS?

> FWIW the first 4 bytes in each line of interrupt-map are actually
> somehow matched against the masked (via interrupt-map-mask) against an
> encoding of the BDF to give the INTx routing, but BDFs aren't
> represented in the sense I think you meant in the example above.
> 
> There is a capability to have child nodes of this root controller node
> which describe individual devices, and there is an encoding for the BDF
> in there, but these are not required. For reference I've pasted a DT
> snippet from a Nvidia Jetson TK1 (Tegra124) system which has child
> nodes. I think the BDF is encoded in assigned-addresses somewhere.

Even if the BDF can be derived out of the addresses there it still
doesn't make clear to me how a more complex topology (namely
including bridges) would be represented. As said above - a bridge
needs to know which buses to route requests for, and hence I
can't see how you can get away without using bus numbers at all
in this process; all I can see is that the DT description can get
away without it, by simply representing the hierarchies that on
x86 one would discover by scanning the config space for devices.
I.e. - as indicated above - if bus number assignment is the OSes
job, then you would want to avoid segments as long as you can
get away with the 256 buses you have.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel