Re: [Xen-devel] [PATCH] xen/mcfg: Call PHYSDEVOP_pci_mmcfg_reserved before PCI enumeration

["Jan Beulich" <JBeulich@xxxxxxxx>]

>>> On 22.09.15 at 16:33, <konrad.wilk@xxxxxxxxxx> wrote:
> On Tue, Sep 22, 2015 at 08:12:56AM -0600, Jan Beulich wrote:
>> >>> On 22.09.15 at 16:09, <konrad.wilk@xxxxxxxxxx> wrote:
>> > On Tue, Sep 22, 2015 at 07:57:29AM -0600, Jan Beulich wrote:
>> >> >>> On 22.09.15 at 15:36, <konrad.wilk@xxxxxxxxxx> wrote:
>> >> > The best I could come up with is to do two loops:
>> >> >  1) for 0:0:0 -> ff:ff:ff call PHYSDEVOP_pci_device_remove
>> >> >     (so blow away what Xen has for its PCI devices.. except for the AMD 
>> >> > IOMMU)
>> >> >  2) list_for_each_pci_device PHYSDEVOP_pci_device_add (or other variants
>> >> >     if VF).
>> >> > 
>> >> > But that is just a hack working around the Linux code.
>> >> 
>> >> The price of being non-intrusive on the Linux side. The above would
>> >> seem okay to me for the (rare?) reassign case. (Not sure why you
>> >> mean to exclude the AMD IOMMU there.)
>> > 
>> > I think we hide it altogether from the dom0 - so when it does the
>> > bus reassignment it does not see the AMD IOMMU.
>> > 
>> > My concern was that the bus number of the AMD IOMMU device may
>> > overlap with other bus numbers - which got moved as a result
>> > of the bridge expanding its subordinate bus numbers.
>> > 
>> > In other words, the AMD IOMMU may have been at bus 0xb, the
>> > bridge in question at 0x01, with an PCI device at 0x02;
>> > some devices between 0x03 down to 0x0a.
>> > 
>> > The bridge would now cover 0x01->0x04 (with the PCI device
>> > still at 0x02). But the devices at the old 0x03->0x0a have now
>> > been shifted to 0x04->0x0b. And the AMD IOMMU is at 0x0c.
>> > 
>> > But Xen has no clue about this and "loses" the AMD IOMMU and
>> > starts writting configuration data to whatever poor device used to
>> > be at bus 0x0b.
>> 
>> And how would that issue be solved by leaving that device alone?
> 
> If it was exposed to dom0, it could make an PHYSDEVOP_pci_device_add
> to add it back to Xen (as a new device at bus 0x0c).
> 
> But Xen wouldn't know what to do with it - it would still think that
> the AMD IOMMU is at bus 0x0b.

Ah, now I see what you're getting at: It's the AMD IOMMU code itself
which would need to get notified. That's even more of an orthogonal
issue than the general mixing in of bus reassignment you allude to ...

> I believe I am going on a tangent here - that is the MMCFG
> conversation is about the extended configuration registers and how
> to make Xen aware of them such that when the VF is added Xen will
> be able to validate them. Ed's idea of making the MMCFG hypercall
> when the PCI notification (with your suggestions) should take care
> of it (I hope).
> 
> The issue I am waxing on about is just Linux reassigning the
> PCI devices and what are some of the repercussions that stem from that.

... here. (As a side note - I don't think we do or even reasonably
could hide that device from being found during a bus scan. All we
do is disallow it being played with by Dom0 or getting assigned to
a DomU.)

> I should start a new thread about it and propose some prototype
> patch, once my TODO queue is a bit sane.

Much appreciated, thanks.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel