Re: [Xen-devel] PCI passthrough (pci-attach) to HVM guests bug (BAR64 addresses are bogus)

["Jan Beulich" <JBeulich@xxxxxxxx>]

>>> On 12.11.14 at 16:14, <konrad.wilk@xxxxxxxxxx> wrote:
> On Wed, Nov 12, 2014 at 10:01:28AM +0000, Malcolm Crossley wrote:
>> I agree with Jan. By using xl pci-attach you are effectively hotplugging
>> a PCI device (in the bare metal case). The only way this will work
>> reliably is if you reserve some MMIO space for the device you are about
>> to attach. You cannot just use space above the 4G boundary because the
>> PCI device may have 32 bit only BAR's and thus it's MMIO cannot be
>> placed at addresses above 4G.
> 
> Is it safe to split the BARs to be in different locations? Say stash
> all 64-bit BARs above 4GB and put all 32-bit under 4GB?

Sure.

>> The problem you have is that you cannot predict how much MMIO space to
>> reserve because you don't know in advance how many PCI device's you are
>> going to hotplug and how much MMIO space is required per device.
> 
> Perhaps following Jan's advice allow "bigger" MMIO ranges to be
> predefined: 4GB, 8Gb, 16GB, etc. And the larger ranges would cover
> space under 4GB (so say max 3GB) while the rest is spilled past the 4GB
> past the 'maxmem' range?

Not sure how you'd put a 4G or even 8G hole below the 4G
boundary... These BIOS settings only ever relate to space up to
4G (at least as far as I had seen them).

>> To do HVM pci hotplug properly we need to reserve MMIO space below 4G
>> and emulate a PCI hotplug capable PCI-PCI bridge device. The bridge
>> device will know the maximum size of the MMIO behind it (as allocated at
>> boot time) and so we can calculate if the device we are hotplugging can
>> fit. If it doesn't fit then we fail the hotplug otherwise we allow it
>> and the OS will correct allocate the BAR behind the bridge.
> 
> I think that can be done right now for the MMIO and _CRS in hvmloader
> and libxc/libxl. I wonder if that can all be done without having an
> PCI-PCI bridge device introduced?

I think it could, even if that possibly wouldn't be 100% spec conforming.

>> BTW, calculating the required MMIO for multi BAR PCI device's is not
>> easy because all the BAR's need to be aligned to their size (naturally
>> aligned).
> 
> Ouch. So two 512MB and an 1GB can't be next to each but would need:
> 512GB BAR<-- 512GB space--->| 1GB BAR.
> 
> Or just put the 1GB first:
> 
> 1GB BAR | 512GB 
> 
> ?

The latter is what one should prefer.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel