[Xen-devel] RE: Resend: RE: enable_ats_device() call site

["Jan Beulich" <JBeulich@xxxxxxxx>]

>>> On 07.10.11 at 04:19, "Kay, Allen M" <allen.m.kay@xxxxxxxxx> wrote:
> Currently I three places where enable_ats_device() can be eventually called 
> because of it is in domain_context_mapping():
> 
> 1) setup_dom0_devices()
> 2) intel_iommu_add_device()
> 3) reassign_device_ownership()
> 
> Calling it in the first two is probably all we need to cover all the 
> devices.   I don't think we need to call it again in 
> reassign_device_ownership().

That would mean that a device that was found during Xen's initial scan
(i.e. pdev->domain set due to it having gone through
setup_dom0_pci_devices()), but for which enable_ats_device() was
unsuccessful due to mmcfg access still being impossible at that point,
would never get ATS enabled. But that's the whole point of the thread
here. My question isn't whether to *re*move call sites, but whether it
would be possible to move them elsewhere. For which I'd like to
understand why this is being done in the places it is now (not the
least why this is done in VT-d specific code in the first place).

Just like in the suggested change to how pci_enable_acs() gets called,
this should really happen from pci_add_device() *without* regard to
whether pdev->domain was already set.

Also, earlier you suggested to remove the call to pci_enable_acs() from
setup_dom0_device() - I'm not convinced anymore that this is correct,
since old Dom0 kernels (forward ports from the 2.6.18 tree up to pretty
recently) can't be relied upon to report all PCI devices to Xen. Which
also suggests that we shouldn't really remove scan_pci_devices()
(although it may be possible to adjust it back to scan only segment 0).
Otoh, when mmcfg isn't available early, on such Dom0 kernels ATS
wouldn't get enabled today either, even with the adjusted call site of
pci_enable_acs().

Consequently, another alternative would be to retry ATS and ACS
enabling when mmcfg becomes available for a certain bus range, i.e.
out of pci_mmcfg_reserved().

Jan

> By the way, due to the lack of production ATS devices, we have not tried ATS 
> for quite a while.  I'm not sure we should make the change now or should we 
> just make a note of it in reassign_device_ownership() for now.
> 
> Allen
> 
> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@xxxxxxxxxx] 
> Sent: Friday, August 19, 2011 2:26 AM
> To: Kay, Allen M
> Subject: Resend: RE: enable_ats_device() call site
> 
> (for some reason the first send to you bounced - please re-add xen-devel as 
> cc 
> if you reply to this one)
> 
>>>> On 18.08.11 at 01:27, "Kay, Allen M" <allen.m.kay@xxxxxxxxx> wrote:
>>>  what is the reason for calling this from VT-d's domain_context_mapping()?
>>> I neither undertsand why this is VT-d specific, nor why it needs to be
>>> re-done with each device re-assignment.
>> 
>> The reason is FLR clears the ATS enabled bit so we need to re-enable it for 
>> every re-assignment.  The reason we don't need to do this for ACS might be 
> ACS 
>> reside on the bridge, not in the PCI endpoint.  ATS on the other hand, 
>> resides in PCI endpoints.
> 
> And why is it VT-d specific then? The problem to solve is that enabling
> may not happen when it is first attempted, in the case where Xen on its
> own can't be certain that using MMCFG is safe. Hence when the device
> gets reported by Dom0 (or when MMCFG gets enabled for the respective
> bus), another attempt needs to be made at enabling it. De-assigning and
> then re-assigning the device to Dom0 seems to be overkill to me.
> 
>>> Alternatively - why do we need scan_pci_devices() at all? We're
>>> supposed to be getting the devices reported from Dom0 anyway
>> 
>> Looks like it is use for building bus2bridge[] which is used for figuring 
>> out upstream bridges which are needed when assigning non-PCIe devices.
> 
> Oh, right, I keep forgetting that, especially as that puts under question
> why we have Dom0 report non-extfn, non-virtfn devices at all. And
> perhaps we should issue a warning if Dom0 reports such a device that
> we didn't know about already?
> 
> Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel