Re: [Xen-devel] [PATCH 5/5] AMD IOMMU: widen NUMA nodes to be allocated from

[Suravee Suthikulanit <suravee.suthikulpanit@xxxxxxx>]

On 3/6/2015 6:15 AM, Andrew Cooper wrote:
> On 06/03/2015 07:50, Jan Beulich wrote:
> > > > > On 05.03.15 at 18:30, <andrew.cooper3@xxxxxxxxxx> wrote:
> > > On 26/02/15 13:56, Jan Beulich wrote:
> > > > --- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
> > > > +++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
> > > > @@ -158,12 +158,12 @@ static inline unsigned long region_to_pa
> > > >       return (PAGE_ALIGN(addr + size) - (addr & PAGE_MASK)) >> PAGE_SHIFT;
> > > >   }
> > > >
> > > > -static inline struct page_info* alloc_amd_iommu_pgtable(void)
> > > > +static inline struct page_info *alloc_amd_iommu_pgtable(struct domain *d)
> > > >   {
> > > >       struct page_info *pg;
> > > >       void *vaddr;
> > > >
> > > > -    pg = alloc_domheap_page(NULL, 0);
> > > > +    pg = alloc_domheap_page(d, MEMF_no_owner);
> > > Same comment as with the VT-d side of things.  This should be based on
> > > the proximity information of the IOMMU, not of the owning domain.
> > I think I buy this argument on the VT-d side (under the assumption
> > that there's going to be at least one IOMMU per node), but I'm not
> > sure here: The most modern AMD box I have has just a single
> > IOMMU for 4 nodes it reports.
> It is not possible for an IOMMU to cover multiple NUMA nodes worth of
> IO, because of the position it has to sit relative to the IO root ports
> and QPI/HT links.
>
> In AMD systems, the IOMMUs lives in the northbridges, meaning one per
> numa node (as it is the northbridges which contain the hypertransport links)
>
> The BIOS/firmware will only report IOMMUs from northbridges which have
> IO connected to their IO hypertransport link (most systems in the wild
> have all IO hanging off one or two Numa nodes).  On the other hand, I
> have an AMD system with 8 IOMMUs in use.

Actually, a single IOMMU could handle multiple nodes. For example, in 
scenario of a multi-chip-module (MCM) setup, there could be at least 2-4 
nodes sharing one IOMMU depending on how the platform vendor configuring 
the system. In the server platforms, IOMMU is in AMD northbridge 
chipsets (e.g. SR56xx). This website has an example of such system 
configuration 
(http://www.qdpma.com/systemarchitecture/SystemArchitecture_Opteron.html).
For AMD IOMMU, the IVRS table specifies the PCI bus/device ranges to be 
handled by each IOMMU. This is probably should be considered here.
> In Intel systems, there is one IOMMU for each socket (to cover the
> on-chip root ports and GPU if applicable) and one IOMMU in the IOH/PCH
> (depending on generation) to cover the legacy IO.
>
>
> In all cases, the IOMMUs are local to a single NUMA node, and would
> benefit from having the control pages and pagetables allocated in local RAM.
As state above, this is not the case for AMD IOMMU.

Thanks,

Suravee
> ~Andrew


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel