RE: [Xen-users] vcpus and cpus, plus cpu affinity = ??

["Petersson, Mats" <Mats.Petersson@xxxxxxx>]

> -----Original Message-----
> From: Liang Yang [mailto:multisyncfe991@xxxxxxxxxxx] 
> Sent: 08 February 2007 17:35
> To: Petersson, Mats
> Cc: Daniele Palumbo; xen-users@xxxxxxxxxxxxxxxxxxx
> Subject: Re: [Xen-users] vcpus and cpus, plus cpu affinity = ??
> 
> Hi Mats,
> 
> You mentioned a very interesting feature of AMD CPU: The AMD 
> setup for IO
> is that each CPU has three HT-links, which can either be used 
> for I/O or
> interprocessor communication, so each CPU can have it's own set of
> hardware that it can access.
> 
> Three questions (please bear with me if I misunderstood you):
> 1. How to bind a HT-link in a physcial CPU to I/O or IPC? Did 
> you do this in 
> BIOS?

The BIOS (or some similar startup software) will "enumerate" the links,
similar to how PCI devices are "detected", and also set up routing
tables to indicate which device connects to what other device (you can
have a PCI-X bridge that connects to a "southbridge" on consecutive
links. The actualy physical linking of devices is hard-linked by the
motherboard designer. Also, all processors can access any other
processors I/O hardware by using the routing tables, as well as using
the links for cache-coherency. Finally, as each of the processors have
it's own memory controller, the HT links are also used to do "remote
memory reads", i.e. processor 2 reading procesor 0's memory via the HT
link. 

> 
> 2. Do you know if there is any CPU from Intel supports such 
> kind of feature?

Not as of current - I have no idea what Intel plans for the future are. 
> 
> 3. Each CPU has three HT-links. Is the HT-link you mentioned 
> the same as the
> Hyper Threads in Intel CPU? I heard most of CPUs only have two HT.

HT in this case stands for HyperTransport, which is a chip interconnect
protocol. See: http://www.hypertransport.org/

HyperThreads is a mechanism to make a processor behave like it's a dual
core processor without implementing (much of) the second core - it's
advantageous on a processor with very long pipelines, as it allows a
second stream of instruction to be fed to the core of the processor, and
thus reduce the latency caused by a pipe-line stall. 

--
Mats
> 
> Thanks,
> 
> Liang
> 
> ----- Original Message ----- 
> From: "Petersson, Mats" <Mats.Petersson@xxxxxxx>
> To: "Daniele Palumbo" <daniele@xxxxxxxxxxxx>; 
> <xen-users@xxxxxxxxxxxxxxxxxxx>
> Sent: Thursday, February 08, 2007 3:08 AM
> Subject: RE: [Xen-users] vcpus and cpus, plus cpu affinity = ??
> 
> 
> 
> 
> > -----Original Message-----
> > From: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
> > [mailto:xen-users-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of
> > Daniele Palumbo
> > Sent: 08 February 2007 09:47
> > To: xen-users@xxxxxxxxxxxxxxxxxxx
> > Subject: [Xen-users] vcpus and cpus, plus cpu affinity = ??
> >
> > hi
> >
> > let's suppose i have something like that:
> > ---
> > Name                              ID  VCPU  CPU  State
> > Time(s)  CPU Affinity
> > Domain-0                           0     0    0   -b-
> > 24842.7  any cpu
> > Domain-0                           0     1    1   r--
> > 4306.2  any cpu
> > Domain-0                           0     2    2   -b-
> > 5171.3  any cpu
> > Domain-0                           0     3    3   -b-
> > 2013.0  any cpu
> > [snip]
> > trip                              18     0    2   -b-
> > 1189.8  any cpu
> > uni147                            19     0    0   r--   112415.0  0
> > uni147                            19     1    1   -b-   35259.4  1
> > [snip]
> > ---
> >
> > trip have 1 vcpu, cpus blank ("").
> > uni147 have 2 vcpus, cpus = "0-3" (i have 4 opteron 844).
> >
> > which is the difference between dom0 and uni147, or better
> > why one have "any
> > cpu" and the other have "0" and "1" in cpu affinity?
> >
> > how xen can decide to allocate to one cpu or another?
> > is supported something like "if cpu 0 will get full, move
> > vcpu 0 of uni147 to
> > cpu 3"?
> > and for trip?
> 
> If you "hard-assign" the cpus in your config file, then 
> that's the CPU's
> you get.
> If you don't, Xen just assings "at random".
> 
> Xen's Credit scheduler, which is what's in Xen by default 
> since 3.0.2 (I
> think), can, and will, move VCPU's between physical CPUs "for best
> utilization of the CPU resources". There's no "automatic" move things
> around between physical CPU's when you have hard-coded which CPU's you
> want (at least not outside the set!).
> 
> As to what works best for any particular setup, that's a much 
> harder to
> answer question - and one that you'll probably be better placed to
> answer than I am.
> >
> > am i wrong if i consider cpu0 the one that will get most of
> > IO of the HW?
> 
> That depends on the hardware on your motherboard. The AMD setup for IO
> is that each CPU has three HT-links, which can either be used 
> for I/O or
> interprocessor communication, so each CPU can have it's own set of
> hardware that it can access. However, in practice, most motherboards
> will attach most of the IO-devices to CPU0, since if you want 
> to run the
> system without a full set of CPU's, the first CPU is the one that you
> have to use... But there are systems with two PCI-X 
> controllers, one for
> CPU0 and another for CPU1 or some such (in which case, you 
> need to have
> two CPU's to access all PCI-X slots).
> 
> That doesn't in itself mean that CPU0 is doing more work 
> tho', as CPU1,
> 2, 3 can all access the hardware that is attached to CPU0, 
> just that it
> takes a fraction longer (but we're talking about PCI access with a
> 33/66/133MHz bus, compared to the 800MHz-1GHz of the HT-link, so the
> latency is most likely not at all noticable - the overhead is roughly
> 40ns per hop, where CPU{1,3} will have 1 hop, CPU2 has two hops). Note
> that a PCI access lasts at least 6-7 cycles in general 
> (200ns+), and not
> unusually longer than that.
> 
> > the cpu time of dom0 is affected by the usage of the domU for
> > that real cpu?
> 
> Dom0's "usage" will be entirely based on what Dom0 is doing, but of
> course, if DomU is sharing time with DomU, then the time used 
> by DomU is
> obviously not available to Dom0. Not sure what you're asking here...
> 
> >
> >
> > thanks
> > Daniele
> >
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@xxxxxxxxxxxxxxxxxxx
> > http://lists.xensource.com/xen-users
> >
> >
> >
> 
> 
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users
> 
> 
> 
> 



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users