Re: [Xen-devel] Possibilities with Xen3 compared to IBM Power5 DLPAR

[Mark Williamson <mark.williamson@xxxxxxxxxxxx>]

> On 15.03.2007 4:18 "Mark Williamson" <mark.williamson@xxxxxxxxxxxx> wrote:
> > Are we talking about Xen/PPC or Xen/x86 here?
>
> Mainly I'm talking about the x86 platform - but the feature list should
> apply to all platforms if possible :-)
> Although the Power5 platform could offer more direct support in hardware...
> I'm not sure if it where possible to use the hardware part of IBM's
> hypervisor in xen.

Well, the IBM guys are working on the PPC port but that's (as far as I know) 
currently directed towards PPC970 etc, rather than towards the higher-end 
POWER processors.  But who knows what we'll see!

> > I think the credit scheduler supports at least some of this.
>
> What exactly does that mean? :-)
> Are you talking about the basic support so that one could step into these
> features some day? I'm asking because I could not find any variables for
> the config files to set an entitlement or so. At the moment I think the
> smallest unit to assign processing power is per physical processor. But how
> can I assign this more granular, e.g. only a half physical processor (0.5
> entitlement) to 1 or x virtual processors? I'm also missing to have a
> variable to weight important domU's with higher priority than some small
> and unimportant domU's. At the moment each dom is equal to the others which
> maybe is not what professionals would like to have...

Hopefully the other information Ewan found for you has answered these 
questions.

My understanding of the credit scheduler is that it can indeed give out "n 
units worth" to a domain, added up across all its processors.  So for 
instance a 2 VCPU and a 1 VCPU domain with "1 unit's" entitlement will get 
the same total CPU bandwidth, even though one has two VCPUs being scheduled.

Usage can be capped so that a domain doesn't get more capacity than its paid 
for or slack time can (somehow) be distributed in the system.

That's how I *think* it works, but wasn't 100% which is why I glossed over 
it ;-)  Sounds roughly like what you wanted?

> > VCPU hotplug Just Works for XenLinux guests, I think.  Memory resizing
> > also works for XenLinux but it's worth noting that if you shrink a domain
> > too aggressively Linux gets confused about where its memory is going and
> > things start to break!
>
> Quite clear that the domU has to be xen-aware in order to re-read the
> memory allocaion table outside the domU space. And I was not really
> thinking a fully virtualized guest esp. like Windows could ever do so :-)

Well...  if the FV guest supported hotplug-capable hardware then maybe it 
could.  But in any case, you could in principle (it's not been done yet, and 
there were some problems to be solved) run the balloon driver in an HVM guest 
to enable it to change its memory footprint like a PV domain can.

I *suppose* one could do a similar thing with a driver that offlined CPUs in 
someway to support effective hotplugging.  This could probably also be done 
in an HVM guest with the addition of a few Xen-aware drivers.

> > DomU's VCPUs can be set to run on any thread context on the system.
>
> Yes sure but that's not my point. As you describe a domU would use each
> physical CPU it has access to but inside the domU each VCPU would look the
> same, the guest cannot make any difference between logical CPUs and virtual
> CPUs - it only has virtual CPUs then. But on the IBM platform the AIX guest
> is still aware that there are two logical CPUs linked with one virtual CPU.
> This is part of the hypervisor I think so the question is if Xen could also
> virtualize logical CPUs. IBM has no SMT on the level of the hypervisor,
> here they only have full physical CPUs. First in the logical partition you
> have the ability to enable SMT in addition to the assigned VCPUs or to
> disable it.

Ah.  Well that seems like a sensible way of working.  You could probably pin a 
domains VCPUs to a particular core if you wanted to achieve this sort of 
effect but I still don't know of a convenient way the domain can discover 
that they're siblings (and when they've changed relationships).  Maybe 
messing with CPUid can provide some hints.  There may be some topology stuff 
in there I don't know about, though, so I may be wrong.

Since Xen currently potentially schedules each VCPU on any logical thread in 
the system, there's probably not a very straightforward way for guests to 
make use of hyperthreading as the sibling relationships are potentially 
changing all the time.  In that case, maybe the IBM way is better, but if SMT 
is going to disappear on x86 machines then maybe it's not so important for 
now?  Just thinking out loud...

> > I'm not sure to what extent guests are aware of the SMT so they can use
> > it themselves; this would be particularly difficult for them to exploit
> > given they could in theory be moved to other logical CPUs at any time.
>
> The guest systems kernel does not really make a difference between logical
> and virtual processors I think, it's only interest is to have two units to
> send it's arithmetic problems to. But the whole guest operating system
> needs to have the ability to enable or disable SMT. On AIX it is only one
> command line but I'm sure it could be a point in the BIOS settings for Xen
> virtual machines. Although I don't even know if I can edit the BIOS
> settings directly when the vmachine starts up, never tried this :-) (with
> VMware you can do this).

I don't *think* you can do that with Bochs, but I don't know.

You couldn't disable hyperthreading in the guest as stands anyhow, but you 
*could* configure the guest such that hyperthreading is not used (by pinning 
CPUs) or you could switch it off for the whole host (in the host BIOS or noht 
on the Xen commandline should work).

> > Running dom0 in a dedicated thread context is useful though.
>
> Absolutely. But I'm not sure if dom0_cpus=1 is everything I need because I
> have no influence on Xen to only use one special core, in theory it can be
> always another core in each time slice. Does Xen have the intelligence here
> not to change the core? What about the guests, should I set cpus=1-7 in
> their config files (with a total of 8 cores)? Would the hypervisor really
> only use core 0 in that case?

There's a CPU-pinning syntax for specifying what host CPUs domains can / can't 
run on.  It's probably documented ... somewhere :-)

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel