[Xen-devel] RE: Xen/ia64 presentation

["Magenheimer, Dan (HP Labs Fort Collins)" <dan.magenheimer@xxxxxx>]

Hi Hollis --

I hope its OK that I respond to you in xen-devel.  I know you
posted a similar question before the Xen summit and I apologize
that the message got buried and I never replied.

> In general, why are you so determined to track Linux code? Although 
> Xen/x86 started that way, it has since diverged, and I think 
> that makes 
> a lot of sense. There is likely to be a ton of Linux code supporting 
> features that don't make sense in a hypervisor, right? That has 
> certainly been my experience working on the PowerPC hypervisor over a 
> few years...

Here's a few reasons (apologies for the free associating :-)...

It is true that there is a "ton" of Linux code supporting features
that don't make sense in a hypervisor.  But there is a "couple hundred
pounds of gold/platinum" Linux code that DOES virtually identical things
to a hypervisor -- at least that is the case for ia64.  Unlike
Linux/x86, much of the core archdep code for Linux/ia64 is still
changing relatively rapidly (bug fixes and performance improvements).
When Xen was originally derived from Linux (2.4.7-ish), Linux/ia64
was very immature.  I really had no choice other than to leverage
a much later version.  And there have been many useful changes in
Linux/ia64 between 2.6.4 (where I started Xen/ia64) and 2.6.11 (which
is currently being used for Xen/ia64).

A friend likes to say that "good programmers are lazy... VERY good
programmers are VERY lazy".  While I can claim only to be lazy (and
the converse of this aphorism may not be true anyway :-), I hate to
waste my time writing code that has already been written by someone
else better than I could have written it anyway.  Assembly coding on
ia64 is very complicated and error prone; code like kernel entry/exit
on Linux/ia64 is some of the most intricate and cryptic code you may
ever see.  I have the highest level of respect for people like David
Mosberger who wrote this code but, frankly, I don't want to try to
maintain a parallel version of it.

IBM's pHype and vHype were written as closed source (and pHype still
is).
Xen and Xen/ia64 were written with open source and GPL in mind and
so there was no problem with leveraging Linux code directly.  I'd
venture to guess that 40%-60% of the code in Xen/x86 is directly
taken or indirectly derived (e.g. minor syntactic changes) from Linux.
Ian argues that, although this might be true, the other 40-60% is
the core value of Xen.  Which exactly supports my point: why should
the Xen team waste time on developing/changing the "non-core" part of
Xen when it can be directly leveraged from Linux?

It's true that Xen is slowly diverging from the original Linux code.
You see this as a good thing.  I disagree.  There are few good examples
of highly portable systems code, but Linux is one of them.  Many
things look "a little weird" in Linux because the code supports
multiple architectures.  A good example is allocating a task struct
(see below).  I'd argue that one of the key factors for Linux's current
and future success is its ubiquity due its portability; and Xen would
do well to follow in its footsteps rather than diverge in ways that
will very likely make it less portable.

Maintaining close ties to Linux makes it much easier to "go back to
the well".  I remember that Xen 1.0 removed a lot of the early
"start of day" code for other (e.g. Cyrix) processors; when the
user community grew and some users wanted to run on other processors,
the Xen team went back and grabbed the code from Linux.  When the
ACPI tables needed to be more fully parsed, the Xen team grabbed
the ACPI code from Linux.  Recently, when I needed to add "user access"
capability to Xen/ia64, I grabbed the exception table code from
Linux/ia64 (and Linux common code); it basically just worked, in part
because Keir had grabbed the x86 equivalent code earlier.

And I expect this trend to continue and not slow down.  For example,
when I start SMP support, all the important places are marked in
Linux code by CONFIG_SMP.  When I want to implement hot plug memory/CPUs
or NUMA or jiffieless timer support, I'll know where to look.  Let's
face it: Linux has and will always have a larger community than Xen.
It just makes sense to me to leverage as much as we can from that
community as easily as we can.

Some may argue that this is all a good reason to put Xen and Linux
in the same source tree.  I'm not sure I would go that far.  But,
please, let's not kill the golden goose for what amounts to a little
convenience and aesthetics.

Whew... sorry to be so long-winded!

> You said SMP support is still on the TODO list, and sure enough 
> CONFIG_SMP is undefined in the unstable tree. But when I did that for 
> PowerPC, I found build breakage in common code. x86 doesn't 
> even build 
> without CONFIG_SMP. Does that work for you?

Yes it does work, although I remember I had to add an ugly hack or
two in asm-ia64/config.h to allow it to compile without CONFIG_SMP.
Personally I think Xen should compile and work properly with
CONFIG_SMP not specified, but once I found a workaround, I gave
up the battle.

> What do you mean by this statement regarding Xen/x86: "struct 
> exec_domain is a state vector (not a memory area)"?

Sorry this was cryptic... I did a poor job of condensing a
complicated topic to a single bullet.

On Linux/x86 and Xen/x86, a task_struct (called an exec_domain
in Xen but the concept is very similar) is just a C struct which
is used to save state.  On Linux/ia64 (and I think some other
architectures) and Xen/ia64, a task_struct is really a page of
memory.  It contains two stacks and state is sprinkled liberally
throughout.  Why?  Because ia64 has so many registers that lots
of optimizations are done to only save/restore registers "lazily"
so registers may be saved at seemingly random places in one
of the stacks.  (This is why kernel entry/exit code is so intricate.)

BTW this is a good example of the "portability" issue with
divergence of Xen from Linux.  Long ago, Keir looked at the code
for alloc_task_struct and decided: "Why bother making a function
call to something which just kmalloc's a struct?" and so he moved
it inline.  The reason became clear with Xen/ia64 and I got him
to change it "back".  But recently, when cleaning up the code,
he nearly removed the abstraction again.

> Unrelated to the presentation, I've noticed a large amount of 
> commented-out code and definitions in Xen/ia64 code, and 
> other oddities 
> like "typedef struct exec_domain VCPU". Are these artifacts of 
> borrowing code from other sources? Are you planning on doing some 
> cleanup in the future?

Yeah, there are some artifacts and some cleanup would be good.
The typedef predates exec_domain (used to be a typedef of
struct domain) and I used it because I knew that exec_domain
was coming.  Personally, I prefer VCPU (virtual CPU) to exec_domain.

Dan

> -----Original Message-----
> From: Hollis Blanchard [mailto:hollisb@xxxxxxxxxx] 
> Sent: Wednesday, April 27, 2005 6:59 AM
> To: Magenheimer, Dan (HP Labs Fort Collins)
> Subject: Xen/ia64 presentation
> 
> Hi Dan, now that the Xen Summit PDFs are online, I just read through 
> your presentation on Xen/ia64. A few questions...
> 
> You said SMP support is still on the TODO list, and sure enough 
> CONFIG_SMP is undefined in the unstable tree. But when I did that for 
> PowerPC, I found build breakage in common code. x86 doesn't 
> even build 
> without CONFIG_SMP. Does that work for you?
> 
> What do you mean by this statement regarding Xen/x86: "struct 
> exec_domain is a state vector (not a memory area)"?
> 
> Unrelated to the presentation, I've noticed a large amount of 
> commented-out code and definitions in Xen/ia64 code, and 
> other oddities 
> like "typedef struct exec_domain VCPU". Are these artifacts of 
> borrowing code from other sources? Are you planning on doing some 
> cleanup in the future?
> 
> In general, why are you so determined to track Linux code? Although 
> Xen/x86 started that way, it has since diverged, and I think 
> that makes 
> a lot of sense. There is likely to be a ton of Linux code supporting 
> features that don't make sense in a hypervisor, right? That has 
> certainly been my experience working on the PowerPC hypervisor over a 
> few years...
> 
> -- 
> Hollis Blanchard
> IBM Linux Technology Center
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel