[Xen-devel] RE: Xen/ia64 - global or per VP VHPT

["Munoz, Alberto J" <alberto.j.munoz@xxxxxxxxx>]

Hi Dan,

Magenheimer, Dan (HP Labs Fort Collins) <mailto:dan.magenheimer@xxxxxx>
wrote on Friday, April 29, 2005 1:05 PM:

>> In my opinion this is a moot point because in order to provide the
>> appropriate semantics for physical mode emulation (PRS.dt, or PSR.it, or
>> PSR.rt == 0) it is necessary to support a 4K page size as the minimum
>> (unless you special case translations for physical mode
>> emulation). Also in
>> terms of machine memory utilization, it is better to have
>> smaller pages (I
>> know this functionality is not yet available in Xen, but I
>> believe it will
>> become important once people are done working on the basics).
> 
> In my opinion, performance when emulating physical mode is
> a moot point.  

Linux IPF TLB miss handlers turn off PRS.dt. This is very performance
sensitive.

> It might make sense to simply not insert
> metaphysical addresses into the VHPT and just rely on the
> TLB (though perhaps a one-entry virtual TLB might be required
> to ensure forward progress).
> 
> Remember, one major difference between full virtualization (VT)
> and paravirtualization is that you have to handle any case that
> any crazy OS designer might try, while I just have to ensure that
> I can tell the crazy OS designer what crazy things need to be
> removed to make sure it works on Xen :-)  This guarantees that
> our design choices will sometimes differ.

I have not forgotten that (just as I have not forgotten this same argument
used in other contexts in the past, let's just do it this way because we
know no reasonable software will ever do that...) 

The way I see you applying this argument here is a bit different, though:
there are things that Linux does today that will cause trouble with this
particular design choice, but all I have to do is to make sure these
troublesome things get designed out of the paravirtualized OS.

In any case, I think it is critical to define exactly what an IPF
paravirtualized guest is (maybe this has already been done and I missed it)
before making assumptions as to what the guest will and will not do
(specially when those things are done by native guests today). I don't think
it is quiet the same as an X-86 XenoLinux, as a number of the hypercalls are
very specific to addressing X-86 virtualization holes, which do not have
equivalents in IPF. 

I know that there have been attempts at paravirtualizing (actually more like
dynamically patching) IPF Linux before (e.g., vBlades, you may be familiar
with it :-), but I am not sure if the Xen project for IPF has decided
exactly what an IPF paravirtualized XenoLinux will look like. I am also not
sure if it has also been decided that no native IPF guests (no binary
patching) will be supported.

>> It is not just purging. Having a global VHPT is, in general,
>> really bad for scalability....
> 
>> Another important thing is hashing into the VHPT. If you have ...
> 
>> As you point out this is ALWAYS the case, but what matters is
>> what are your target workloads and target systems are...
> 
> All this just says that a global VHPT may not be good for a
> big machine.  This may be true.  I'm not suggesting that
> Xen/ia64 support ONLY a global VHPT or even necessarily that
> it be the default, just that we preserve the capability to
> configure either (or even both).

Let's define "big" in an environment where there are multiple cores per
die...

Another argument (independent of scalability) here is that interference
between guests/domains in a virtualization environment should be minimized.
This particular design of a single vhpt is fostering this interference.

> 
> I wasn't present in the early Itanium architecture discussions
> but I'll bet there were advocates for both lVHPT and sVHPT who
> each thought it a terrible waste that the architecture support
> both.  That was silicon and both are supported; this is a small
> matter of software :-)

I was present during those early discussions and the argument went this way:
we need to support both Windows (a MAS OS) and HP-UX (a SAS OS) => we need
to support both short and long format VHPT.

> 
>> Memory footprint is really not that big a deal for these
>> large machines, but
>> in any case, the size of the VHPT is typically proportional
>> to the size of
>> physical memory (some people suggest 4 PTEs per physical page
>> frame and some
>> people suggest 2, but in any case, there is a linear
>> relationship between
>> the two). If you follow this guide line, then individual
>> VHPTs for 5 guests
>> should be 1/5 of the size of the combined VHPT for all 5 guests.
> 
> The point is that significant memory needs to be reserved in advance
> or dynamically recovered whenever a domain launches.  Maybe this
> isn't a big deal with a good flexible memory allocator and
> "hidden ballooning" to steal physical memory from running domains.

Going back to the example of 5 VHPTs of size X vs. one VHPT of size 5X, I
would say that this problem is worse with the single VHPT, as it either has
to have the ability to grow dynamically as domains get created, or has to be
pre-allocated to a size that supports a maximum number of domains.

> 
> E.g., assume an administrator automatically configures all domains
> with a nominal 4GB but ability to dynamically grow up to 64GB.  The
> per-guest VHPT would need to pre-allocate a shadow VHPT for the
> largest of these (say 1% of 64GB) even if each of the domains never
> grew beyond the 4GB, right?  (Either that or some kind of VHPT
> resizing might be required whenever memory is "hot-plugged"?)

I am not sure I understand your example. As I said in my previous posting,
experience has shown that the optimal size of the VHPT (for performance) is
dependent of the number of physical pages it supports (not how many domains,
but how many total pages those domains will be using). In other words, the
problem of having a VHPT support more memory is independent of whether it
represents one domain or multiple domains. It depends on how many total
memory pages are being supported. 

I believe that you somehow think that having a single VHPT to support
multiple domains would save you some memory, or rather the need to grow a
VHPT? Or put another way, why do you think that the situation you describe
above is unique to the multiple VHPT design and not to the single VHPT
design?

> 
> Again, there's a lot of interesting questions and discussion around
> this... which means its best to preserve our options if possible.

I see it a bit more black and white than you do.
 
> Dan

Bert

Attachment: smime.p7s
Description: S/MIME cryptographic signature

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel