Re: [Xen-devel] Xen, Windows 2003 and APIC

[Gary Pennington <Gary.Pennington@xxxxxxx>]

On Fri, Dec 05, 2008 at 05:06:28PM +0000, George Dunlap wrote:
> On Fri, Dec 5, 2008 at 4:57 PM, Gary Pennington <Gary.Pennington@xxxxxxx> 
> wrote:
> > However, what I was mostly curious about was why specifying apic when the
> > guest was created made so much different to performance.
> 
> Because of how the interrupt blocking works with APIC vs non-apic.
> For APIC (on Intel boxes), you set the level of interrupts that you
> want to receive (called the TPR, or Task Priority Register) by writing
> to an MMIO register.  Under Xen, this means a vmexit every time the
> TPR is written.  w2k3sp0 writes to the TPR every time the priority
> changes, which happens a lot.  Each write causes a
> vmexit/update/vmenter cycle, which is pretty expensive.
> 
> w2k3sp2 has a technique called "lazy TPR", where it manages to avoid
> the vast majority of TPR writes.  This results in a much lower
> overhead, probably on bare hardware as well, but especially in
> HVM-virtualized environments.
> 
> If you take a xen trace of w2k3sp0, you will see a ton of MMIO writes
> to ffe00080; that's the TPR register (if I recall correctly from the
> top of my head).  xenalyze (Coming Soon[tm]) would look at the trace
> and tell you that you were spending a non-negligible percentage of
> your time handling MMIO writes to the TPR register.
> 
> If you disable the APIC, the processor uses other methods of setting
> interrupt levels (don't know these off the top of my head) which are
> virtualized as a part of the HVM spec, and are therefore don't involve
> the hyperivsor at all.  As Keir said, newer Intel hardware has TPR
> virtualization, so even with the APIC accessing the TPR doesn't
> involve the hypervisor.
> 

Thanks very much for the explanation. That makes it clear(ish) why
specifying APIC alters performance so much. Now what I need to understand
is why certain Windows variants default to specify APIC and others don't...

It seems like this area is a bit of a minefield and it would be good to get
authoritative data on how to specify apic/acpi for windows guests, depending
on hardware available and flavour of windows...

Gary

>  -George
> 
> >
> > Gary
> >
> >> Hopefully, sometime in the next few weeks, I'll be able to release my
> >> 'xenalyze' tool, which will help a lot with analyzing what's really
> >> going on with these kinds of workloads.
> >>
> >>  -George
> >>
> >> On Fri, Dec 5, 2008 at 2:45 PM, Keir Fraser <keir.fraser@xxxxxxxxxxxxx> 
> >> wrote:
> >> > On 05/12/2008 14:03, "Gary Pennington" <Gary.Pennington@xxxxxxx> wrote:
> >> >
> >> >> My next question is: What is really happening when APIC is specified for
> >> >> a windows guest and why does performance vary so much according to 
> >> >> whether
> >> >> it's specified or not?
> >> >
> >> > Older Windows kernels update the APIC TPR a lot, and unless you have a 
> >> > very
> >> > modern Intel processor every one of those TPR updates causes a vmexit.
> >> >
> >> > Modern Windows (including possibly latest w2k3 service pack, but I'm not
> >> > totally certain) includes lazy TPR, which gets rid of the vast majority 
> >> > of
> >> > TPR updates, and hence will go much faster.
> >> >
> >> >  -- Keir
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > Xen-devel mailing list
> >> > Xen-devel@xxxxxxxxxxxxxxxxxxx
> >> > http://lists.xensource.com/xen-devel
> >> >
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@xxxxxxxxxxxxxxxxxxx
> >> http://lists.xensource.com/xen-devel
> >
> > --
> > Gary Pennington
> > Solaris Core OS
> > Sun Microsystems
> > Gary.Pennington@xxxxxxx
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxxxxxxxx
> > http://lists.xensource.com/xen-devel
> >

-- 
Gary Pennington
Solaris Core OS
Sun Microsystems
Gary.Pennington@xxxxxxx

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel