Re: [Xen-devel] Re: [RFC] RAS(Part II)--MCA enalbing in XEN

[Christoph Egger <Christoph.Egger@xxxxxxx>]

I realize from this and earlier MCE patches from Intel,
that Intel tries to change the machine check design
on its ground.

The basic ideas behind current design:

1. Xen collects error telemetry
2. Xen delivers correctable errors to Dom0 via VIRQ
3. Xen delivers uncorrectable errors to Dom0 via trap handler
4. Xen delivers uncorrectable errors to DomU only if Dom0 tells Xen to do so
5. Xen performs health measurements as told by Dom0 via hypercalls
   such as cpu- or page-offlining
6. Dom0 performs error analysis, figures out what is going on,
    calls hypercalls for the right health measurement


The basic ideas behind Intel's new design (as far as I can see them from their 
patches I have seen so far):

1. Xen collects error telemetry
2. Xen performs error analysis, figures out what is going on
3. Xen automatically does health measurements automatically
    like cpu- and page-offlining
4. Xen delivers error telemetry to Dom0 via VIRQ for error logging only
    independent of the error type
5. Inject MCEs into the guest directly
6. Don't use the MCE trap handler at all


IMO, any design change should be discussed first and not changed
silently, since this will confuse everyone and noone will know 
what is the right thing to do in Xen and in Dom0 and this
in turn will lead to error prone, unmaintainable code in both
Xen and in Dom0

Christoph


On Monday 16 February 2009 14:34:36 Christoph Egger wrote:
> To me, it seems, the design has not been understood
> and now, the code becomes more and more unmaintainable
> bloat. I mean, the code is going to do far too much.
>
> - The MCE routines in Xen are only for error data *collection*.
>   Just pass it to Dom0 and that's it.
>   Dom0 will do the error analysis and figure out what do to.
>   It is the Dom0 which will do a hypercall to do things like
>   page-offlining or cpu offlining or whatever is needed.
>   Your code tries to move everyting back from Dom0 into the
>   hypervisor. I remember Keir having rejected my MCE patches
>   because he feared this bloat.
>
> - Dom0 VIRQ is for correctable errors only. Uncorrectable errors
>   are delivered via MCE trap. Dom0 and DomU register a handle
>   via set_trap_table hypercall. A non-registrated handler means,
>   the guest can't handle it by itself. Dom0 is always notified,
>   the guest becomes only notified
>   This seperation is completely ignored and misuse Dom0 VIRQ for everything
>   (therefore the bunch of superflous flags (see next point))
>
> - MCA flags: what are the differences between correctable
>   and recoverable ? what are the differences between uncorrectable,
>   polled, reset and cmci and mce types ?
>
> - You use dynamic memory allocation (which uses spinlocks) in MCE code
>   and you roll your own mce handling instead of using the generic API in
> mce.c I suppose, you don't understand it at all.
>
> - I attach the design document again, since I have the impression, noone
>   at Intel read it, hence the misunderstandings.
>
> I think, it is best to get Gavin's generic mce improvements upstream first.
>
> On Monday 16 February 2009 06:35:14 Ke, Liping wrote:
> > Hi, all
> > These patches are for MCA enabling in XEN. It is sent as RFC firstly to
> > collect some feedbacks for refinement if needed before the final patch.
> > We also attach one description txt documents for your reference.
> >
> > Some implementation notes:
> > 1) When error happens, if the error is fatal (pcc = 1) or can't be
> > recovered (pcc = 0, yet no good recovery methods), for avoiding losing
> > logs in DOM0, we will reset machine immediately. Most of MCA MSRs are
> > sticky. After reboot, MCA polling mechanism will send vIRQ to DOM0 for
> > logging. 2) When MCE# happens, all CPUs enter MCA context. The first CPU
> > who read&clear the error MSR bank will be this MCE# owner. Necessary
> > locks/synchronization will help to judge the owner and select most severe
> > error. 3) For convenience, we will select the most offending CPU to do
> > most of processing&recovery job. 4) MCE# happens, we will do three jobs:
> > a. Send vIRQ to DOM0 for logging
> >     b. Send vMCE# to Impacted Guest (Currently Only inject to impacted
> > DOM0) c. Guest vMCE MSR virtualization
> > 5) Some further improvement/adds might be done if needed:
> >     a) Impacted DOM judgement algorithm.
> >     b) Now vMCE# injection is controlled by centralized data(vmce_data).
> > The injection algorithm is a bit complex. We might change the algorithm
> > which's based on PER_DOM data if you preferred. Notes for understanding:
> >         1) If several banks impact one domain, yet those banks belong to
> > the same pCPU, it will be injected only once. 2) If more than one bank
> > impact one domain, yet error banks belong to different pCPU, ith will be
> > injected nr_num(pCPU) times. 3) We use centralized data [two arrays
> > impact_domid, impact_cpus map in vmce_data] to represent the injection
> > algorithm. Combined the two array item (idx, impact_domid) and (idx,
> > impact_cpus) into one item (idx, impact_domid, impact_cpus). This item
> > records the impact_domain id and the error pCPU map (Finding UC errors on
> > this CPU which impact this domain). Then, we can judge how to inject the
> > vMCE (domid, impact_times[nr_pCPUs]).
> >         4) Although data structure is ready, we only inject vMCE# to
> > DOMD0 currently. c) Connection with recovery actions (cpu/memory
> > online/offline) d) More refines and tests for HVM might be done when
> > needed.
> >
> > Patch Description:
> > 1. basic_mca_support: Enable MCA support in XEN.
> > 2. vmsr_virtualization: Guest MCE# MSR read/write virtualization support
> > in XEN. 3. mce_dom0: Cooperating with XEN, DOM0 add vIRQ and vMCE#
> > handler. Translate XEN log to DOM0, re-use Linux kernel and MCELOG
> > mechanisms and MCE handler. This is mainly a demonstration patch.
> >
> > About Test:
> > We did some internal test and the result is just fine.
> >
> > Any feedback is welcome and thanks a lot for your help! :-)
> > Regards,
> > Criping



-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Jochen Polster, Thomas M. McCoy, Giuliano Meroni
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel