1. EM&V Planning and EM&V Issues
Kentucky PSC 9/11/09 Schiller Consulting, Inc.

2. Planning and the Program Implementation Cycle
Kentucky PSC 9/11/09 Schiller Consulting, Inc.

3. Evaluation in Context: The Cycle of Program Planning and Implementation
Adapted by CEE from Pierre Landry, SCE, and Demand- Side Management, Vol. 4: Commercial Markets and Programs, EPRI, 1987.
7. A visual to help explain where evaluation fits in within the program AND resource planning process (refer them to a handout of this which we’ll send in advance)
Good evaluation planning begins very early, once program objectives are defined. The program objectives should be determined within the framework of the policy objectives.
Many different types of issues should be taken into account and balanced, such as
How important is peak demand reduction vs. overall energy usage reduction?
How important is equity? (EE benefits for disadvantaged customers or in proportion to contribution to EE funding)
These need to be considered in planning evaluation (box 2) to ensure that evaluation results will provide appropriate indicators to use in measuring progress towards these broad goals.
Then various program alternatives should be looked at to design a portfolio of programs (like a portfolio of stocks) and design individual programs within the portfolio. Successful portfolio design requires knowing what the EE potential is for various end uses and approaches within the state or service territory. Individual program design & selection involves assessing measures to cover (end uses), approaches to use to persuade customers to adopt the measures (marketing techniques), and selecting among them (which programs are most cost-effective at meeting the policy goals?). Eval planning should begin at this stage, taking into account program objectives in box #1. Getting evaluators involved at this stage, developing logic modeling to identifying early indicators of progress towards program goals, makes eval more efficient & effective.
Program implementation (first they should be tested, then pilot programs run, and then go full-scale after tweaking). Ideally, evaluation data collection should begin when programs begin. Reduces data collection problems and evaluation challenges later.
Once program is implemented process evaluation can take place, and savings estimates for impact evaluation can begin to be calculated with the data that should have been collected from beginning of program. The results of evaluation should be used to improve the design & execution of current programs. They should also be use to inform the selection and design of a new crop of programs, including determination of program objectives, and to inform the next round of resource planning and development of policy objectives.
Ideally, what results is a process of continuous improvement of program information and programs to inform resource planning and meet policy objectives.
Kentucky PSC 9/11/09 Schiller Consulting, Inc.

4. Determining Gross and Net Energy Savings
Kentucky PSC 9/11/09 Schiller Consulting, Inc.

5. Planning Issues
Defining evaluation goals and scale, and which benefits to evaluate
Setting time frame for evaluation and reporting expectations
Establishing budget vis-à-vis expectations for quality of reported results
Defining baseline, baseline adjustments, and data collection requirements
Selecting impact evaluation approaches for gross and net savings calculations and avoided emissions calculations
Selecting who (or which type of organization) will conduct the evaluations
Identifying and integrating stakeholders
Successful evaluations harmonize the costs of evaluation with the value of the information received—that is, they appropriately balance risk management, uncertainty, and cost considerations.
Kentucky PSC 9/11/09 Schiller Consulting, Inc.

6. How Good is Good Enough? -
Incremental certainty
Value of information
value of improved certainty
cost of improved EM&V
Cost of incremental EM&V
Balance point
of diminishing returns
Cost
< cost of improved EM&V
Kentucky PSC 9/11/09 Schiller Consulting, Inc.

7. Precision and Bias
Kentucky PSC 9/11/09 Schiller Consulting, Inc.

8. Technical Reference Manuals and Evaluation Protocols – EM&V Resources
In the simplest form ‘TRMs’ are databases of deemed savings values with applicability conditions
Examples of database contents are:
kWh, kW, therm savings values
Measure effective lifetimes
Net to gross ratios
Measure cost information
Local/regional utility cost information for TRC calculations
Work papers that document basis for values
Documents with varying level of detail that define how the evaluations (and M&V) are to be conducted with a particular jurisdiction. Contents might include:
Which approaches to use
Sample size requirements
Evaluation timing
Reporting requirements
Kentucky PSC 9/11/09 Schiller Consulting, Inc.

9. The Documentation (EM&V) Issue and Solution
Addressing the challenge of quantifying a counterfactual situation whose characteristics can never be proven ex-post, i.e. how much energy would have been consumed or emissions produced in the absence of the efficiency activity
Solution
Applying the experience and protocols and TRMs already in place - EM&V has and is being done for billions of dollars worth of programs
Conclusion
EM&V can be defined and implemented to a level of sufficient accuracy for any international, national, regional, state or local energy efficiency program
EM&V is not a barrier for efficiency, it is an opportunity
Kentucky PSC 9/11/09 Schiller Consulting, Inc.