EM&V Planning and EM&V Issues Kentucky PSC 9/11/09 Schiller Consulting, Inc.

Planning and the Program Implementation Cycle Kentucky PSC 9/11/09 Schiller Consulting, Inc.

Evaluation in Context: The Cycle of Program Planning and Implementation Adapted by CEE from Pierre Landry, SCE, and Demand- Side Management, Vol. 4: Commercial Markets and Programs, EPRI, 1987. 7. A visual to help explain where evaluation fits in within the program AND resource planning process (refer them to a handout of this which we’ll send in advance) Good evaluation planning begins very early, once program objectives are defined. The program objectives should be determined within the framework of the policy objectives. Many different types of issues should be taken into account and balanced, such as How important is peak demand reduction vs. overall energy usage reduction? How important is equity? (EE benefits for disadvantaged customers or in proportion to contribution to EE funding) These need to be considered in planning evaluation (box 2) to ensure that evaluation results will provide appropriate indicators to use in measuring progress towards these broad goals. Then various program alternatives should be looked at to design a portfolio of programs (like a portfolio of stocks) and design individual programs within the portfolio. Successful portfolio design requires knowing what the EE potential is for various end uses and approaches within the state or service territory. Individual program design & selection involves assessing measures to cover (end uses), approaches to use to persuade customers to adopt the measures (marketing techniques), and selecting among them (which programs are most cost-effective at meeting the policy goals?). Eval planning should begin at this stage, taking into account program objectives in box #1. Getting evaluators involved at this stage, developing logic modeling to identifying early indicators of progress towards program goals, makes eval more efficient & effective. Program implementation (first they should be tested, then pilot programs run, and then go full-scale after tweaking). Ideally, evaluation data collection should begin when programs begin. Reduces data collection problems and evaluation challenges later. Once program is implemented process evaluation can take place, and savings estimates for impact evaluation can begin to be calculated with the data that should have been collected from beginning of program. The results of evaluation should be used to improve the design & execution of current programs. They should also be use to inform the selection and design of a new crop of programs, including determination of program objectives, and to inform the next round of resource planning and development of policy objectives. Ideally, what results is a process of continuous improvement of program information and programs to inform resource planning and meet policy objectives. Kentucky PSC 9/11/09 Schiller Consulting, Inc.

Determining Gross and Net Energy Savings Kentucky PSC 9/11/09 Schiller Consulting, Inc.

Planning Issues Defining evaluation goals and scale, and which benefits to evaluate Setting time frame for evaluation and reporting expectations Establishing budget vis-à-vis expectations for quality of reported results Defining baseline, baseline adjustments, and data collection requirements Selecting impact evaluation approaches for gross and net savings calculations and avoided emissions calculations Selecting who (or which type of organization) will conduct the evaluations Identifying and integrating stakeholders Successful evaluations harmonize the costs of evaluation with the value of the information received—that is, they appropriately balance risk management, uncertainty, and cost considerations. Kentucky PSC 9/11/09 Schiller Consulting, Inc.

How Good is Good Enough? - Incremental certainty Value of information value of improved certainty cost of improved EM&V Cost of incremental EM&V Balance point of diminishing returns Cost < cost of improved EM&V Kentucky PSC 9/11/09 Schiller Consulting, Inc.

Precision and Bias Kentucky PSC 9/11/09 Schiller Consulting, Inc.

Technical Reference Manuals and Evaluation Protocols – EM&V Resources In the simplest form ‘TRMs’ are databases of deemed savings values with applicability conditions Examples of database contents are: kWh, kW, therm savings values Measure effective lifetimes Net to gross ratios Measure cost information Local/regional utility cost information for TRC calculations Work papers that document basis for values Documents with varying level of detail that define how the evaluations (and M&V) are to be conducted with a particular jurisdiction. Contents might include: Which approaches to use Sample size requirements Evaluation timing Reporting requirements Kentucky PSC 9/11/09 Schiller Consulting, Inc.

The Documentation (EM&V) Issue and Solution Addressing the challenge of quantifying a counterfactual situation whose characteristics can never be proven ex-post, i.e. how much energy would have been consumed or emissions produced in the absence of the efficiency activity Solution Applying the experience and protocols and TRMs already in place - EM&V has and is being done for billions of dollars worth of programs Conclusion EM&V can be defined and implemented to a level of sufficient accuracy for any international, national, regional, state or local energy efficiency program EM&V is not a barrier for efficiency, it is an opportunity Kentucky PSC 9/11/09 Schiller Consulting, Inc.