Comparison of CA Evaluation Protocols, CA Framework, IPMVP and CPUC Policy Manual* A preface to group discussion *In terms of how they define program gross energy and demand EM&V requirements Mary Sutter & Tim Caulfield Equipoise Consulting Inc. October 27, 2004

2 Discussion Overview Purpose of this presentation Overview of some documents used in California evaluation of energy efficiency programs  CPUC Energy Efficiency Policy Manual Version 2  The California Evaluation Framework  Protocols and Procedures for the Verification of Costs, Benefits, and Shareholder Earnings from Demand-Side Management Programs (The “Protocols”)  International Performance Measurement and Verification Protocol - Concepts and Options for Determining Water and Energy Savings (IPMVP) Comparisons Group Discussion

3 Purpose of this Presentation CALMAC is the appropriate venue for substantive discussion on evaluation Energy efficiency program evaluation in California in a state of flux New evaluation protocols are planned Through review of past / present documents, determine what are the most crucial questions to discuss at this point in time Narrow scope for now (Gross energy and demand impacts)

4 Energy Efficiency Policy Manual Purpose: Sets the policy rules in the development and evaluation of energy efficiency (EE) programs in California. Broad set of objectives that evaluation must meet. Specific components indicated with supporting evidence required if not including in evaluation.

5 Policy Manual The objectives below must have strong supporting arguments for omission from evaluation plan Policies for Programs with Energy Impacts  Evaluation must measure the level of energy and peak demand savings achieved by a program.  Measure Cost-effectiveness  Up-front market assessments and baseline analysis (especially for new programs)  Ongoing feedback, and corrective and constructive guidance regarding the implementation of the program  Measure indicators of the effectiveness of specific programs, including testing of the assumptions that underlie the program theory and approach  Assess overall levels of performance and success of programs  Inform decisions regarding compensation and final payments  Help assess whether there is a continuing need for the program

6 Policy Manual These components are required in the evaluation plan Baseline Information  Baseline data upon which to base energy savings measurement  Perform study if none available or prove why cannot do study Energy Efficiency Measure Information  Description of EE measures in program  Includes assumptions about important variables and unknowns M&V Approach  Reference appropriate IPMVP option  Describe deviation from IPMVP  Schedule for acquiring project-specific data Evaluation Approach  Questions to be answered through evaluation  Evaluation tasks / activities  Describe how evaluation will meet all policy objectives

7 The California Evaluation Framework Purpose: The California Evaluation Framework (Framework) provides a consistent, systemized, cyclic approach for planning and conducting evaluations of California’s energy efficiency and resource acquisition programs. The primary purpose of impact evaluation is to obtain the most accurate and unbiased estimate of energy and demand savings due to a program. Gross savings are calculated from program participants relative to their prior participation usage.

8 Framework – Gross Savings Billing Analysis path  Based on statistical principals  Multiple methods / regression models Engineering Analysis path  Based on basic rules of physics  Simple engineering models or building energy simulations  M&V incorporated into field data collection (IPMVP)

9 Framework – Gross Savings Issues Billing Analysis  Difficult / impossible for evaluation of 3 rd party programs.  Finding a non-participant group not effected by any EE program (market noise) Engineering Analysis  The most uncertain parameter may be the most expensive to obtain  Cannot collect pre-retrofit measurements in many cases

10 IPMVP Purpose: The IPMVP provides an overview of current best practice techniques available for verifying results of energy efficiency, water efficiency, and renewable energy projects. This document can help in the selection of the M&V approach that best matches:  i) project costs and savings magnitude  ii) technology-specific requirements  iii) risk allocation between buyer and seller (i.e., which party is responsible for installed equipment performance and which party is responsible for achieving long term energy savings).

11 IPMVP M&V – the process of determining savings using one of the four IPMVP options Option A  Engineering calculations with some parameters in algorithms stipulated and other parameters onsite measured Option B  Engineering calculations with all parameters in algorithm onsite measured Option C  Whole facility analysis with billing or metered data – from simple pre/post comparison to regression analysis on a single building over time Option D  Computer simulation calibrated with hourly or monthly utility billing data and/or end use metering

12 “Protocols” Purpose: Protocols and procedures to be used by the IOUs to document and verify costs and benefits of major DSM program activities for shareholder earnings and measurement agreements for resource planning purposes.

13 “Protocols” Gross Impacts  Provides various types of models/approaches acceptable and generally when they are applicable  Sample design requirements for approaches  Reporting requirements  Documentation requirements  Measurement schedule Net Impacts Retention Studies

14 “Protocols” Summary Prescriptive protocols – Specify what, when and how Contain ability to request a waiver from specified approaches and schedules; reviewed by CADMAC. Supplied relative surety that if protocols were followed results would be acceptable.

15 Comparisons EE Policy Manual CA Evaluation Framework “Protocols”IPMVP Specific to energy efficiency programs Yes No Protocols with known success parameters and avenue for disputes No YesNo Energy impact evaluation procedures NoYes Demand impact evaluation procedures No Program level sampling procedures NoYes No Net issues addressed NoYes No Specific reporting formats and procedures No YesNo Persistence impact evaluation procedures No YesNo Timing of evaluations discussed NoYes No

16 Discussion Points What are the relevant questions for our next set of protocols? Who are the ultimate users of the energy and demand impacts, what data is required and when is the information needed? How can the protocols be structured such that you can put resources where there is the most uncertainty? Do evaluations require precise demand impacts or can we use the Savings Goals GWh to MW conversion? What are the ramifications of 3 rd party programs being unable to use billing analysis that includes nonparticipants? Will evaluation budgets support yet-to-be-determined precision requirements for energy/demand impact evaluation as well as allowing for logic modeling and process evaluation? Others?