Local Government Partnerships Impact Evaluation Research Plan Itron Study Manager: John Cavalli CPUC Study Manager: Jeremy Battis July 20, 2016

2 Presentation Outline Introduction – Goals and Research Questions – LGP Programs to be Studied Data Sources and Sample Design – Key Existing Data Sources – NTGR Telephone Survey Sample Design Evaluation Approach – Net-to-Gross Analysis – Ex Post Savings and Cost Effectiveness Assessment Timeline and Next Steps

INTRODUCTION

4 Study Goals There are two overarching goals of this impact study of Local Government Partnerships (LGPs): Ex Post Net-to-Gross (NTG) Analysis – Estimate net-to-gross ratios (NTGRs) for key LGP programs and for groups of similar/smaller programs. – Investigate reasons for free ridership. Ex Post Savings and Cost Effectiveness Assessment – Utilize the NTGRs from this study and existing ex post gross savings values from recent CPUC managed impact evaluations to update net ex post savings values. – Develop estimates of key cost-effectiveness metrics for each of the LGP resource programs.

5 Key Research Questions to be Addressed NTG Related Research Questions: What are the NTGRs over the timeframe: – Overall by program administrator (PA) for PG&E and SCE’s LGP resource programs? – Overall for large individual LGP resource programs or groups of similar/smaller programs? Which LGP programs or groups of programs have the lowest rates of free ridership? Early Replacement Assessment: – What percent of installed equipment was influenced by the program to perform an early replacement as opposed to a replacement on burnout? – Which key LGP programs or group of programs were most successful in influencing early replacement installations? Note that the NTGR is set equal to one minus free ridership. The spillover component of net is being evaluated under the cross cutting Nonresidential Spillover Study.

6 Research Questions Continued Ex Post Savings and Cost Effectiveness Related Questions: What are the ex post savings for each PG&E and SCE LGP resource program over the timeframe? – First year and lifecycle; gross and net; kw, kWh and therms. What are the net and gross realization rates (NRRs and GRRs) for each LGP resource program? – Ratio of ex ante to ex post savings. What are the Total Resource Cost (TRC) and Program Administrator Cost (PAC) values for each LGP resource program? – Based on the ex post savings values. How do the ex post TRC/PAC values compare to those reported (i.e., ex ante) for each LGP resource program? Which LGP resource programs are most cost-effective with respect to the TRC and PAC metrics? Examine other program effectiveness metrics developed by the Comprehensiveness Analysis Study.

7 Programs Evaluated Only LGP Resource Programs Claiming Ex Ante Savings are Evaluated Only PG&E and SCE have LGP resource programs that install incented measures and claim ex ante savings. – SDG&E and SCG track savings and process rebates and incentives from the LGPs through their nonresidential incentive programs. This was documented in their EE Applications (A and A ). – This does not allow for the direct evaluation of SDG&E’s and SCG’s LGP program portfolio from a cost-effectiveness standpoint. – Many of SDG&E and SCG’s nonresidential incentive programs expect to be evaluated under a similar study: The Deemed, Direct Install and Third Party Impact Evaluation which has not yet been fully planned. The draft report for this study is targeted for July 2017, and is expected to start Fall This evaluation will not attempt to quantify the effects of non-resource activities that may influence customers to adopt energy efficient measures or practices. The Local Government Partnerships Value and Effectiveness Study included non-resource programs in its evaluation. ED evaluators attempted to assess and attribute the contribution and value of SDG&E’s and SCG’s programs from a societal versus savings perspective.

8 PG&E’s LGP Programs Seventeen PG&E LGP resource programs will be evaluated:

9 SCE’s LGP Programs Twenty-three SCE LGP resource programs will be evaluated:

10 SCE’s LGP Programs Continued

DATA SOURCES AND SAMPLE DESIGN

12 Key Data Sources Existing Data Sources – Program Tracking Data for all LGP Resource Programs Serves as the source of the ex ante claimed savings NTG sample will be drawn from these data – ESPI Impact Evaluation Results Ex post gross savings for LGP participants Site-specific NTGR results for LGP participants Evaluations utilized from 2013, 2014 and 2015 studies: o Industrial, Agricultural and Large Commercial (IALC) ESPI Impact Evaluations o Custom Lighting ESPI impact evaluations o Nonresidential Deemed ESPI Impact Evaluations

13 Key Data Sources Continued – Comprehensiveness Analysis Study Results Various program performance and effectiveness metrics are being developed by the Comprehensiveness Analysis study. Metrics will be updated for LGP programs based on the results of this study. – CIS Billing Data CIS billing data will be merged to participant tracking records where possible to support the estimate of key performance metrics. New Data Sources – NTG Telephone Surveys This study will implement a telephone survey of nonresidential LGP resource program participants to support the NTG analysis.

14 NTGR Phone Survey Sample Design Sample Design for the telephone survey is based on Net Program Groups (NPGs) Ten PG&E Segments: – Programs with similar implementers were grouped into five segments: Ecology Action, RHA Staples and TEAA Note, one program is jointly implemented by RHA and Staples – Remaining 5 programs will be estimated separately Seven SCE Segments: – The three County EEP programs were combined into one segment. – Five segments representing the largest remaining programs in terms of savings. – All other programs are combined into one segment.

15 Unique Net Program Groups by IOU

16 Survey Completes by NPG A sample size of 30 telephone survey completes per segment is targeted with a few exceptions: – The Sierra Nevada program will likely be reclassified into the Staples or RHA NPG segments. Only 20 telephone surveys will be conducted, and reclassified. – Marin County and Redwood Coast programs contribute a small amount of savings. Only 15 telephone surveys will be conducted for each program. – Three SCE programs (Community ELP, Orange County ELP, and San Gabriel Valley ELP) have a relatively small number of participants for which 30 completes may not achievable. In these cases, a census will be attempted. Target sample size of 30 includes combining both existing NTG surveys and new data collection.

17 Targeted Relative Precision The NTGRs should be estimated at a relative precision of 10% or better, measured at the 90% confidence level (90/10). – Based on a coefficient of variation (COV) of 0.3, based on the existing telephone survey responses. – The smaller sample sizes of 15 should still provide a relative precision in the 90/15-90/20 range.

18 Sample Design by Net Program Group Telephone survey data will be collected for the following net program groups:

EVALUATION APPROACH

20 Evaluation Approach Overview Evaluation approach includes: Net-to-Gross Analysis – The analysis will develop ex-post net-to-gross ratios (NTGRs) for a select number of LGP programs. Ex Post Savings and Cost Effectiveness Analysis – This analysis will develop ex post savings values and key cost- effectiveness metrics for each of the LGP programs.

21 Net-to-Gross Analysis Objective – Develop ex-post NTGRs for a select number of LGP programs and program groups, as well as by PA, for the timeframe Utilize approximately 30 phone survey responses for each NPG segment to estimate NTGRs. Utilize the existing survey battery and analysis approach for estimating NTGRs from the nonresidential deemed ESPI impact evaluation studies. NTGR results will be compared across NPG to identify those with the lowest rates of free ridership. LGP NTGR results will be compared to other sectors at the PA level. – Such as Deemed, Direct Install, Third Party, and Custom programs by utilizing results from the ESPI impact evaluations.

22 Net-to-Gross Ratio Algorithm Utilizes the approach implemented for the Nonresidential ESPI Impact Evaluations Three separate program attribution indices (PAI) are developed and averaged together to create an estimate of the NTGR: – PAI–1 - relative influence of the most influential program and non-program factors – PAI–2 - the overall influence of various program related factors – PAI–3 - likelihood customer would have installed the same measure in the absence of the program. An attempt to identify the drivers of free ridership will be conducted through an examination of each PAI score, and the components of each score. – For example, what are the most influential program and non-program factors in the customer’s decision to install the rebated equipment.

23 Early Replacement Analysis Objective: Determine which NPGs were most successful in influencing early replacement installations. Utilize the existing survey battery and analysis approach for estimating early replacement from the nonresidential ESPI impact evaluation studies. – Examines age and condition of replaced equipment and likelihood of installing at the same time in the absence of the program. Compare early replacement rates across NPG. – Analysis limited to measures where early replacement is applicable. – May not be able to conduct analysis for all NPGs due to smaller samples. Identify reasons why early replacement did not occur. LGP early replacement rates will be compared to other sectors at the PA level. – Such as Deemed, Direct Install, Third Party, and Custom programs by utilizing results from the ESPI impact evaluations.

24 Ex Post Savings and Cost Effectiveness Analysis Objective – Develop ex post savings values and key cost- effectiveness metrics for each of the LGP programs Ex Post Gross - Develop ex post gross first year and lifecycle savings values based on existing Nonresidential ESPI Impact Evaluations. Ex Post Net - Utilize NTGRs from this study to develop ex post net first year and lifecycle savings values for the timeframe. GRRs and NRRs - Comparisons between the ex ante and ex post savings values can be used to develop net and gross realization rates. – The ratio between ex post and ex ante values. TRC and PAC - Ex post savings values will be used to develop key cost- effectiveness metrics for each LGP resource program. – Ex post TRC and PAC values will be compared to both planned and ex ante estimates to determine how well each program performed relative to expectations.

25 Ex Post Savings and Cost Effectiveness Analysis Estimate other program effectiveness metrics developed by the Comprehensiveness Analysis Study using this study’s ex post results: – Depth of Retrofit metrics: Number of end uses and measure classes installed Savings expressed as a percentage of whole building consumption – Ex post savings per program dollar spent – Integrated metric combining depth of retrofit and cost-effectiveness metrics Identify the most successful LGP programs with respect to ex post TRC and PAC values and other program effectiveness metrics. Compare LGP TRC and PAC values and other program effectiveness metrics to other sectors at the PA level. – Such as Deemed, Direct Install, Third Party, and Custom programs by utilizing results from the ESPI impact evaluations.

TIMELINE AND NEXT STEPS

27 Timeline

28 Next Steps Research Plan comments are being accepted through July 29 th. Comments will be reviewed and responses will be posted in August. The Research Plan will be revised, finalized, and posted after comments are addressed by August 19 th. Data collection through participant telephone surveys will begin in September.

THANK YOU For questions, please contact: Itron Project Manager: John Cavalli, CPUC Project Manager: Jeremy Battis,

30 Mapping of PG&E LGPs to NPGs Both Staples and RHA implement the Sierra Nevada program, so for the time being this will remain a separate segment

31 Mapping of SCE LGPs to NPGs