1. Comparison of Pooled and Household-Level Usage Impact Analysis Jackie Berger Ferit Ucar IEPEC Conference – August 14, 2013

2. Presentation Outline Motivation Billing Analysis Usage Impact Models Model Results Summary and Next Steps 2

3. MOTIVATION 3

4. Usage Impacts Were expected energy savings results obtained? Are the treatments cost-effective? Should measure selection procedures be revised? Should installation procedures be reviewed? Should contractors be re-trained? 4

5. Analysis Method Goal: develop most accurate estimate of program savings. Weigh costs and benefits of various approaches to measurement. Consider possible causes of mis-measurement or bias. 5

6. What Are You Measuring? 6 Approach MeasuresIssues Engineering Estimate Expected usage change based on measures alone Assumptions Installation quality Other usage changes Usage 2 - Usage 1 Actual change in usage Weather Other factors Weather Norm Usage 2 - Usage 1 Change in usage if both periods had average weather Other factors Weather Norm Usage 2 - Usage 1 w/Comp. Group Other factors held constant (prices, economy, market information, etc.) Best estimate of program impact

7. Analysis Approaches 7 Approach CostAccuracyAttrition Reasons for Exclusion Engineering Estimate $****None Engineering Estimate with Retrofit Data $$**** Retrofit Data Missing Billing Analysis $$$***** Usage Data Missing or Inadequate Metering$$$$*****Cost

8. Billing Analysis 8

9. Data Requirements 9 Core Data RequiredSupplemental Data Energy billing data Read date, real or estimated, usage, units Energy efficiency measures Measure-specific impacts Service delivery date Divides period into pre- and post- treatment Service delivery providers Provider-specific impacts Weather data Local weather station, daily temp for pre and post period and longer normalization period Housing unit characteristics Relation between housing /household characteristics and savings Household characteristics

10. Challenges 10 Concern Data Attrition Savings do not represent treated population Sample Size Low precision for savings estimates Cannot estimate for sub-groups Comparison Group Need to control for exogenous factors Not able to do random assignment Difficult to find comparable population Later program participants Earlier program participants Comparable households

11. Treatment and Comparison Group Example 11 Post-Treatment Period PreYr 2 – Quasi Post 2010 Post Yr 2 – Quasi Post SERVICE DELIVERY DATE 2011 2012 SERVICE DELIVERY DATE Pre-Treatment Period Post Yr 1 – Quasi Pre Pre Yr 1 – Quasi Pre Treatment Group Comparison Treated Year Before Comparison Treated Year After

12. Usage Impact Models 12

13. House-by-House 13 NAC = 365α + βHo(τ) H o (τ) = long term average heating degree days Fi = average daily consumption in time interval i H i (τ) = heating degree days to reference temperature τ in interval i i = random error term Regression Analysis on Each Individual Home: Fi = α + βHi(τ) + i

14. Pooled Analysis 14 F it = average daily usage during the pre- and post-treatment periods H it = average daily base 60 HDDs POSTt = a dummy variable that is 0 in the pre-period and 1 in the post-period ε it = estimation error term PRE USAGE α i = average daily baseload usage in pre-treatment period. β1 = average daily usage per HDD in the pre-treatment period. POST USAGE α i + β2= average daily baseload usage in the post-treatment period. β1 + β3= average daily usage per HDD in the post-treatment period. SAVINGS β2 = average daily baseload savings β3 = heating usage savings per HDD. F it = α i + β1* H it + β2*POST t + β3*POST t *H it + ε it

15. Advantages 15 House-by-HousePooled Detailed attrition analysis Utilizes all billing data Post-analysis of usage and savings Direct estimate of savings is furnished Analysis of high- and low-saving homes Regression models can be used to estimate savings by measure Exogenous factors can be included in the model Relationship between usage and housing/ household characteristics can be explored

16. Disadvantages 16 House-by-HousePooled Less robust when energy use response to degree days varies within household Inclusion of many parameters can make final results difficult to interpret Requires close to a full year of pre- and post-treatment data Alternative functional form may be required Substantial attrition can bias the analysis Limited ability to furnish information on savings distribution and conduct exploratory analysis

17. When to Use 17 House-by-HousePooled Availability of close to one year of pre-and post-treatment usage data for significant % of treatment and comparison Limited data availability and concern for attrition bias associated with excluding homes Data available on treatment, home, or households, that can be used to assess factors related to higher or lower savings Supplemental data not available Study sponsors not interested in supplemental analysis

18. Model Results 18

19. Program 1 Results Gas Heating Jobs 19 ModelObs. Pre- Use Post- Use Savings ccf% Not Normalized 1,1661,06099070 (±11)6.6% House-by-House 1,1661,05299161 (±10)5.8% Pooled Regression 1,1661,03096466 (±10)6.4% Pooled-Month Dummy 1,1661,0841,02064 (±10) 5.9% Pooled-all obs.1,439 1,03196665 (±9)6.3%

20. Program 2 Results Gas Heating Jobs 20 ModelObs. Pre- Use Post- Use Savings ccf% Not Normalized 1,2111,042831 210 (±12) 20.2% House-by-House 1,2111,025959 67 (±10) 6.5% Pooled Regression 1,211999936 63 (±9) 6.3% Pooled-Month Dummy 1,2111,044976 68 (±10) 6.5% Pooled-all obs. 1,6651,00293369 (±8)6.9%

21. Program 1 Results Electric Baseload Jobs 21 ModelObs. Pre- Use Post- Use Savings kWh% Not Normalized 4,05511,15310,792361 (±73)3.2% House-by-House 4,05511,37010,1471,223 (±78)10.8% Pooled Regression 4,05510,6249,735889 (±55) 8.4% Pooled-Month Dummy 4,05510,7989,957841 (±56) 7.8% Pooled-All Obs. 5,375 10,7289,893835 (±53) 7.8% Pooled-Month-All 5,375 11,19010,425765 (±55) 6.8%

22. Program 2 Results Electric Baseload Jobs 22 ModelObs. Pre- Use Post- Use Savings kWh% Not Normalized 2,44011,0229,7651,257 (±93)11.4% House-by-House 2,44010,75810,148610 (±99)5.7% Pooled Regression 2,44010,1399,501638 (±69)6.3% Pooled-Month Dummy 2,4409,7799,123656 (±82)6.7% Pooled-All Obs. 4,65410,2879,726 561 (±56) 5.5% Pooled-Month-All 4,6549,8539,277575 (±66)5.8%

23. Program 1 Results Electric Heating Jobs 23 ModelObs. Pre- Use Post- Use Savings kWh% Not Normalized 14417,84617,77967 (±541)0.4% House-by-House 14419,66218,5341,128 (±503)5.7% Pooled Regression 14417,94017,084857 (±559) 4.8% Pooled-Month Dummy 144 19,73818,826912 (±586) 4.6% Pooled-All Obs. 220 17,83016,992838 (±491) 4.7% Pooled-Month-All 220 20,84620,028818 (±515) 3.9 %

24. Program 2 Results Electric Heating Jobs 24 ModelObs. Pre- Use Post- Use Savings kWh% Not Normalized 13418,10314,2983,805 (±646)21.0% House-by-House 13419,40217,8991,503 (±665)7.7% Pooled Regression 13417,02015,5051,515 (±543)8.9% Pooled-Month Dummy 134 17,17715,6141,562 (±647) 9.1% Pooled-All Obs. 28216,88415,2631,621 (±391)9.6% Pooled-Month-All 282 18,19316,3741,819 (±457) 10.1%

25. Electric Baseload Pre-Treatment UsageObs. Pre- Use Post- Use Savings kWh% < 8,000 kWh7036,8626,5682944.3% 8,000 – 12,000 kWh1,9729,7208,8298919.2% > 12,000 kWh1,38016,02413,8522,17213.6% Program 1 Results Household Characteristics 25 OwnerObs. Pre- Use Post- Use Savings kWh% Owner2,26311,43310,1551,27811.2% Renter1,79211,29110,1371,15410.2%

26. Electric Baseload Supplemental HeatObs. Pre- Use Post- Use Savings kWh% Supplemental Heat2,05412,29310,8121,48112.0% No Supp Heat2,00110,4239,4649599.2% Program 1 Results Household Characteristics 26 Level of ServiceObs. Pre- Use Post- Use Savings kWh% Basic3,53311,29110,1611,13010.0% Major52211,90610,0511,85515.6% Major measures include refrigerators, air conditioner, and water heater replacements.

27. Electric Baseload MeasuresObs. Pre- Use Post- Use Savings kWh% Air Conditioner7811,3639,6261,73715.3% No Air Conditioner3,97711,37010,1571,21310.7% Refrigerator42011,2119,3761,38612.4% No Refrigerator3,63511,38810,2351,15310.1% AC/Refrigerator2711,4329,2322,20019.2% AC/ No Refrigerator5111,3279,8341,49313.2% No AC/ Refrigerator39311,1969,3861,81016.2% No AC/ No Refrigerator3,58411,38910,2411,14810.1% Program 1 Results Household Characteristics 27

28. Summary and Next Steps 28

29. Summary Overall savings results fairly consistent Differences between models rarely statistically significant Gas usage results were more consistent Electric baseload varied most 29

30. Conclusions Sources and potential biases caused by large data attrition should be explored. When additional analysis is desired for many subgroups and data attrition is low, house-by- house may be favored. When data attrition is high and only overall usage results are desired, the pooled regression may be preferred. 30

31. Next Steps Additional exploration of differences. Explore deletion of various types and numbers of observations from house by house. Compare results with different levels of attrition. Test different functional forms for the pooled model. 31