1. Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May 7, 2014 May 2014 1

2. 2 Presentation Outline  Objectives  Methodology  Data  Performance measures  Aggregator program (CBP and AMP) results  Demand Bidding Program (DBP) results

3. May 2014 3 Objective: Assess Performance of Alternative Baseline Types  For each Utility and Notice type:  All customers, with BL adjustment as chosen  All customers, simulated with universal selection of the BL adjustment  Sum of individual BL vs. portfolio BL (constructed from aggregated customer loads), for AMP and CBP only  Examine unadjusted and day-of adjustments with 20%, 30%, 40%, 50% caps, and uncapped

4. May 2014 4 Analysis Details  For actual program event days  The “true” baseline is the estimated reference load from the ex post evaluation  For event-like non-event days  The “true” baseline is the observed load

5. May 2014 5 Performance Measures (1) Percentage Baseline Error  Percentage BL error for each customer/portfolio- event day is:  Percentage error = (L P d – L A d ) / L A d  L A d = actual, or “ true ” baseline load on day d  L P d = “predicted” baseline to be evaluated  Positive value = over-estimated baseline (implies over-stated program load impact)  Negative value = under-estimated baseline (implies under-stated program load impact)

6. May 2014 6 Performance Measures (2) Accuracy  Accuracy is measured as the median absolute percentage error (MAPE)  Calculate the absolute value of the percentage error for each customer/event-day  Calculate the median of values across customer/event- days (mean can be misleading due to extreme values)  Higher values correspond to larger baseline errors

7. May 2014 7 Performance Measures (3) Bias  Bias is measured by the median percentage error, without taking the absolute value  Positive values indicate upward bias (i.e., the program baseline tends to over-state the “true” baseline)  Negative values indicate downward bias (i.e., the program baseline tends to under-state the “true” baseline)

8. Nominated Customers by Choice of BL Adjustment – CBP and AMP May 2014 8

9. 9 Accuracy (Median Abs. % Error) PG&E CBP-DO

10. May 2014 10 Bias (Median % Error) PG&E CBP-DO

11. May 2014 11 Percentiles of % Errors – PG&E CBP-DO Actual Events, by Adjustment Cap

12. May 2014 12 Percentiles of % Errors – PG&E CBP-DO Simulated Events, by Adjustment Cap

13. Summary: Accuracy & Bias (Aggregated Indiv.; Universal Adj.; 40% cap) May 2014 13

14. Summary: Percentiles of % Errors (Aggregated Indiv.; Universal Adj.; 40% cap) May 2014 14

15. May 2014 15 Summary of Findings  Accuracy and bias measures vary by utility, program and notice type  Suggests that factors other than baseline type and adjustment caps may be most important, such as types of customers (e.g., highly variable load) and event-day characteristics (e.g., event on isolated hot day)  Day-of adjustment often improves accuracy and reduces bias, but level of cap is less important  Largest errors typically occur for Unadjusted BL and Unlimited cap  BL with small median error (e.g., 1%) can have >10% errors in 20 percent of cases

16. May 2014 16 DBP Results: PG&E Distribution of % Errors

17. May 2014 17 DBP Results: SCE Distribution of % Errors

18. Summary  Day-of adjustments tend to improve baseline accuracy and reduce bias  The analysis provides support for making the day-of adjustment the default option  The effectiveness of the day-of adjustment is not very sensitive to the level of the cap May 2014 18

19. May 2014 19 Questions?  Contact – Steve Braithwait or Dan Hansen, Christensen Associates Energy Consulting Madison, Wisconsin  Steve@CAEnergy.com Steve@CAEnergy.com  Danh@CAEnergy.com Danh@CAEnergy.com  608-231-2266

20. Appendix  SCE – CBP DO  SDG&E – CBP DO  PG&E – AMP DO  SCE – AMP DO May 2014 20

21. May 2014 21 Accuracy (Median Abs. % Error) SCE CBP-DO

22. May 2014 22 Bias (Median % Error) SCE CBP-DO

23. May 2014 23 Percentiles of % Errors – SCE CBP-DO Actual Events, by Adjustment Cap

24. May 2014 24 Percentiles of % Errors – SCE CBP-DO Simulated Events, by Adjustment Cap

25. May 2014 25 Accuracy (Median Abs. % Error) SDG&E CBP-DO

26. Accuracy – Med. Abs. Err. (MW) SDG&E CBP DO May 2014 26

27. May 2014 27 Bias (Median % Error) SDG&E CBP-DO

28. May 2014 28 Accuracy (Median Abs. % Error) PG&E AMP-DO

29. May 2014 29 Bias (Median % Error) PG&E AMP-DO

30. May 2014 30 Accuracy (Median Abs. % Error) SCE AMP-DO

31. May 2014 31 Bias (Median % Error) SCE AMP-DO