Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May 7, 2014 May

2 Presentation Outline  Objectives  Methodology  Data  Performance measures  Aggregator program (CBP and AMP) results  Demand Bidding Program (DBP) results

May Objective: Assess Performance of Alternative Baseline Types  For each Utility and Notice type:  All customers, with BL adjustment as chosen  All customers, simulated with universal selection of the BL adjustment  Sum of individual BL vs. portfolio BL (constructed from aggregated customer loads), for AMP and CBP only  Examine unadjusted and day-of adjustments with 20%, 30%, 40%, 50% caps, and uncapped

May Analysis Details  For actual program event days  The “true” baseline is the estimated reference load from the ex post evaluation  For event-like non-event days  The “true” baseline is the observed load

May Performance Measures (1) Percentage Baseline Error  Percentage BL error for each customer/portfolio- event day is:  Percentage error = (L P d – L A d ) / L A d  L A d = actual, or “ true ” baseline load on day d  L P d = “predicted” baseline to be evaluated  Positive value = over-estimated baseline (implies over-stated program load impact)  Negative value = under-estimated baseline (implies under-stated program load impact)

May Performance Measures (2) Accuracy  Accuracy is measured as the median absolute percentage error (MAPE)  Calculate the absolute value of the percentage error for each customer/event-day  Calculate the median of values across customer/event- days (mean can be misleading due to extreme values)  Higher values correspond to larger baseline errors

May Performance Measures (3) Bias  Bias is measured by the median percentage error, without taking the absolute value  Positive values indicate upward bias (i.e., the program baseline tends to over-state the “true” baseline)  Negative values indicate downward bias (i.e., the program baseline tends to under-state the “true” baseline)

Nominated Customers by Choice of BL Adjustment – CBP and AMP May

9 Accuracy (Median Abs. % Error) PG&E CBP-DO

May Bias (Median % Error) PG&E CBP-DO

May Percentiles of % Errors – PG&E CBP-DO Actual Events, by Adjustment Cap

May Percentiles of % Errors – PG&E CBP-DO Simulated Events, by Adjustment Cap

Summary: Accuracy & Bias (Aggregated Indiv.; Universal Adj.; 40% cap) May

Summary: Percentiles of % Errors (Aggregated Indiv.; Universal Adj.; 40% cap) May

May Summary of Findings  Accuracy and bias measures vary by utility, program and notice type  Suggests that factors other than baseline type and adjustment caps may be most important, such as types of customers (e.g., highly variable load) and event-day characteristics (e.g., event on isolated hot day)  Day-of adjustment often improves accuracy and reduces bias, but level of cap is less important  Largest errors typically occur for Unadjusted BL and Unlimited cap  BL with small median error (e.g., 1%) can have >10% errors in 20 percent of cases

May DBP Results: PG&E Distribution of % Errors

May DBP Results: SCE Distribution of % Errors

Summary  Day-of adjustments tend to improve baseline accuracy and reduce bias  The analysis provides support for making the day-of adjustment the default option  The effectiveness of the day-of adjustment is not very sensitive to the level of the cap May

May Questions?  Contact – Steve Braithwait or Dan Hansen, Christensen Associates Energy Consulting Madison, Wisconsin   

Appendix  SCE – CBP DO  SDG&E – CBP DO  PG&E – AMP DO  SCE – AMP DO May

May Accuracy (Median Abs. % Error) SCE CBP-DO

May Bias (Median % Error) SCE CBP-DO

May Percentiles of % Errors – SCE CBP-DO Actual Events, by Adjustment Cap

May Percentiles of % Errors – SCE CBP-DO Simulated Events, by Adjustment Cap

May Accuracy (Median Abs. % Error) SDG&E CBP-DO

Accuracy – Med. Abs. Err. (MW) SDG&E CBP DO May

May Bias (Median % Error) SDG&E CBP-DO

May Accuracy (Median Abs. % Error) PG&E AMP-DO

May Bias (Median % Error) PG&E AMP-DO

May Accuracy (Median Abs. % Error) SCE AMP-DO

May Bias (Median % Error) SCE AMP-DO