Discussion of PJM Forecasting Model Tim McClive OPSI Annual Meeting October 12, 2015

Navigant Overview » Navigant’s core business areas › Management Consulting, Economics, Financial Advisory, Disputes & Investigations › Publicly traded since 1996 (NYSE: NCI), 35 offices in N.A., Europe, Asia » Navigant’s global energy practice › Clients: 50 largest electric and gas utilities, 20 largest independent power generators, 20 largest gas distribution and pipeline companies, Federal/State governments, new entrants, investors › Personnel: 450+ consultants, average 15 years experience, 60% with advanced degree, 51% with engineering degree Page 2

Overview of Navigant’s Engagement with P3 » Provide an independent review of › PJM’s proposed changes of its load forecasting models › the structure, data, and estimation techniques of the models » Work with PJM LAS to investigate and recommend steps to improve accuracy and stability of the forecasts » Preliminary findings › Near term – improvements to a key energy efficiency variable › Longer term – potential theoretical and empirical modifications to the models Page 3

Econometrics – not as hard as rocket science, but close » “Econometrics is a special type of economic analysis in which the general theoretical approach is combined with empirical measurement of economic phenomena.” Leontief, 1948 » Econometrics has two elements, and both are important: › Economics – e.g., demand is higher with a stronger economy, lower prices, hotter summers, or colder winters (or lower when the opposite happens), and demand is lower when more efficient technology is used to meet the same service needs › Mathematics – the estimated models for demand should meet specified statistical conditions with an objective of producing unbiased estimates with minimum variance and error Page 4

Issue: SEER Forecasts Underestimate Peak Demand » PJM used available SEER data, and Navigant offered a way to augment that with publicly available data to improve accuracy » Cooling demand forecast – conditioned by projected efficiency improvements for different equipment classes. › For residential AC and heat pumps, PJM uses efficiency indexes based on seasonal energy efficiency ratio (SEER) › But SEER is calculated using full-load and part-load test conditions, while the energy efficiency ratio (EER) measures equipment running at full load › Correlations show that large improvements in SEER correlate with only moderate improvements in EER » Forecasts based on SEER may overestimate efficiency gains over time, and underestimate peak electrical demand Page 5

Illustration of SEER-EER Relationship » There is no direct relationship between SEER and EER » AC, HP manufacturers can increase SEER in ways that do not increase EER (advanced motors, compressors, controls) Page 6 Points represent SEER and EER ratings of individual AC systems in the AHRI database of certified products. Note metrics’ ranges: SEER: 13.0 to 26.0, 100% above lowest EER: 9.0 to 16.5, 83% above lowest

The Effect on PJM’s Total Cooling Index » Graphical comparison of load-weighted average Total Cooling Index (SEER-based vs. EER-based). Page 7

Sensitivity of Forecast to Projected Index Values » PJM’s May 2015 forecast update demonstrated that the forecast is highly sensitive to the efficiency index. › Efficiency growth that is one % point faster than projected results in peak demand forecast falling by 5,000 – 7,000 MW by 2020 » Navigant tested the sensitivity of the forecast to a change from a SEER-based to an EER-based projected index and calculated a material impact on the forecast. › Efficiency-related parameters in the forecast equations will likely change when the historical SEER-based index is replaced › Nevertheless, the transition from SEER to EER-based index would still materially affect the peak demand forecast. Page 8

On Forecast Accuracy Page 9 » From Nov ‘14 LAS presentation » The over-forecast in prompted the current reviews » Forecasts are not consistently high nor low over the long-term » PJM’s work in 2015 is a good start to resolve issues

Future Work and Recommendations » PJM Staff and LAS members work continuously and proactively to monitor forecast model performance and improve the accuracy of the models » Navigant’s review has identified areas for consideration and future study and will raise these with PJM and the LAS, e.g. › Simplify interactive structure of the explanatory variables to separate the discernible impacts of economic, technology and weather variables › Consider a two-part process – use current models to control for the short-term variances (“normalization”) and develop new model for the long term effects of economic and technology adoption variables › Various “in the weeds” ideas around technical methods to address data pooling, multicollinearity, and missing-variables issues Page 10

Navigant Contacts Tim McClive, Director, Washington DC (202) Ken Seiden, Director, Boulder, CO (303) Peter Steele-Mosey, Managing Consultant, Toronto, ON (416) J. Decker Ringo, Managing Consultant, Burlington MA (781) Page 11