1. MAPE Publication Neil McAndrews For Bob Ryan of Deutsche Bank

2. What makes a forecast a forecast? 1. Forecasts are designed to estimate, in an unbiased manner, the future behavior of a directly observable event or variable. In power markets, ERCOT included, forecasts transfer market information from public to private market participants. 2. Forecasts must be unbiased estimates, otherwise they are not valid forecasts. Unbiased means that they are wrong as often as they are right. 3. The Expected Value of a Forecast’s Error is, by definition, zero. These error terms should be normally distributed. 4. A forecast’s accuracy – hence its reliability – is measured by the use of statistical tools such as Mean Absolute Percentage Error (MAPE), Absolute Percent Error (APE), and Percent Error (PE).

3. MAPE When assessing load forecasts, MAPE is the preferred measurement of performance. It tells the user how the forecast is performing over time. It is calculated by the equation: Average of all intervals where ”interval” is defined as  Absolute value of (Actual minus Forecast) divided by Actual times 100. A large or steadily increasing MAPE typically is the result of persistent misspecification in a model – i.e., overestimation and underestimation of actual loads.  Overestimation of actual load in electric markets leads to inefficient allocation of scarce reserve resources because the market is uncertain of actual load levels.  Underestimation leads to shortages and increases in use of reserves because the market is uncertain of actual load levels.

4. Transparent MAPE data benefits the market Currently ERCOT publishes monthly MAPE from time to time. This PRR makes it monthly. Forecasts of load are really the combination of two forecasts: the load model forecast, and the weather forecast (predominately temperature forecast). Publishing MAPE data for both model error and weather error will lead to corrections of particular problems. Market participants and the ISO can continually iterate on their forecasts to improve forecast accuracy (i.e., minimize forecast error). This is common practice in other regions, e.g., PJM. For instance, weather models are periodically updated and not all of the updates immediately improve performance in all geographic regions. Through the use of timely weather MAPE publication, problems with weather forecasts can be more quickly identified by both ERCOT and private participants reviewing the weather data.

5. Recent forecast performance Sept 1 to Oct 31. ERCOT Peak Forecast and Actual (from PUC data) 1. MAPE 4.44% 2. Average Bias is 1677 MWs.

6. Summary of Current Forecast The forecast is overestimating peak load for the months of September and October by an average of 1,677 MWs (3.6%). Actual error should be much closer to zero. (Overestimation does appear to be declining later in the period) The error is not normally distributed about actual peak load. 77% of time the forecast overestimates load. This should be approximately 50%. Suggest a review of this data for accuracy.

7. Value of Information and the Cost of AS Reserves due to Forecast Error We in ERCOT should be asking the question: “How does one percentage point improvement in MAPE effect the cost of reserves?” We suggest an estimate can be made by running the Ancillary Service selection process with actual load data and comparing it to the current forecast process. With enough samples an estimate of the reduction of reserve cost in AS market can be made. Then a calculation can be performed that estimates the cost of improving the forecast by one percent. A cost benefit will result from the two estimates.

8. Summary MAPE is a measurement tool that improves fundamental market information for market participants. Publishing load and weather MAPEs will help to lower forecast error by the natural collaboration between ERCOT and market participants. ERCOT has published summary monthly MAPE in the past (see handout: “Load Forecasting Methodology Forum Update”, July 6, 2006 TAC Meeting). ERCOT has not isolated the weather forecast error from the overall model. This PRR addition would isolate the causes of errors, and lead to improved forecast performance.