Using Industrial Operating Models for Cost Benefit Analysis 2007 MacArthur Foundation Benefit-Cost Analysis Conference 6/9/2019

Organizations Like NOAA Must Evaluate Their Programs First, there must be a credible rationale for funding a program from public revenue. If a rationale exists, the program must also have benefits that exceed its costs. Cost-Benefit Analysis is used to determine whether programs pass this test. Using Industrial Operating Models for Cost Benefit Analysis

In NOAA, Weather Forecasting is an Important Program Benefits of weather forecasts can be estimated in three general ways: Stated Preference - forecast users are asked to self-assess the benefits they receive Data Analysis - benefits are inferred from outcome data, with and without forecasts Economic Modeling - benefits are estimated by simulating decisions and outcomes Using Industrial Operating Models for Cost Benefit Analysis

My Focus Here is on Economic Modeling To use this method, it is necessary to model decisions and outcomes that are often complicated and esoteric. Typically, an organization like NOAA lacks the resources and industry knowledge to do such modeling from scratch. Thus it becomes critical to find and adapt existing industrial operating models. Using Industrial Operating Models for Cost Benefit Analysis

Example: Benefits of Forecasts Used by Electricity Generators Electricity load (use) is weather dependent, especially in regions where air-conditioning is a large component of total load. Good temperature forecasts allow the most cost-effective set of plants to be used to serve the load. The benefits of temperature forecasts are the generation cost savings they create. Using Industrial Operating Models for Cost Benefit Analysis

The Decision-Making here is Really Complicated A temperature forecast for the next 24 hours must be translated into an electricity load forecast for each specific electricity system. Then generating units must be selected to minimize cost, given the load forecast. Finally, real-time adjustments must be made to compensate for inevitable errors between forecasted and actual loads. Using Industrial Operating Models for Cost Benefit Analysis

To Estimate Benefits, We Used Existing Operating Models A study by Hobbs et. al. [1] modeled the cost savings from better load forecasts. Hobbs used a model from Baldick [2] to represent choice among generating units, given a load forecast. And Hobbs developed an electricity dispatch model to simulate real-time adjustments to load forecast errors. Using Industrial Operating Models for Cost Benefit Analysis

We Still Needed a Link between Temperature and Load Forecasts In our study [3], we used an operating model developed by one of our co-authors -- an electricity load forecaster known as Neural Electric Load Forecaster (NELF). NELF uses a variety of information, including the next day temperature forecast, to estimate next day electricity load. This provided the needed link between temperature forecasts and load forecasts. Using Industrial Operating Models for Cost Benefit Analysis

We Picked Sites Representing Each of Three US Regions Since we needed both weather and load data for specific utility service areas, there were limitations on the number and location of sites we could consider. We choose two sites each representing respectively the northern US, the southern US, and the western US. Using Industrial Operating Models for Cost Benefit Analysis

At Each Site, We Used NELF to Model Four Weather Forecasts Naïve -- tomorrow’s weather will be the same as today’s. MAV -- Model Output Statistic Aviation, an NWS guidance product. NWS Forecast -- represents 7 to 11 percent improvement relative to MAV Perfect -- tomorrow’s actual weather is used as the forecast for tomorrow. Using Industrial Operating Models for Cost Benefit Analysis

Load Forecast Errors Decline as Weather Forecasts Improve Using Industrial Operating Models for Cost Benefit Analysis

We Used Hobbs’ Study to Value Load Forecast Improvements Hobbs et. al. used actual and forecasted load data from two electric utility systems, one in the northeast and one in the south. Hobbs considered two alternative configurations of generating plants to serve each type of load. This produced four cases spanning a range of real-world possibilities. Using Industrial Operating Models for Cost Benefit Analysis

For Each Case, Hobbs Estimated Benefits of Better Load Forecasts Hobbs scaled load forecasting errors up or down to simulate varying forecast quality. Generation decisions were simulated using Baldick’s plant commitment model coupled with an Hobbs’ electricity dispatch model. Generation costs were calculated and used to estimate percentage savings from better load forecasts. Using Industrial Operating Models for Cost Benefit Analysis

We Averaged Hobbs’ Results Across Generating System Types We averaged Hobb’s results across his alternative configurations of generating systems, for each of his two regions. This produced a southern system cost savings that we applied to our Southern region, and a northeastern system cost savings that we applied to our North and West regions. Using Industrial Operating Models for Cost Benefit Analysis

We Next Needed US Electricity Generation Costs by Region Since Hobbs’ costs savings were expressed as percentage cost savings, we needed to estimate regional generation costs. We allocated total US utility generation to our three regions (South, North, and West). For each region, we estimated costs using $20 per million watt-hours as a typical cost. Using Industrial Operating Models for Cost Benefit Analysis

Electricity Benefits of Weather Forecasts Are Mainly in South Using Industrial Operating Models for Cost Benefit Analysis

There are Reasons Why Benefits Are Mainly in South Weather forecasts reduce load forecast error much more in the South (see earlier chart). Hobbs results show much higher costs of load forecast errors for his southern system. Both these explanations in turn probably reflect the fact that space cooling demand is weather sensitive and very important in the South. Using Industrial Operating Models for Cost Benefit Analysis

Benefits of Improved Forecasts Can Be Estimated Two Ways The MAV and NWS forecasts differ by small amounts, and so can be directly used to estimate benefits of small improvements from the current forecast quality. We can also fit a curve through the cases we estimated (Naïve, MAV, NWS, Perfect) and use this curve to interpolate benefits of a variety of alternative forecast qualities. Using Industrial Operating Models for Cost Benefit Analysis

A One-Percent Improvement Has Similar Benefits Both Ways Estimating the benefit of a one-percent improvement in forecast quality from the NWS and MAV cases implies a value of $1.3 million per year. Estimating this benefit from a fitted curve implies a value of $1.1 million per year. Using Industrial Operating Models for Cost Benefit Analysis

We Can Also Estimate the Value of a One Degree Improvement Here we used the fitted curve, since a one degree improvement represents a relatively large change in forecast quality (about 1/3 of existing error). Using the fitted curve, a one-degree improvement in forecast quality has a value of about $35 million per year. Using Industrial Operating Models for Cost Benefit Analysis

These Benefits Can Be Expressed in Present Value Compare a one-time investment to improve forecasts by one-degree with its cost and the present value of it benefits. At 5% discount rate, the present value of a one-degree improvement in weather forecasts is about $700 million; one percent is $22-26 million. Using Industrial Operating Models for Cost Benefit Analysis

Summary Overall, official NWS forecasts appear to produce total benefits for electricity generators of about $156 million per year, mostly in the south. A perfect forecast would add $70 million. These are benefits from plant scheduling, and do not include other possible benefits, e.g. plant maintenance decision-making. Using Industrial Operating Models for Cost Benefit Analysis

Summary (Concluded) Benefits of a small improvement in weather forecast quality were estimated to be $1.1 to 1.3 million per year, per percentage point of error reduced. Benefits of a larger improvement (one degree, or around 1/3 of the current error) were estimated at about $35 million per year or about $700 million in present value. Using Industrial Operating Models for Cost Benefit Analysis

References [1] Hobbs, Benjamin F., et. al. Analysis of the Value for Unit Commitment Decisions of Improved Load Forecasts. IEEE Transactions on Power Systems, Vol. 14, No. 4. 1999, pp. 1342-1348. [2] Baldick, R., A Generalized Unit Commitment Model, IEEE Transactions on Power Systems, Vol. 10, No. 1. 1995, pp. 465-475. [3] Teisberg, T.J., R.F. Weiher, and A. Khotanzad, The Economic Value of Temperature Forecasts In Electricity Generation, Bulletin of the American Meteorological Society, Vol. 86, No. 12, 2005, pp. 1765-1771. Using Industrial Operating Models for Cost Benefit Analysis