1. Using Industrial Operating Models for Cost Benefit Analysis
2007 MacArthur Foundation Benefit-Cost Analysis Conference
6/9/2019

2. Organizations Like NOAA Must Evaluate Their Programs
First, there must be a credible rationale for funding a program from public revenue.
If a rationale exists, the program must also have benefits that exceed its costs.
Cost-Benefit Analysis is used to determine whether programs pass this test.
Using Industrial Operating Models for Cost Benefit Analysis

3. In NOAA, Weather Forecasting is an Important Program
Benefits of weather forecasts can be estimated in three general ways:
Stated Preference - forecast users are asked to self-assess the benefits they receive
Data Analysis - benefits are inferred from outcome data, with and without forecasts
Economic Modeling - benefits are estimated by simulating decisions and outcomes
Using Industrial Operating Models for Cost Benefit Analysis

4. My Focus Here is on Economic Modeling
To use this method, it is necessary to model decisions and outcomes that are often complicated and esoteric.
Typically, an organization like NOAA lacks the resources and industry knowledge to do such modeling from scratch.
Thus it becomes critical to find and adapt existing industrial operating models.
Using Industrial Operating Models for Cost Benefit Analysis

5. Example: Benefits of Forecasts Used by Electricity Generators
Electricity load (use) is weather dependent, especially in regions where air-conditioning is a large component of total load.
Good temperature forecasts allow the most cost-effective set of plants to be used to serve the load.
The benefits of temperature forecasts are the generation cost savings they create.
Using Industrial Operating Models for Cost Benefit Analysis

6. The Decision-Making here is Really Complicated
A temperature forecast for the next 24 hours must be translated into an electricity load forecast for each specific electricity system.
Then generating units must be selected to minimize cost, given the load forecast.
Finally, real-time adjustments must be made to compensate for inevitable errors between forecasted and actual loads.
Using Industrial Operating Models for Cost Benefit Analysis

7. To Estimate Benefits, We Used Existing Operating Models
A study by Hobbs et. al. [1] modeled the cost savings from better load forecasts.
Hobbs used a model from Baldick [2] to represent choice among generating units, given a load forecast.
And Hobbs developed an electricity dispatch model to simulate real-time adjustments to load forecast errors.
Using Industrial Operating Models for Cost Benefit Analysis

8. We Still Needed a Link between Temperature and Load Forecasts
In our study [3], we used an operating model developed by one of our co-authors -- an electricity load forecaster known as Neural Electric Load Forecaster (NELF).
NELF uses a variety of information, including the next day temperature forecast, to estimate next day electricity load.
This provided the needed link between temperature forecasts and load forecasts.
Using Industrial Operating Models for Cost Benefit Analysis

9. We Picked Sites Representing Each of Three US Regions
Since we needed both weather and load data for specific utility service areas, there were limitations on the number and location of sites we could consider.
We choose two sites each representing respectively the northern US, the southern US, and the western US.
Using Industrial Operating Models for Cost Benefit Analysis

10. At Each Site, We Used NELF to Model Four Weather Forecasts
Naïve -- tomorrow’s weather will be the same as today’s.
MAV -- Model Output Statistic Aviation, an NWS guidance product.
NWS Forecast -- represents 7 to 11 percent improvement relative to MAV
Perfect -- tomorrow’s actual weather is used as the forecast for tomorrow.
Using Industrial Operating Models for Cost Benefit Analysis

11. Load Forecast Errors Decline as Weather Forecasts Improve
Using Industrial Operating Models for Cost Benefit Analysis

12. We Used Hobbs’ Study to Value Load Forecast Improvements
Hobbs et. al. used actual and forecasted load data from two electric utility systems, one in the northeast and one in the south.
Hobbs considered two alternative configurations of generating plants to serve each type of load.
This produced four cases spanning a range of real-world possibilities.
Using Industrial Operating Models for Cost Benefit Analysis

13. For Each Case, Hobbs Estimated Benefits of Better Load Forecasts
Hobbs scaled load forecasting errors up or down to simulate varying forecast quality.
Generation decisions were simulated using Baldick’s plant commitment model coupled with an Hobbs’ electricity dispatch model.
Generation costs were calculated and used to estimate percentage savings from better load forecasts.
Using Industrial Operating Models for Cost Benefit Analysis

14. We Averaged Hobbs’ Results Across Generating System Types
We averaged Hobb’s results across his alternative configurations of generating systems, for each of his two regions.
This produced a southern system cost savings that we applied to our Southern region, and a northeastern system cost savings that we applied to our North and West regions.
Using Industrial Operating Models for Cost Benefit Analysis

15. We Next Needed US Electricity Generation Costs by Region
Since Hobbs’ costs savings were expressed as percentage cost savings, we needed to estimate regional generation costs.
We allocated total US utility generation to our three regions (South, North, and West).
For each region, we estimated costs using $20 per million watt-hours as a typical cost.
Using Industrial Operating Models for Cost Benefit Analysis

16. Electricity Benefits of Weather Forecasts Are Mainly in South
Using Industrial Operating Models for Cost Benefit Analysis

17. There are Reasons Why Benefits Are Mainly in South
Weather forecasts reduce load forecast error much more in the South (see earlier chart).
Hobbs results show much higher costs of load forecast errors for his southern system.
Both these explanations in turn probably reflect the fact that space cooling demand is weather sensitive and very important in the South.
Using Industrial Operating Models for Cost Benefit Analysis

18. Benefits of Improved Forecasts Can Be Estimated Two Ways
The MAV and NWS forecasts differ by small amounts, and so can be directly used to estimate benefits of small improvements from the current forecast quality.
We can also fit a curve through the cases we estimated (Naïve, MAV, NWS, Perfect) and use this curve to interpolate benefits of a variety of alternative forecast qualities.
Using Industrial Operating Models for Cost Benefit Analysis

19. A One-Percent Improvement Has Similar Benefits Both Ways
Estimating the benefit of a one-percent improvement in forecast quality from the NWS and MAV cases implies a value of $1.3 million per year.
Estimating this benefit from a fitted curve implies a value of $1.1 million per year.
Using Industrial Operating Models for Cost Benefit Analysis

20. We Can Also Estimate the Value of a One Degree Improvement
Here we used the fitted curve, since a one degree improvement represents a relatively large change in forecast quality (about 1/3 of existing error).
Using the fitted curve, a one-degree improvement in forecast quality has a value of about $35 million per year.
Using Industrial Operating Models for Cost Benefit Analysis

21. These Benefits Can Be Expressed in Present Value
Compare a one-time investment to improve forecasts by one-degree with its cost and the present value of it benefits.
At 5% discount rate, the present value of a one-degree improvement in weather forecasts is about $700 million; one percent is $22-26 million.
Using Industrial Operating Models for Cost Benefit Analysis

22. Summary
Overall, official NWS forecasts appear to produce total benefits for electricity generators of about $156 million per year, mostly in the south. A perfect forecast would add $70 million.
These are benefits from plant scheduling, and do not include other possible benefits, e.g. plant maintenance decision-making.
Using Industrial Operating Models for Cost Benefit Analysis

23. Summary (Concluded)
Benefits of a small improvement in weather forecast quality were estimated to be $1.1 to 1.3 million per year, per percentage point of error reduced.
Benefits of a larger improvement (one degree, or around 1/3 of the current error) were estimated at about $35 million per year or about $700 million in present value.
Using Industrial Operating Models for Cost Benefit Analysis

24. References
[1] Hobbs, Benjamin F., et. al. Analysis of the Value for Unit Commitment Decisions of Improved Load Forecasts. IEEE Transactions on Power Systems, Vol. 14, No , pp
[2] Baldick, R., A Generalized Unit Commitment Model, IEEE Transactions on Power Systems, Vol. 10, No , pp
[3] Teisberg, T.J., R.F. Weiher, and A. Khotanzad, The Economic Value of Temperature Forecasts In Electricity Generation, Bulletin of the American Meteorological Society, Vol. 86, No. 12, 2005, pp
Using Industrial Operating Models for Cost Benefit Analysis