1. A Credible Approach to Benefit-Cost Evaluation for Federal Research &Technology Programs: A U.S. Department of Energy Approach Presented at American Evaluation Association Conference November 2009 Gretchen Jordan, Sandia National Laboratories Rosalie Ruegg, TIA Consulting, Inc. Work presented here was completed for the U.S. Department of Energy (DOE) Office of Energy Efficiency and Renewable Energy by Sandia National Laboratories, Albuquerque, New Mexico, USA under Contract DE-AC04- 94AL8500. Sandia is operated by Sandia Corporation, a subsidiary of Lockheed Martin Corporation. Opinions expressed are solely those of the authors. TIA Consulting, Inc. SAND Number: 2009-6686C

2. Jordan and Ruegg 2009 2 Outline  Overview of EERE  Objectives for Benefit-Cost Study Guidelines  Economic benefits and costs  Specifying next best  Assessing additionality  Environmental, Security, and Knowledge benefits  Challenges and Summary

3. Jordan and Ruegg 2009 3 Office of Energy Efficiency and Renewable Energy (EERE) EERE accomplishes its mission through 10 Technology Development (TD) Programs and the Office of Technology Advancement and Outreach (TAO): –Fuels & Vehicles Vehicles Technologies Biomass/Biofuels Hydrogen –Power Generation Wind & Hydropower Solar Geothermal –Energy Efficiency Building Technologies Industrial Technologies Weatherization Federal Energy Management

4. Jordan and Ruegg 2009 4 4 Purpose: to answer the following questions about EERE programs over their 30-year history To what extent have the programs thus far: Produced economic benefits in terms of resource savings relative to program costs? Yielded environmental benefits in terms of Green House Gas reductions and health effects from reduced air pollution? Yielded energy security benefits in terms of reduced imported oil and threats to the U.S. energy infrastructure? Built a knowledge base within each respective field and disseminated knowledge in and outside those fields? What has been the return on public investment in the EERE R&D energy programs thus far?

5. Jordan and Ruegg 2009 5 5 Approach: Key Features of the EERE R&D Benefit-Cost Studies Retrospective, following best-practice B-C methodology Cluster approach to extend study usefulness Comprehensive -- treatment of 4 types of benefits - Economic - Environmental - Security - Knowledge Consistent & uniform across studies (as appropriate) - Use of unifying framework (Mansfield Model ) - Same set of Benefit-Cost conventions - Same set of economic performance measures - Uniform formatting of reports

6. Jordan and Ruegg 2009 6 “Cluster Analysis” Approach: benefits of elements of a research/technology cluster” compared to entire cluster costs Quantitative benefits of selected elements of a research/technology “cluster” Investment costs of the selected elements for detailed study Investment costs of entire cluster Qualitative effects of other elements in the cluster

7. Jordan and Ruegg 2009 7 What It Provides : Economic Benefits Economic Performance Metrics –Net benefits –Benefit-cost ratio –Internal rate of return Perspective is Return on Public Investment (Cost of EERE Program/Subprogram) All affected resources in the economy are included in Economic Metrics, such as –Investment costs –Energy costs –Labor costs

8. Jordan and Ruegg 2009 8 8 Areas of Special Focus in Estimating Economic Benefits Specifying the “Defender Technology” What would have been “the next-best alternative” used in lieu of the subject technology? Accounting for “Additionality” What was different about the subject technology and market as a result of the EERE Program/Subprogram?

9. Jordan and Ruegg 2009 9 9 Specifying the “Defender Technology” Merits of the subject technology are judged retrospectively against the “next best alternative” at the time the investment decision was made Counterfactual – what would otherwise have been used Factors affecting the selection of the “defender” - were investment decisions constrained or unconstrained? - was the subject technology new to the world or an incremental improvement over an existing system? - was the subject technology a total system or a component? - was the subject technology a product or a process? Is static or dynamic modeling needed?

10. Jordan and Ruegg 2009 10 Accounting for “Additionality” Effects - Ways the program may have made a difference accelerated technology entry into the marketplace improved the performance characteristics of the technology changed the cost of a technology increased market size http://www.davisoninternational.com/knowledge/life_cycles.php

11. Jordan and Ruegg 2009 11 A Matrix for Assessing Attribution by Technology Stage Categories of Information Needed for Additionality Assessment Technology Timeline (Stage of Research, Development, and Commercialization)  Preliminary & detailed investigation Develop components Develop system Validate/ demonstrate Commer- cialize Market Adoption History of the technology What DOE Did What Others Did (Rival Explanations: Private Sector, Other Nations) What Others Did (Rival Explanations – US & State Government) The DOE Effect Type Description of DOE Influence And its strength Basis of evidence of influence

12. Jordan and Ruegg 2009 12 What It Provides: Environmental Benefits Greenhouse Gas Effects - physical units of GHG emissions avoided - equivalent changes required to produce similar GHG effects (use of EPA’s GHG Equivalencies Calculator) Public health benefits from reduced air pollution (NOx, SO2, PM, etc.)– calculated using EPA’s COBRA Model - mortality and morbidity measures - health cost measures (which may be combined with economic estimates if data quality is sufficiently high) Any notable other effects -- water resource use, water discharges, land resource use, and solid waste generation (treated at a minimum qualitatively)

13. Jordan and Ruegg 2009 13 What It Provides: Security Benefits Barrels of oil equivalent units avoided Monetary value will not be applied to barrels of oil equivalent units, as the methodology is considered at this time to require further development Notable effects on the security of infrastructure will be identified Future potential political and military security issues linked to GHG emissions will be acknowledged where the reduction of GHG emissions is notable

14. Jordan and Ruegg 2009 14 What It Provides: Knowledge Benefits Outputs and dissemination paths of patents/papers/prototypes/models to commercial users within the target industry & to other industries “Comparison of citing of EERE-attributed patents/papers with other organizations Hot” patents/papers (with greater than expected citing intensity) traceable to EERE Program Knowledge creation and exchange through partnerships with companies and universities Licensing of intellectual property (limited) International knowledge flows

15. Jordan and Ruegg 2009 15 Jordan and Ruegg 2009 15 Challenges Maintaining quality of measures across types of benefits Consistent and appropriate designation of each next-best alternative to use as a baseline for estimating economic benefits across studies Assessment of external influences that may constitute rival explanations of outcomes in estimating economic benefits Difficulties in measuring other types of benefits, e.g., security benefits, for which methodology is less well developed Inclusion of other important effects, e.g., international effects Interpretation of evaluation measures given in a combination of dollars and other units

16. Jordan and Ruegg 2009 16 Summary Essential Study Content Characteristics Appropriate study design Clear account of technologies selected for detailed case study Appropriate designation of each next-best alternative Assessment of the context and rival explanations of outcome Systematic and appropriate data collection and analyses Findings are evidence-based, conservative, and study limitations are identified

17. Thank you! gbjorda@sandia.gov ruegg@ec.rr.com http://www1.eere.energy.gov/ba/pba/performance_evaluation.html

18. Back up

19. Jordan and Ruegg 2009 19 Mansfield Model of Social Benefits from a Product Innovation that Reduces the Costs of the Industries Using It Source: Edwin Mansfield, Estimating Social and Private Returns from Innovations Based on the Advanced Technology Program, 1996.

20. Jordan and Ruegg 2009 20 Definitions

21. Jordan and Ruegg 2009 21 An example (preliminary data) PDC Drill Bit -- one of a cluster of geothermal technologies The nature of the technology: PDC Drill bits have a harder and longer- lasting cutting surface which results in more feet drilled per hour and use of fewer drill bits per hour Cluster: Entire EERE Geothermal Program, given historical cost data constraints Next best alternative: the existing roller bit technology Economic Benefits. PDC drill bits have been adopted by the oil and gas sector (off shore drilling) –$20+ billion total estimated benefits over the 1997-2007 period Attribution: Evidence suggests that DOE Role “very Important” - DOE worked closely with industry to develop and commercialize the technology