1. Evaluating Research Portfolios I: An Analytical Perspective Impact Evaluation of Energy R&D Portfolios* Rosalie Ruegg, TIA Consulting, Inc. ruegg@ec.rr.com www.tiaconsultinginc.com American Evaluation Association Conference Washington, DC October 18, 2013 * Based on work funded by the U.S. Department of Energy Office of Energy Efficiency and Renewable Energy

2. Impact Evaluation of Energy R&D Portfolios Approach Background Why perform R&D evaluation at a portfolio level? Methodological extensions required for DOE/EERE applications Current status: (1) DOE/EERE Retrospective Portfolio Impact Studies (2) Revised B-C Guide (3) Aggregation Tool Development Ruegg 2

3. R&D Evaluation at a Portfolio Level: Approach Ruegg 3 Note: “Portfolio” is used here to mean a grouping of investments but not to imply risk analysis or diversification.

4. Background ATP, more than a decade ago, began performing traditional project-level impact studies. OMB response was that project-level studies were not meaningful because they represented “cherry picking.” ATP moved to evaluating portfolios of R&D projects to provide conservative estimates of return on larger pieces of its investment. DOE/EERE extended the portfolio approach by: a) using a more systematic approach according to a Guide, b) broadening coverage of benefits. Ruegg4

5. Advantages of DOE/EERE’s Portfolio Approach Rigorous evidence of return on EERE’s R&D investment Systematic treatment allows aggregation across portfolios Efficient use of the evaluation budget Ruegg5

6. Areas of Emphasis in DOE/EERE’s Evaluation Characterizing technology content by type (e.g., infratechnologies, platform or generic technologies, product technologies) Identifying next-best technology alternative Detailed treatment of additionality Data collection Estimation of: - Resource impacts - Environmental impacts - Energy security impacts - Life-cycle effects - Knowledge effects - Combined impacts Multiple economic performance metrics Lessons learned Ruegg6

7. Methodological Extensions to Broaden Coverage: Impacts Measured in Dollars Ruegg7

8. Methodological Extensions to Broaden Coverage: Impacts Measured in Physical Units Ruegg8

9. Methodological Extensions to Broaden Coverage: Impacts Measured Qualitatively Ruegg9

10. Methodological Extensions: Overview Ruegg10

11. Economic Performance Metrics Ruegg11

12. Evaluative Questions Addressed What has been the return on DOE/EERE R&D investment? To what extent has EERE's investment produced energy, environmental, energy security, knowledge, and economic benefits? Would today’s commercialized technologies likely have happened at the same time, and with the same scope and scale, without EERE’s efforts? Was the public investment worth it? What factors led to return on public investment? What lessons learned can be applied to future R&D investments? Ruegg12

13. Current Status  Retrospective Impact Guide: 2008, 2011(R), 2013(R) 2008 & 2011 editions prepared by Ruegg & Jordan 2013 edition (in review) prepared by Ruegg, O’Connor, & Loomis  Portfolio Studies Completed: 5  Data Aggregation Tool under Development Ruegg13

14. Portfolio Aggregation Tool Select from the following: SelectModuleImpact Studies of DOE/EERE R&D Portfolios Level of Analysis Detailed Summary 1Geothermal Program 2Solar Photovoltaic Subprogram 3Wind Energy Program 4Advanced Batteries for HEV & EV 5Advanced Combustion 6Aggregation Across Portfolio StudiesSummary

15. Case example Ruegg15

16. Example from DOE/EERE: Advanced Combustion Engine R&D Portfolio (Link, 2010) CRF & laser and optical diagnostics tools = Infratechnology “technologies that influence the R&D, production, or marketing of other technologies but are usually not physically embodied in resulting products” Next-Best Alternative Identified: Ruegg16 Technology Type Identified:

17. Comparing Actual and Counterfactual BTE (Link, 2010) Ruegg17

18. Analysis Linked MPG to Change in BTE and Translated Results to Fuel Savings and Reduced Air Emissions Experts estimated 4.5% improvement in Brake Thermal Efficiency (BTE) from DOE-supported R&D MPG was linked to BTE with an econometric model Fuel savings were calculated from MPG fuel economy improvements and miles driven. Resulting energy savings: 17.6 billion gallons of diesel fuel from 1995 to 2007. (Other effects were also calculated). Ruegg18