Evaluating Research Portfolios I: An Analytical Perspective Impact Evaluation of Energy R&D Portfolios* Rosalie Ruegg, TIA Consulting, Inc. American Evaluation Association Conference Washington, DC October 18, 2013 * Based on work funded by the U.S. Department of Energy Office of Energy Efficiency and Renewable Energy

Impact Evaluation of Energy R&D Portfolios Approach Background Why perform R&D evaluation at a portfolio level? Methodological extensions required for DOE/EERE applications Current status: (1) DOE/EERE Retrospective Portfolio Impact Studies (2) Revised B-C Guide (3) Aggregation Tool Development Ruegg 2

R&D Evaluation at a Portfolio Level: Approach Ruegg 3 Note: “Portfolio” is used here to mean a grouping of investments but not to imply risk analysis or diversification.

Background ATP, more than a decade ago, began performing traditional project-level impact studies. OMB response was that project-level studies were not meaningful because they represented “cherry picking.” ATP moved to evaluating portfolios of R&D projects to provide conservative estimates of return on larger pieces of its investment. DOE/EERE extended the portfolio approach by: a) using a more systematic approach according to a Guide, b) broadening coverage of benefits. Ruegg4

Advantages of DOE/EERE’s Portfolio Approach Rigorous evidence of return on EERE’s R&D investment Systematic treatment allows aggregation across portfolios Efficient use of the evaluation budget Ruegg5

Areas of Emphasis in DOE/EERE’s Evaluation Characterizing technology content by type (e.g., infratechnologies, platform or generic technologies, product technologies) Identifying next-best technology alternative Detailed treatment of additionality Data collection Estimation of: - Resource impacts - Environmental impacts - Energy security impacts - Life-cycle effects - Knowledge effects - Combined impacts Multiple economic performance metrics Lessons learned Ruegg6

Methodological Extensions to Broaden Coverage: Impacts Measured in Dollars Ruegg7

Methodological Extensions to Broaden Coverage: Impacts Measured in Physical Units Ruegg8

Methodological Extensions to Broaden Coverage: Impacts Measured Qualitatively Ruegg9

Methodological Extensions: Overview Ruegg10

Economic Performance Metrics Ruegg11

Evaluative Questions Addressed What has been the return on DOE/EERE R&D investment? To what extent has EERE's investment produced energy, environmental, energy security, knowledge, and economic benefits? Would today’s commercialized technologies likely have happened at the same time, and with the same scope and scale, without EERE’s efforts? Was the public investment worth it? What factors led to return on public investment? What lessons learned can be applied to future R&D investments? Ruegg12

Current Status  Retrospective Impact Guide: 2008, 2011(R), 2013(R) 2008 & 2011 editions prepared by Ruegg & Jordan 2013 edition (in review) prepared by Ruegg, O’Connor, & Loomis  Portfolio Studies Completed: 5  Data Aggregation Tool under Development Ruegg13

Portfolio Aggregation Tool Select from the following: SelectModuleImpact Studies of DOE/EERE R&D Portfolios Level of Analysis Detailed Summary 1Geothermal Program 2Solar Photovoltaic Subprogram 3Wind Energy Program 4Advanced Batteries for HEV & EV 5Advanced Combustion 6Aggregation Across Portfolio StudiesSummary

Case example Ruegg15

Example from DOE/EERE: Advanced Combustion Engine R&D Portfolio (Link, 2010) CRF & laser and optical diagnostics tools = Infratechnology “technologies that influence the R&D, production, or marketing of other technologies but are usually not physically embodied in resulting products” Next-Best Alternative Identified: Ruegg16 Technology Type Identified:

Comparing Actual and Counterfactual BTE (Link, 2010) Ruegg17

Analysis Linked MPG to Change in BTE and Translated Results to Fuel Savings and Reduced Air Emissions Experts estimated 4.5% improvement in Brake Thermal Efficiency (BTE) from DOE-supported R&D MPG was linked to BTE with an econometric model Fuel savings were calculated from MPG fuel economy improvements and miles driven. Resulting energy savings: 17.6 billion gallons of diesel fuel from 1995 to (Other effects were also calculated). Ruegg18