1. TIA Consulting March 2002 Estimating the Contribution of a Government Program Rosalie T. Ruegg TIA Consulting ruegg@ec.rr.com Conference on Estimating the Benefits of Government-Sponsored Energy R&D Crystal City, VA March 5, 2002

2. TIA Consulting March 2002 Attributing Benefits to a Government Program inputsoutputsoutcomesimpacts What would have happened without the gov’t effort? [counterfactual studies] What happened in similar cases where there was no gov’t intervention? [control group and comparison studies] It’s not enough to measure these effects; evaluation seeks to determine how much is attributable (caused by) the gov’t intervention.

3. TIA Consulting March 2002 Overall Net Benefits to Innovation X Government +Industry R&D Investment NPV Benefits computed against a baseline

4. TIA Consulting March 2002 To estimate the return on government investment, we need to isolate its effect Benefits w/o gov’t measured against the baseline Private Actions w/o Gov’t Intervention

5. TIA Consulting March 2002 A Gov’t Program Might Affect Benefits By: Accelerating development & deployment - start earlier - progress faster - pursue what otherwise would not be done Increasing scale  raising likelihood of success Increasing scope  widening applications Building new innovation networks/pathways Other effects (It might also substitute for private investment)

6. TIA Consulting March 2002 Impact of Acceleration on Social Returns Source: RTI, NIST GCR 97-737, 1998, fig. 2-3

7. TIA Consulting March 2002 Illustrative Approaches to Estimating Gov’t Impact (examples from ATP) Purposes: 1. To support specific project benefit-cost case studies 2. To provide program-wide evidence of attribution Study Examples: Survey Method e.g., Silber, Powell, Laidlaw –counterfactual analysis Feldman/Kelley--control group study Case Study Method e.g., Gompers/Lerner, RTI, Pelsoci, 50 completed Projects, etc. Econometric/Sociometric Methods e.g., Branstetter/Sakakibara, Zucker/Darby, Fogarty/Sinha/Jaffe, etc.

8. TIA Consulting March 2002 Acceleration: Laidlaw Survey of 28 ATP Projects Applied research cycle time Reductions: Median reported values - 50% reduction - 3 year cut Range of reported values - % reduction: 25% to infinity - years cut: 1-2 to unbounded

9. TIA Consulting March 2002 Other Relevant Effects: Laidlaw Survey 86% expected flow-thru of time cuts to later stages 79% gave “ballpark estimates” of the value of cutting time by 1 year Respondents listed ways the program cut time - upfront planning and mangt requirements - achieving a critical funding mass - enhancing ability to attract additional funding - increasing project stability - encouraging collaboration 86% reported carry-over of acceleration to other projects

10. TIA Consulting March 2002 Feldman/Kelley Non-Winner “Comparison Group” Analysis as Evidence that Gov’t Program (ATP) Makes a Difference Extent of Follow-up ActivityPercentage Did not proceed with the project at all 61.4 Began project on a much smaller scale 16.4 Began project on a somewhat smaller scale 11.7 Began project at about the same scale as proposed to ATP 5.3 Began project at somewhat larger scale 2.9 Began project on a much larger scale 0.6 Refused to answer/don’t know1.8 Source: Feldman & Kelley, NRC Report on ATP, 2001, p. 134.

11. TIA Consulting March 2002 Discussion Topic: Use of a 5-Year Acceleration Rule for DOE Program Evaluation NRC basis for? Is it “conservative”? Is it generalizable? Is it feasible to determine specific time lags on a program-by-program basis or should it be on a project-by-project basis? (ATP’s experience suggests no fixed Rule)