1. National Institute of Standards and Technology U.S. Department of Commerce Technology Program Evaluation: Methodologies from the Advanced Technology Program Survey Development, Uses, and Findings Stephen Campbell Technology Innovation Program National Institute of Standards and Technology Collaborative Expedition Workshop #71 National Science Foundation March 18, 2008

2. National Institute of Standards and Technology U.S. Department of Commerce Framework  Concepts to measure  Mission  Broader research interests  Operationalizing the concepts  Quantitative and/or qualitative  Absolute and/or relative  Uses and findings from the data  Summary stats  Fuller analysis (regression)

3. National Institute of Standards and Technology U.S. Department of Commerce Different Surveys for Different Questions  Did ATP select the “right” types of projects?  “Process” evaluation  Driven by mission and selection criteria  What are the determinants of success in ATP projects?  “Project” evaluation  Define measures of success  Are the selection criteria correlated with success? What are the other determinants?

4. National Institute of Standards and Technology U.S. Department of Commerce Evaluating the Selection Process  Selection criteria  Innovative  High-risk  Encourage longer research time horizons  Foster collaboration  Broad-based benefits (diffusion)  Appropriate need for ATP funding (counterfactual)  Used the Survey of ATP Applicants  Three surveys fielded for award competitions in 2000, 2002, 2004  Surveyed both awardee and non-awardee companies  Combined with administrative data

5. National Institute of Standards and Technology U.S. Department of Commerce Operationalizing Selection Criteria (Example for Risk)  Quantitative measures for technical risk of ATP project and for a “typical” project at the company  Did ATP fund higher risk projects and/or projects with a greater “stretch” in the risk profile?  Survey questions  “From 0 to 100%, what would you say is the approximate probability that your proposed ATP project could fully achieve its technical goals? (Try to answer based on how you thought about your project when you proposed it to ATP)”  “What is the approximate probability that a typical R&D project at your company could fully achieve its technical goals?”

6. National Institute of Standards and Technology U.S. Department of Commerce Operationalizing Risk (cont’d)

7. National Institute of Standards and Technology U.S. Department of Commerce Operationalizing Selection Criteria (Example for Time Horizon)  Quantitative measures for time to impact on revenues of ATP project and for a “typical” project at the company  Did ATP fund longer time horizon projects and/or projects with a greater “stretch” in the time horizon?  Survey questions  “Approximately how many years after the start of your ATP proposed project could you expect results to first have an impact on company revenues? (Try to answer based on how you thought about your project when you proposed it to ATP)”  “Approximately how many years after the start of a typical R&D project could you expect results to first have an impact on company revenues?”

8. National Institute of Standards and Technology U.S. Department of Commerce Operationalizing Time Horizon (cont’d)

9. National Institute of Standards and Technology U.S. Department of Commerce Regression Analysis of the Selection Process  Projects with greater technical risk were more likely to receive an award  Projects with a longer time horizon were more likely to receive an award  Companies receiving an ATP award are able to leverage additional funding to the line of research around the ATP project in the post-application period  Evidence of a “value-added” for the ATP selection panels

10. National Institute of Standards and Technology U.S. Department of Commerce Evaluating Project Success  Content areas  Key Personnel  Subcontracting  Company characteristics  ATP project characteristics  Research effort  Project management  Research outputs  Technology commercialization  Uses the Business Reporting System  Awarded companies surveyed annually from project start to project end  Awarded companies surveyed 2, 4, and 6 years after project completion in a “Post Project Survey”

11. National Institute of Standards and Technology U.S. Department of Commerce Operationalizing Technical Diffusion  Categorical measures for research results and know-how becoming known to others outside of the company  Within 2 years of project end  2 to 5 years after project end  5 to 10 years after project end  10 or more years after project end  Never  Survey questions  “How quickly do you expect critical research results from this project to become known to other outside your company?”  “How quickly do you expect critical research ‘know-how’ from this project to become known to other outside your company?”

12. National Institute of Standards and Technology U.S. Department of Commerce Technical Diffusion – cont’d  Research results  Within 2 years of project end72%  2 to 5 years after project end24%  5 to 10 years after project end 3%  10 or more years after project end 0%  Never 1%  Research know-how  Within 2 years of project end39%  2 to 5 years after project end42%  5 to 10 years after project end 12%  10 or more years after project end 5%  Never 2%

13. National Institute of Standards and Technology U.S. Department of Commerce Operationalizing Continued Technology Development  Categorical measures for the significance of technical and non- technical challenges to overcome before achieving widespread benefits of the ATP technology  Very significant  Moderately significant  Somewhat significant  Not significant  Survey questions  “How significant are any additional technical (research, development) challenges that still need to be addressed in order to achieve widespread commercialization?”  “How significant are any additional non-technical (regulatory, business) challenges that still need to be addressed in order to achieve widespread commercialization?”

14. National Institute of Standards and Technology U.S. Department of Commerce Technology Development – cont’d  Technical challenges  Very significant22%  Moderately significant42%  Somewhat significant 29%  Not significant 7%  Non-technical challenges  Very significant22%  Moderately significant41%  Somewhat significant 27%  Not significant11%

15. National Institute of Standards and Technology U.S. Department of Commerce Regression Analysis of the Determinants of Project Success  Small companies are more likely than medium or large size companies to experience early revenues from products incorporating the ATP-funded technology  Joint venture (non-lead) partners are less like than single companies or joint venture leads to realize both research and commercial outputs  University participation in a project has a positive effect on knowledge creation and dissemination, as measured by journal publications  Projects with university participation are less like to stop early for adverse reasons, which may reflect essential differences in motivation or characteristics of these projects  Projects with greater innovativeness and risk (closer to the research frontier) are more likely to realize technical and commercial outputs

16. National Institute of Standards and Technology U.S. Department of Commerce Concluding Thoughts  ATP was “blessed” to be such a controversial and scrutinized program  Early dedication to evaluation must be accompanied by resources (don’t forget about underlying database construction, maintenance, and integration issues)  Will always face a budget constraint and must prioritize  Basic reporting  Larger, broader research portfolio interests  “Boundary of the firm” issues