1. Theme 2: Investing in Science and Innovation Summary December 4, 2008

2. High-Level Observations Questions 4 and 7, understanding the value of research and the effectiveness of investments, appear to be particularly important ~85% rated as “high importance” compared to 20-40% for other questions

3. High-Level Observations There appears to be general agreement with the Roadmap findings and recommendations, with noted exceptions: 1.There is a lack of broad awareness about approaches and techniques used by other agencies 2.There is concern about the value and applicability of bibliometrics—but broader support for more “innovative” rather than “standard” applications??

4. Data on Recommendations FindingsNumber who agreedPercent who agreed 1) Recommendation 5: Agencies should work together to develop a pilot data infrastructure4677.97% 2) Recommendation 6: Agencies should work together to develop pilot standards for identifying ways of measuring the value of knowledge5084.75% 3) Recommendation 7: Agencies should work together to develop standard approaches for using bibliometrics to assess science impact.3661.02% 4) Recommendation 8: The academic research community should continue to be supported to perform the best research to develop new analytical tools, methods, and metrics to support the emerging science of science policy.5288.14%

5. Possible Roles/Contributions Agencies: Funding/personnel = low, but information sharing = high Academia: Funding/personnel = high, but information sharing = low International: Less relevant Private sector??

6. Top Models Peer/Expert Review #1 Econometric; Cost/benefit Data Collection: Survey Visualization tools rated very low

7. Top Metrics Bibliometrics (citations; patents; scientific papers) highest rated Nothing else highly rated

8. Moving Forward Agree/Disagree… 1.To move forward, we must first develop common definitions and understanding of SOSP concepts and tools. 2.It is essential that we more clearly determine the roles that agencies, academia, and in particular, the private sector can play in developing the field.

9. Moving Forward In the near-term, do these tools merit greater attention given their high-level of maturity: a) cost effectiveness/ benefit modeling, b) case studies, c) surveys? Number who ranked Relative Maturity High Percent who ranked Relative Maturity High Number who ranked Potential Value High Percent who ranked Potential Value High Cost Benefit 23 71.88% 14 41.18% Cost Effectivenes s 15 62.50% 15 48.39% Case Studies 27 72.97% 22 56.41% Survey 31 83.78% 18 46.15%

10. Moving Forward Do micro- and portfolio-level outcome metrics offer the highest potential value to science policymaking? Number who ranked Relative Maturity High Percent who ranked Relative Maturity High Number who ranked Potential Value High Percent who ranked Potential Value High Outcomes Scientific/ Micro Level Innovation 3 10.34% 21 70.00% Competitive ness 3 10.71% 13 41.94% Knowledge Increase 7 26.92% 20 66.67% Program/Po rtfolio Effectivenes s 3 10.71% 22 75.86% Value 3 11.11% 20 74.07%