1. Improving Measures of Science, Technology, and Innovation: Interim Report (2012) Committee on National Statistics Division of Behavioral and Social Sciences and Education ROBERT E. LITAN, ANDREW W. WYCKOFF, AND KAYE HUSBANDS FEALING, EDITORS; PANEL ON DEVELOPING SCIENCE, TECHNOLOGY, AND INNOVATION INDICATORS FOR THE FUTURE; NATIONAL RESEARCH COUNCIL OF THE NATIONAL ACADEMY OF SCIENCES Kathleen Padova, December 1, 2014

2. Chapters 6 & 7  What indicators do we want  How can we get them

3. What Indicators Do We Want?

4. Current Indicators: Current State & Issues  Current Indicators on National or State level; often not granular enough  Statistics at geographic levels finer than states often not published  Previously, physical and political boundaries strongly influenced the location and transfer of innovation activities; the knowledge economy observes no limitations  Can’t track technology transfer between multi-state companies or multi- university collaborations  Economic growth, worker displacement, etc. occurring at a subnational level, data at a subnational level needed to support policy decisions at this level

5. state, county, and metropolitan tables of data from the Business R&D and Innovation Survey degrees granted in science, technology, engineering, and mathematics (STEM) (production and migration) academic R&D expenditures federal R&D expenditures total R&D STEM jobs STEM workforce migration patent applications, grants, and citations STI equity investments STEM occupational projections STEM occupation classification STEM graduate and workforce migration firm innovation processes propensity to innovate ratings mappings of entrepreneurial density industry support for R&D in universities firm births, mergers and acquisitions, deaths venture capital investments state and federal grants and loans initial public offerings new products drug and other approvals data on dealmakers and entrepreneurs, including number of connections among dealmakers and entrepreneurs) data on emerging industries, based on universities, government laboratories, firms, value chains, key occupations, and individuals

6. Potential Indicators: Challenges  Very difficult or expensive to collect  Data might not maintain same type of meaning at disaggregate level  Place matters; but is “leaky:  not necessarily clear what the geographical span of impact is for a university or a firm in a given locale  multiplant firms that span more than one state might have difficulty allocating activities accordingly  These potential problems would affect data quality and reliability at finer geographical levels.  Politics!

7. Panel Recommendations  “The National Center for Science and Engineering Statistics should host working groups in the near future to further develop subnational science, technology, and innovation indicators. Participants in the working groups should be both users and providers of the data. A main focus of the discussion should be on data reliability, particularly at fine geographical scales. Potential indicators should include subnational research and development statistics, and subnational science, technology, engineering, and mathematics workforce statistics.”

8. How Can We Collect Desired Indicators?

9. Collection: Current State & Issues  Surveys, lots of surveys  Declining survey response rate  Processing time lag between collection and publication  Data sources, standards, and taxonomies are disparate, expanding, and evolving  Users’ expectations increasing

10. Potential Collection Sources/Methods  Web scraping - semistructured data from a public web page and register it into a structured database  Can occur continuously  No need to cooperate  Partnerships/Negotiating with sources  Data is already structured  Data feed from sources (e.g. XML feed)  Combination of traditional surveys and alternate methods  Contests!  Facebook, Google+: number of students at a university, how many major in which fields  LinkedIn, Monster.com, Zerply: the composition of the labor force, geographic breakdown, skill sets, etc.  Mendeley, Academia.edu, CiteULike: how many researchers are active in which fields, how many collaborations, who collaborates with whom, how useful is a given piece of research

11. Potential Collection: Challenges  Ownership and legality  Willingness of data sources to cooperate  Scope, consistency, and quality of data  Dependence on external source for updating and maintaining data  Determining appropriate statistics for alternate data sources  Data integration  Resources

12. Panel Recommendations  “The National Center for Science and Engineering Statistics should fund exploratory activities on frontier data extraction and development methods. These activities should include:  research funding or prize competitions to harness the computing power of data specialists with a view to (a) analyzing existing public databases to develop better indicators of science, technology, and innovation activities and (b) analyzing the huge and growing amount of information on the Internet for similar purposes  pilot programs or experiments to produce a subset of indicators using web tools  convening a workshop of experts on multimodal data development, to explore the new territory of developing metrics and indicators from surveys, administrative records, and scientometric sources.”

13. Final Conclusions

14. Recommended Actions & Future Work Interim Report (2012) – Recommended actions  Develop new and revised indicators based on existing survey data  Develop new methods for obtaining data  Develop subnational STI indicators, more comprehensive measures of trade in research and development services  Develop a conceptual framework for measuring innovation activities Final Report (2014)  National Research Council. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press, 2014.  http://www.nap.edu/catalog/18606/capturing-change-in-science-technology- and-innovation-improving-indicators-to

15. Discussion  How reliable do you suppose some of the considered external data sources are for developing indicators to be used for policy making?  Given the purpose of collecting and reporting these indicators for other to respond to them, how “agile” do agencies need to be?