Presentation is loading. Please wait.

Presentation is loading. Please wait.

Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.

Similar presentations


Presentation on theme: "Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior."— Presentation transcript:

1 Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior Researcher Research Trends Seminar October 17, 2012

2 2 Outline Science of science policy A proposed conceptual framework Empirical approaches: NSF Engineering Dashboard ASTRA – Australia HELIOS – France Final thoughts

3 3 Outline Science of science policy A proposed conceptual framework Empirical approaches: NSF Engineering Dashboard ASTRA – Australia HELIOS – France Final thoughts

4 4 The emergence of a science of science policy Jack Marburgers challenge (2005) Science of Science & Innovation Policy Program at the National Science Foundation (2007) An emerging, highly interdisciplinary research field Science of Science Policy Interagency Task Group publishes a Federal Research Roadmap (2008): The data infrastructure is inadequate for decision- making STAR METRICS (2010)

5 5 Why a science of science policy? Evidence-based investments Good metrics = good incentives Science is networked and global Build a bridge between researchers and policymakers Researchers ask the right questions The adjacent possible: leverage existing and new research and expertise New tools to describe & measure communication

6 6 The timing is right:

7 7 A conceptual framework for a science of science policy

8 8 Getting the right framework matters What you measure is what you get Poor incentives Falsification Usefulness Effectiveness

9 9 A proposed conceptual framework Adapted from Ian Foster, University of Chicago

10 10 A framework to drive person-centric data collection WHO is doing the research WHAT is the topic of their research HOW are the researchers funded WHERE do they work With WHOM do they work What are their PRODUCTS

11 11 Challenge – The data infrastructure didnt exist However, some of the data do exist

12 12 Empirical Approaches Leveraging existing data to begin describing results of the scientific enterprise

13 13 An empirical approach Enhance the utility of enterprise data Identify authoritative core data elements Develop an Application Programming Interface (API) Data platform that provides programmatic access to public (or private) agency information Develop a tool to demonstrate value of API

14 14 Topic modeling: Enhancing the value of existing data David Newman - UC Irvine NSF proposals Topic Model: - Use words from (all) text - Learn T topics t49 t18 t114 t305 t49 t18 t114 t305 Topic tags for each and every proposal Automatically learned topics (e.g.): … t6. conflict violence war international military … t7. model method data estimation variables … t8. parameter method point local estimates … t9. optimization uncertainty optimal stochastic … t10. surface surfaces interfaces interface … t11. speech sound acoustic recognition human … t12. museum public exhibit center informal outreach t13. particles particle colloidal granular material … t14. ocean marine scientist oceanography … …

15 15 Stepwise empirical approach Enhance the utility of enterprise data Identify authoritative core data elements Develop an Application Programming Interface (API) Data platform that provides flexible, programmatic access to public (or private) agency information Develop a tool to demonstrate value of API

16 16

17 17 Stepwise empirical approach Enhance the utility of enterprise data Identify authoritative core data elements Develop an Application Programming Interface (API) Data platform that provides programmatic access to public (or private) agency information Develop a tool to demonstrate value of API

18 18 Outline Science of science policy A proposed conceptual framework Empirical approaches: NSF Engineering Dashboard ASTRA – Australia HELIOS – France Final thoughts

19 19

20 20

21 21

22 22

23 23

24 24 Outline Science of science policy A proposed conceptual framework Empirical approaches: NSF Engineering Dashboard ASTRA – Australia HELIOS – France Final thoughts

25 25 Linking administrative and grant funding data in Australia

26 26 Outline Science of science policy A proposed conceptual framework Empirical approaches: NSF Engineering Dashboard ASTRA – Australia HELIOS – France Final thoughts

27 27 People Describing public-private partnerships in France

28 28 What does getting it right mean? A community driven empirical data framework should be: Timely Generalizable and replicable Low cost, high quality The utility of Big Data: Disambiguated data on individuals - Comparison groups New text mining approaches to describe and measure communication ??

29 29 Final thoughts

30 30 Policy makers can engage SciSIP communities: Patent Network Dataverse; Fleming at Harvard and Berkeley Medline-Patent Disambiguation; Torvik & Smalheiser at U Illinois) COMETS (Connecting Outcome Measures in Entrepreneurship Technology and Science); Zucker & Darby at UCLA

31 31 The power of open research communities Internet and data technology can transform effectiveness of science: Informing policy Communicating science to the public Enabling scientific collaborations Interoperability is key Publishers are an important part of the community

32 32 Rebecca F. Rosen, PhD E-Mail: rrosen@air.org 1000 Thomas Jefferson Street NW Washington, DC 20007 General Information: 202-403-5000 TTY: 887-334-3499 Website: www.air.org THANK YOU!

33 33


Download ppt "Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior."

Similar presentations


Ads by Google