1. EVALUATING NSF EPSCOR INITIATIVES: TRACKING AND ASSESSING THE DEVELOPMENT OF EPSCOR-SUPPORTED RESEARCH CAPACITY Julia Melkers, Associate Professor School of Public Policy, Georgia Institute of Technology Eric Welch, Associate Professor Department of Public Administration, University of Illinois at Chicago Prepared for presentation at the National Research Council Washington, DC December 17, 2012

2. Evaluation frames are based on the understanding that:  Building research capacity is a complex and continuous process. Tracking magnitude of changes over time is critical to capturing effects (interim effects).  Given its mission and the structure of the awards, EPSCoR has the potential to affect research capacity in range of ways.  Capacity is reflected in (and affected by): State politics and science objectives Institutions Infrastructure Human capital Collaborative relationships Resources Among others....

3. Year 1Year 2Year 4Year 5

4. Year 1Year 2Year 4Year 5 Institutionaliza tion Development of early collaborative outputs Increased production & other impacts Movement toward sustainability

5. Year 1Year 2Year 4Year 5 Institutionaliza tion Development of early collaborative outputs Increased production Movement toward sustainability Interaction Collaboration Production

6. Dimensions of Research Capacity  Capacity as infrastructure  Lab equipment  Cyberinfrastructure  Institutional processes and policy  Capacity as human capital  Scientists  Postdocs  Graduate Students  Capacity as integration  Integrating different expertise and perspectives on the research problem (disciplinary, translational, etc)  Capacity as productivity  Grant activity that extends and builds upon award work Research Faculty developme nt Workforce /Students Education Institutions External ties/ community

7. Using Theories of Change in Evaluation: Example Project Logic Model

8. Sample Guiding Evaluation Questions: Collaboration  Collaboration: What are the barriers and facilitators to collaboration in [state] EPSCoR?  within and between sites/groups?  interdisciplinary and cross institutional?  early career engagement?  mid career role?  Native and rural community partners?  private and non profit sector integration?  international partners?  critical success factors? Barriers to success Opportunities to enhance success

9. Sample Guiding Evaluation Questions: Impacts  Research Impacts: What are the knowledge impacts of [EPSCoR] research and activities?  Stakeholder Impacts: How has [EPSCoR] developed interactions /tangible relationships with external partners/stakeholders over the course of the project?  Faculty and Staff Impacts: What research and management capacity is increasing as a result of [EPSCoR] work?  Student Impacts: What benefits and capacities do students accrue from [EPSCoR] ?  Institutional Impacts: How has the research capacity of [state] institutions changed via [EPSCoR] ?

10. Data Sources Interviews/Focus Groups Surveys (Quant & Qual) Grant proposals Publications (conf papers?) Other data (data access?) Research Faculty developme nt Workforce /Students Education Institutions External ties/ community

11. Example: Some Observable Outcomes  Science  Knowledge advancement and integration  Research productivity  Research impacts  Increased know-how  Faculty/Researchers  Production  Advancement  Enhanced or new collaborative ties  Student engagement  Leadership  Visibility  “Next generation”  Student learning and other impacts  STEM interests and advancement  Institutional capacity  New programs, processes  Systems  Support staff  Culture

12. Measuring Research Activities & Outcomes Knowledge networks Co-authorship ties and affiliation Co-PI/grant team ties Integration scores E-I Index Collaborative Development Publication activity Journal subject categories Journal Impact rates Citation rates Citation rapidity Citation location Knowledge Impacts

13. Collaborative Development Intermediate outcomes that precede academic production  Emerging collaborative relationships  Integration of students in research  Knowledge transfer and Impacts  Research-relevant behavioral changes  Nature of linkages and interactions Operationalized through Network Analysis  Pre-publication interaction  Faculty-faculty & faculty-student collaboration  Changed conceptual/ theoretical/ methodological approaches  Changed research outlets and communities  Mechanisms/modes for boundary- spanning linkages and outcomes Measurable through network analysis

14. Example: Student Interaction

15. Knowledge Impacts Alaska-Based Orgs (Federal, State, Private, NGO) International Top Ten Subject Categories (2007-12 EPSCoR Publications # Rec ords Subject Category 137 Environmental Sciences & Ecology 227Ecology 318Geology 417 Environmental Sciences 512 Environmental Studies 612Zoology 711 Physical Geography 810 Geosciences, Multidisciplinary 98 Evolutionary Biology 107Biochemistry & Molecular Biology Other EPSCoR States Other States

16. Rafols, Porter and Leydesdorff (2009) Cognitive Sci. Agri Sci Biomed Sci Chemistry Physics Engineering Env Sci & Tech Matls Sci Infectious Diseases Psychology Social Studies Clinical Med Computer Sci. Business & MGT Geosciences Ecol Sci Econ. Polit. & Geography Health & Social Issues Global Map of Science, 2007 221 SCI-SSCI Subject Categories Knowledge Impacts: Disciplinary Placement & Knowledge Sources

17. Reflections on EPSCoR Outcomes  Research  Integration across disciplines – new questions, approaches  Added value of collaborative engagement -- generation of new ideas and new collaborative relationships  Knowledge impacts -- National and global visibility and impacts  STEM workforce  Early career affiliates – early grants, network reach, production, time to tenure, mobility and competitiveness, mentoring and advice relationships  Students – research networks, apprenticeship structures, mentoring relationships, production, next career step  Research – related support staff, sustained research relationships, new grant awards  Institutions  EPSCoR offices – communication, administrative structures, management  Smaller institutions – research activity, developed capacity and integration, statewide representation in decision making (eg. state S&T plan)  Curricular issues – new courses, content  Social Impacts  engaged stakeholder organizations – economic development, social and economic objectives, support for education and research by stakeholders

18. Issues Relevant to Evaluation of EPSCoR  EPSCoR projects and settings vary.  Disciplinary variance  Partner relationships  Under-represented populations  Changes over time (historically)  Multiple and competing programmatic goals  Lack of clarity/changing evaluation needs from EPSCoR  Data consistency, across states and over time.  Increased attention to strategic plans  Unpacking role/impacts of senior leads vs other faculty/researchers  Institutional variation in data availability (esp. small schools).  Institutional effects not always evident  Futures are unclear -- difficulty/lack of support to track students and faculty post-award.  Issue of relationships across awards and cumulative effects on a state are not captured.