1. Evaluating the Alaska EPSCoR Phase III: Resilience and Vulnerability in a Rapidly Changing North: The Integration of Physical, Biological and Social Processes Dr. Julia Melkers, Associate Professor of Public Policy Georgia Institute of Technology Dr. Eric Welch, Associate Professor of Public Administration University of Illinois at Chicago Year 3 Evaluation Presentation Alaska EPSCoR All Hands Meeting May 2010 Fairbanks, Alaska

2. Stating the Obvious Collaboration is difficult, problems of: disciplinary language, distances, funding, methodological, conceptual and theoretical approaches lack of clarity about why research questions are “important”. Requires commitment to learn from other fields Variation in goals and interests of stakeholders and rural campuses Institutional barriers and facilitators are key to success

3. Three Year Evaluation of Alaska EPSCoR Phase III A formative and interactive approach. Can also provide lessons learned for future EPSCoR proposals and initiatives. An summative evaluation process to serve as an accountability mechanism. A research driven evaluation, based on studies of science, collaboration, networks, among others.

4. Year 1Year 2Year 3  What are the foundational capacities of AK EPSCoR faculty and students?  What foundational relationships exist?  What early outcomes can be observed?  How are collaborative interactions and related outputs developing in AK EPSCoR?  How integrative is AK EPSCoR and what are the related barriers?  What benefits are EPSCoR students experiencing?  How are collaborative interactions and related outcomes continuing to develop among students and faculty?  How integrative are these interactions and outcomes?  How are rural campuses and native communities benefiting from EPSCoR? What are the key issues in this?  How are institutional and administrative factors relevant to the ability of AK EPSCoR to meet its goals? Evaluation Questions

5. Year 1Year 2Year 3 Evaluation Focus  Collaboration  Familiarity  Interdisciplinary orientation  Collaboration  Productivity  Factors affecting productivity  Integration  Student impacts  Continuation of year 2 issues  Collaboration  Administrative issues  Rural campus Data Collection Baseline Data Collection:  Survey of faculty and students  Interviews with AK EPSCoR leadership Interim Data Collection:  Student interviews  Survey of faculty and students Final Data Collection:  University administrator interviews  Native organizations interviews  Rural campus interviews  Survey of faculty and students

6. Social Networks & Evaluation Social network analysis can add to the program evaluation methodological toolbox. Why use SNA in program evaluation? Allows us to capture knowledge development, social and human capital, and other interim outcomes that cannot be captured in other data. Adds rigor to attitudinal and self reported data regarding behavior, outputs, outcomes, and relationships. Adds detailed dyadic data on specific relationships and exchange. Can be combined with other data to provide a comprehensive picture. Data may be drawn from surveys, interviews and existing documentation.

7. Some Network Characteristics Network Size Number of nodes that are connected to others Number of Ties Number of ties that link these nodes individuals can have multiple ties) Network Density measures the percentage of ties that exist compared the number of possible ties. This measure provides one way of describing the extent to which potential ties are unexploited.

8. Some Network Characteristics Average Degree Centrality measures the average number of immediate connections that each individual has in the network. Provides a way to examine the level of participation in network activity by the ‘average’ person in the network.

9. Some Network Characteristics (E-I) Index the extent to which the network is made up of individuals outside as compared to inside a particular environment or context. The EI index is calculated as: (external ties – internal ties) / (external ties + internal ties) and ranges from negative one to one.

10. Our Final Report Today’s Presentation Collaborative interactions – Year 3 and Overtime Integration with rural campuses and communities Student Impacts Observations and some lessons learned Written Report Collaborative Interactions and Outcomes Student benefits and learning Integration with native communities Rural Campus issues Attitudes and feedback regarding EPSCoR issues Lessons Learned Recommendations for future EPSCoR efforts

11. Overview of Survey Respondents Year 1 Faculty: 86% response rate (n=59) Students: 85% response rate (n=57) Year 2: Faculty: 75% response rate (n=79) Students: 72% response rate (n=48) Year 3 Faculty: 75% response rate (n=111) Students: 57% response rate (n=52)

12. Year 3 Faculty/Researcher Respondents (n=111) Rank: Assistant Professor = 35% Associate Professor = 26% Full Professor = 21% Research faculty & other = 24%

13. Year 3 Student Respondents (n=52) 84% respondents at UAF

14. Comparing Developments Across Time Useful to track the EPSCoR community General observations on community of researchers. More meaningful to track core group of individuals who have been involved over time. Longitudinal Faculty Comparison Groups  Survey Respondents from Years 1 & 3 (n=44)  Year 1 respondents and “active” individuals who responded to Year 3 survey (n=65) (59% of Year 3 respondents)  Changes in Year 2 & 3

15. COLLABORATION & INTEGRATION

16. Sole Authoring by Component Y3 (mean number of faculty reporting these products)

17. Sole Authoring by Institution Y3 (mean number of faculty reporting these products)

18. What became of Desired Collaboration Year 1? Respondents // Named desired collaborators 48 // 60 Percent collaborators named in same discipline 44/60 (73.3%) Percent collaborators named in other disciplines 16/60 (26.7%) Same discipline Bio to biological science 25 Phys to physical science 4 Soc to social science 15 Different discipline Bio to physical science 3 Bio to social science 10 Phys to social science 3 Percent disciplinary collaborations 2/16 (12.5%) Percent interdisciplinary collaborations 10/44 (22.7%) Who would you like to collaborate with, but have not done so to date?

19. Reasons for Successful Collaborations Sharing common research interests in permafrost related arctic engineering and science issues. Common interest in phylogenetic methodology, especially methods of DNA sequence alignment. We are both interested in future trends in moose and caribou populations. We are both interested in developing programs that link arts with the sciences. Collaborative style and our complementarity [That person’s] energy and enthusiasm for doing challenging empirical research.

20. Production: Journal Articles Red=Biological Science Blue= Social Science Green=Physical Science Square= Year 1 & 3 participant

21. Impacts on Communication Red=Biological Science Blue= Social Science Green=Physical Science Square= Year 1 & 3 participant

22. Impacts on Research Red=Biological Science Blue= Social Science Green=Physical Science Square= Year 1 & 3 participant

23. Growing Collaboration Around Key Research Themes Red=Biological Science Blue= Social Science Green=Physical Science Square= Year 1 & 3 participant

24. Coalescing of Research Themes Species response to climate change Disturbance regimes Human migration & climate change Perma -frost Social A aspects of ground engineering Ecosystem services 2009-10 10012086955295 2008-10 1277869795795 Change in the Number of Collaborative Ties over Time

25. Do Networks Matter for Continued Activity? Dependent Variable – Indicator of Continuation Have you developed any new ideas or plans for research as a result of your interaction with EPSCoR Phase III? (1/0) Independent Variables Familiarity “I understand this person’s knowledge and research skills. This does not necessarily mean that I have these research skills or am knowledgeable in these domains, but that I understand the skills this person has and domains in which they are knowledgeable.” (Strongly Disagree to Strongly Agree) Size of Collaboration Network: Since July 2007, with which of the following EPSCoR faculty members did you work with on a journal article. (1/0) Other controls: Tenured, Biology, UAF

26. Example Findings BSESignificanceExp(B) Familiarity 1.6470.778**5.191 Collaboration Network Size 1.3410.687**0.262 Biology 1.9621.4210.141 UAF -1.4741.0894.366 Tenured -1.2231.0163.399 Constant 3.4791.920*0.031 Have you developed any new ideas or plans for research as a result of your interaction with EPSCoR Phase III? (1/0) *= p<0.10, ** = p<0.05)

27. INTEGRATION WITH RURAL CAMPUSES AND COMMUNITIES

28. Stakeholder Interaction Faculty report limited importance of interaction with stakeholders. Faculty report few meetings with stakeholders over the course of EPSCoR Phase III (1= never, 2= once, 3= 2 to 3 times, 4= more than three times) UAAUASUAF Private Companies 1.861.501.96 Rural campuses 1.371.571.50 K-12 education 1.642.432.05 Native communities 2.221.712.16 Native Alaskan corporations 1.461.001.62 State government officials 2.071.862.26 Local government officials 2.101.571.95 Federal government officials 2.112.002.29

29. Research in Native Communities Native organizations and local contacts critical. Provide access Experiences with researchers are varied EPSCoR can facilitate the capacity of faculty to interact with native communities Nature of outreach matters Input / agreement of community Follow-up is key Relevant research has implications for attraction and retention in STEM

30. Rural Campuses Play key role as liaison to native communities Faculty are important nodes for connection Student are important nodes for connection Serve multiple roles Skills and training Research for understanding local context and for ensuring community security Capacity building in community EPSCoR is an important supporter of research dissemination through Western Alaska Interdisciplinary Science Conference (WAISC).

31. STUDENT IMPACTS

32. EPSCoR Faculty and Students 28% of faculty report that they are currently working with an EPSCoR supported student. 50% of those report that the research would not have been conducted without EPSCoR funding.

33. Students Report Positive Impacts: Research Interests 75% of student respondents are required to do a thesis. Of these: 65% report that their thesis or dissertation includes some aspects of climate change research. 55% report that their thesis or dissertation topic “has been influenced by their EPSCoR work.”

34. To what extent has your Alaska EPSCoR experience helped you to develop skills and knowledge in the following areas? (1= not at all, 2= some, 3= a great deal)

35. PROSPECTS FOR THE FUTURE

36. Faculty Collaboration Interests UAAUASUAF I would like to collaborate more with faculty in different disciplines. 3.193.433.17 I would like to collaborate more with faculty on other UA campuses. 3.323.573.00 I would like to collaborate more with students in different disciplines. 3.003.172.94 I anticipate continuing to collaborate with individuals I have worked with in Alaska Phase III 3.283.863.31 Generally positive attitudes across institutions regarding collaboration. (4 point scale, 1=strongly disagree)

37. Faculty Collaboration Interests Faculty component-based responses show similar variation in interest in interacting with faculty and students in other disciplines. BiologicalSocialPhysical I would like to collaborate more with faculty in different disciplines. 3.053.283.30 I would like to collaborate more with faculty on other UA campuses. 3.133.143.10 I would like to collaborate more with students in different disciplines. 2.843.023.10 I anticipate continuing to collaborate with individuals I have worked with in Alaska Phase III 3.393.283.38

38. Production: Grant Proposals to Extend Phase III Research Red=Biological Science Blue= Social Science Green=Physical Science Square= Year 1 & 3 participant

39. Capacity Development in EPSCoR Phase III Research: Evidence that collaborative networks have increased in size and diversity over Phase III. Research theme-based networks are gaining more research ties across components. Production and grant activities to sustain the capacity developed through Phase III are evident. Faculty report positive impacts of EPSCoR engagement: Production Connections Research ideas Changes in research processes

40. Capacity Development in EPSCoR Phase III Education : Students report positive impacts of EPSCoR Phase III Level of student research interests in climate change, as evidenced by thesis and dissertation topics, indicates potential capacity impacts on new generation of climate change researchers. Strong student attribution of research skill development to EPSCoR Phase III, particularly among undergraduates.

41. Capacity Development in EPSCoR Phase III Rural Campuses and Communities: Significant interest in developing stronger ties to bring research and education opportunities and resources to rural campuses. Researchers in the larger campuses express concern about “access to” rural communities. Increasing exposure of climate change research in rural communities may attract and retain students there. There are key individuals in rural campuses and communities that can help catalyze productive interaction.

42. Structural and Other Issues Relevant to Future EPSCoR Importance of support of EPSCoR at all levels of administration and management. In smaller institutions it is not always obvious how to best benefit from EPSCoR. Administrative cultures constitute barriers to effective participation in and management of EPSCoR. Ambitious efforts to cross disciplines and institutions require strong institutional support. Student network development across institutions can also assist in this.

43. Questions and Comments