1. Evaluating the impact of research projects in tertiary learning and teaching: exploring the geography of change Plenary Address, HERDSA Conference, Auckland 1:45 –2:30 pm, Tuesday 2 nd July 2013 Kirsty Weir & Peter Coolbear (Ako Aotearoa) Tilly Hinton (Office of Learning and Teaching & University of the Sunshine Coast

2. Plan of our presentation Why? – why are we doing this? Context – the impact landscape Practice - activities undertaken in Australia and New Zealand to better understand the impacts of funded learning and teaching research Reflections - what this means for researchers, institutions and funding agencies

3. Why Measure impact of research into teaching and learning? Moral purpose Addressing a value for money agenda - part of the funding landscape (PBRF, ERA, etc): Identifying ways to gain additional leverage from work already done Improving funding body processes Understanding the benefits that result from learning and teaching research Surfacing stories from project teams that could otherwise go unheard

4. Funding overview Ako Aotearoa funds evidence-based change projects with a high potential to benefit learners. There is only one, primary objective of Ako Aotearoa funding – to enhance educational outcomes for learners. The OLT funds “academics and professional staff to investigate, develop and implement innovations in learning and teaching. Grants facilitate scholarship and research into learning and teaching, and promote systemic change in the sector.”

5. What do we mean by impact? Outcomes Outputs Deliverables Research assessment Journal impact factor Value Translation Transfer Valorisation Dissemination Engagement Communication Buy-in Uptake Outreach Embedding Sustainability Sustained change Project logic Evidence-based change Evidence-based improvement Deep change Scalability Learner benefit

6. What do we mean by impact? “None of these terms is entirely satisfactory, but we lack a better vocabulary.” Levin (2004, p. 2) Louise Carter Year 2 Massey University Design student

7. “…impact occurs when research, in any of its multiple forms, makes a difference to subsequent actions that people take or refrain from taking.” (Levin 2004) impact is determined when there is a “clear demonstration of learner benefit.” (Alkema 2012) “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia” (UK Research Excellence Framework definition) “impact is about people not publications” (Stilgoe 2011)

8. So we need to recognise and manage the risks here Adopting an overly transactional model Over simplification of the change process Assumptions about causality Assumptions about generalisability

9. The Ako Aotearoa Impact Evaluation Framework (IEF) APPLIED IN NEW ZEALAND: Conversations with project teams at 6, 12 and 24 months post project completion. Focusing on: outputs impact on practice impact on learners impact on project teams Evaluative conversations placed in the context of the original project rationale and goals Applied to all projects Ako funds (including retrospectively) Influence Stories written for 15 projects APPLIED IN AUSTRALIA: A pilot of the Ako approach all aspects but for a random sample of 18 projects Conversations with project leaders/teams once only, focussing on: –Changes in practice –Student benefits –Impact on the project team –Future plans –Deliverables Applied to a random sample of projects across all grant types Influence Summaries written for 18 projects

10. Why post-project conversations? Conversations surface unrecognised benefits Change processes take time, almost always beyond the final report date Relationships between projects and funding body are valuable Student learning is central Evidence-based culture Meaning is made collaboratively Opportunities for further benefits can be unearthed through the conversation An opportunity to reflect and be heard

11. Why stories or case studies? Case studies address the causality problem by providing context and identifying:  Enablers  Barriers  Other change drivers As case studies develop, they tell us about the directions of travel and differing time-frames of change Projects are taken on their own terms, rather than applying metrics or indicators “stories” “narratives” “case studies” “summaries”

12. What this means for researchers A research plan which moves to implementation of findings Understand climate of readiness for change, engagement, and transfer Willingness to tackle tough change agendas Move from requesting to dollars to selling an opportunity for change/benefit Recognition from funders that change takes time and is ongoing

13. What has funding made possible? Online graduate attributes system deeply embedded across an entire school Education students supported to develop resilience, a factor in retaining staff in the teaching profession Report cited as key evidence for a wage increase across the profession Emergent academics have ongoing access to mentorship and peer support Almost three quarters of a million dollars of industry and philanthropic funds secured to further the work started by the grant Special issues of journals that further explore themes of the grant Publications from a grant cited fifty-eight times in other publications Work integrated learning experience changed for all students in the discipline after project negotiated a change in accreditation standards

14. What does this mean for learners?

15. Ako Aotearoa National Projects – the running score sheet: 14 of 15 projects have had identifiable impact on practice and / or learners. In total these have had an: impact on the practice of 2,400 staff 50,000 students have been offered enhanced learning opportunities

16. What this means for institutions Processes for tracking and enabling impact Understand, lever and strategically request funding from funders A longer term investment, projects only notionally “finish” on submission of the final report Build a culture focussed on impact Aspects could be scaled into institutional funding models

17. What this means for funding agencies Ako Aotearoa Government endorsement The Impact Evaluation Conversations are interventions in their own right Further leverage by Ako Aotearoa and project teams to enhance impact Office for Learning and Teaching The pilot during 2012-13 has been well-received in the sector. The model is not scalable directly into the Australian context. Work is underway to translate the strengths of the approach into a scalable set of strategies

18. Ako Aotearoa has established rubrics for the assessment of future proposals

19. Mapping the complexity of impact “Necessarily then the academic community ought move towards a critical map of impact networks via a cartography of impact(s).” (Watermeyer, 2012) Massey University engineering design students

20. Peter Coolbear P.Coolbear@massey.ac.nz www.akoaotearoa.ac.nz Tilly Hinton tilly.hinton@innovation.gov.au tilly.hinton@usc.edu.au @matilda_marie http://sayat.me/TillyHinton http://olt.gov.au/secondment- projects