2 What is Impact? Definition for the Research Excellence Framework [UK] An effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academiaIncludes an effect, change or benefit to: The activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understandingOf an audience, beneficiary, community, constituency, organisation or individualsIn any geographic location whether locally, regionally, nationally or internationallyExcludes impacts on research or the advancement of academic knowledge within HE; and impacts on teaching or other activities within the submitting HEI
3 Types of Impact Economic Social Public Policy and services Health CulturalQuality of LifeEnvironment
4 Why Assess Impact? Advocacy Accountability to taxpayers Funding allocation
5 Advocacy Demonstrate the importance of university research increasing productivity and wellbeingengaging effectively with current and future national and global challenges. focus on showcasing or measuring the societal benefits from research,
6 Accountability to taxpayers Provide assurance to Government and general public of the return from their investment in research Increasingly important in a tight fiscal government environment where returns on investment in research will need to be demonstrated in terms of environmental, economic and social impact.
7 Funding AllocationUK Research Evaluation FundAustralia possibility of funding some research block grants
8 Concerns Measuring impact is not easy Attribution or contribution Time lags between research and impactResearch user engagementTransaction costs of impact assessmentThat some impact is negativeThat all research must have impactThat impact is only economicDemonstrating impact in the arts and humanities
9 Concerns That impact cannot be ‘measured’ That the requirement to demonstrate it distorts academic behaviourThat it takes time for impact to happenThat the expectation of impact is a threat to academic freedomThat impact will become an industryThat measures will become targets
10 Impact ContextResearch Quality Framework preparation in AustraliaExcellence in Research for Australia 2010, 2012Impact and UK REF – pilot in 2010, full REF in 2014RAND Corporation: Capturing Research Impacts – A review of international practice. (for HEFCE)OECD projects – 5% funding to demonstrate impactsEconomic Impact of Research & Commercialisation at Leiden University.
11 UK RESEARCH AND IMPACTResearch Excellence Framework (REF) requires impact case studies (1 per every 10 academics submitted) and an impact strategy for each unit of assessment submitted. Impact judged on significance and reach.REF will be used at allocate UK Quality-related research block grant funding.Research Councils UK (including Arts and Humanities Research Council) requires applicants to submit a ‘pathways to impact’ statement
12 Australian ContextAustralian Government 2011 Focusing Australia’s Publicly Funded Research Review recommended a feasibility study on possible approaches for assessing the wider benefits arising from publicly funded research.2013 New Government with greater emphasis on research with immediate benefit
13 AUSTRALIAN GOVERNMENT FEASIBILITY STUDY Options for university-based research impact assessmentTo provide evidence for public investment in researchTo increase incentives for universities/end-user engagementimprove understanding of pathways between research and innovation to inform policy.Findings:feasible to introduce two new research impact mechanismsUniversity assessment based on case studies and metric data,integrating data from government, research agencies and universities.
14 AUSTRALIAN REPORTS Research skills for an innovative future Smarter Manufacturing for a smarter AustraliaReview of Collaborations between the Public and Private SectorsReview of Higher Education Access and Outcomes for Aboriginal and Torres Strait Islander PeopleThe Place of Science in Policy Development in the Public Service
15 RELATED STUDIES Excellence in Innovation for Australia The Conversation: public engagement metricsIP Australia: patent-based metricsUniversity of Melbourne: STAR metricsHigher Education Funding Council of England: Research Excellence FrameworkResearch Councils UK: pathways to impact.UNSW: income from Technology TransferCooperative Research Centres: Impact StudyCSIRO: Impact 2020
16 Australian Research Council Definitions and Principles Research impact is the demonstrable contribution that research makes to the economy, society, culture, national security, public policy or services, health, the environment, or quality of life, beyond contributions to academia.Impact Measurement and operational principles.Respect the diversity in research disciplines/sectors in demonstrating research impact.
17 ATN – Go8 Excellence in Innovation for Australia (EIA) Trial There are compelling stories that need to be told of research impact arising from research at Australian universities.EIA arose out of a commitment shared by the ATN and the Go8 – in the national interest - to produce research that is both excellent in and of itself and also benefits the broader Australian community.The excellence of research produced by Australian universities has been assessed by the Excellence in Research for Australia (ERA)EIA seen as being potentially complementary to ERA in assessing the impact of research
18 The EIA in a nutshellA trial assessing the impact of university research by case studiesConducted over 2nd half of 2012 – finished yesterday12 universities, 162 case studies across Defence, Economic Development, Society, and Environment7 assessment panels – 70% non-academic membershipMain conclusions:Can assess impact across a broad range of disciplines using case studiesUniversities produce high-impact research
19 Context of the Trial RQF preparation in Australia 2005 - 2007 ERA: Trial 2009, v , vRecognise work in UK REF – pilot in 2010, full REF in 2014RAND Corporation: Capturing Research Impacts – A review of international practice. (for HEFCE)OECD projects – 5% funding to demonstrate impactsDIISRTE feasibility study into assessment of impactATN/Go8 Symposium in November, 2011
20 Who was involved 12 universities ATN: UniSA, Curtin, UTS, RMIT, QUT Go8: UQ, UNSW, Melbourne, UWANewcastle, UTas, CDUVCs, DVCs-R, Research Directors + many moreAdvisory Board: Phil Clark, Robin Batterham, Patricia Kelly, Laurie Hammond, Craig Roy, David Sweeney + uni representatives75 assessment panellists – 53 non-academics including from Rio Tinto, Chevron, IBM, Microsoft, Salvation Army, CSIRO, Deloitte, KPMG, …Australian Government – supporting assessment panel processLots of researchers
21 General principles of the EIA That the development of the EIA should look to leverage the work performed in developing the UK REF where appropriateThat the EIA examine retrospective impact rather than prospective impactThat there be no explicit excellence threshold for submissions, but the impact should show a link to underpinning researchThat the EIA take a verifiable approach to evidence in submissionsThat impact will be primarily assessed by means of case- studies for all disciplines.That research underpinning a submitted impact should be attributed to the university (ies) at which the research was generated
22 Guidelines for the Trial Four broad sectors, Defence, Economic Development, Society, EnvironmentTime period – impacts from 1st January 2007, underpinning research from 1st January, 1992Unit of evaluation – the impact itself, not discipline, academic unit or individualA maximum of five case studies for each of the four Sectors (i.e. 20 in total)A minimum of two case studies in three of the four Sectors (i.e. 6 in total)
23 The mechanics of submissions Case Studies on prescribed templates ≤ 15 pagesInformation sought onContextDetails of the impactUnderpinning research and research publications – submissions to draw the link to research qualityIncome and other relevant metricsValidation material availablePeople involved in research and impact
24 The assessment of submissions Each case study rated A-E based on reach & significance of impactA = outstanding impactB = very considerable impactC = considerable impactD = recognised impactE = limited impact7 assessment panels based on broad SEO code categories1 x Defence2 x Economic Development2 x Society2 x Environment
27 The 9 Key Learnings of the EIA There are compelling stories that need to be told of research impact arising from research at Australian universities.Using a case study methodology with SEO codes to assess research impact is applicable as a way forward to a national assessment of research impact – with provisosThe use of expert Panels for the impact assessment process, with significant input from representative external panel members, is applicable for a national impact assessment process.The range of impact ratings was wider than anticipated.Many case studies put forward cases based on prospective impact rather than demonstrated impact
28 The 9 Key Learnings of the EIA The link between impact and good quality research was able to be identified.It was appropriate to assess impact using the industry- based grey literature.Panel Chairs all commented on the time involved assessing case studiesUniversities reported challenges in putting together case studies.
29 What the EIA has demonstrated High quality research carried out in Australian universities has had enormous benefits for the health, security, prosperity, cultural and environmental wellbeing of Australia, the region and the world.Approximately 87% of the case-study assessments demonstrated at least ‘considerable’ impact.The case-study method adopted in the Trial enables these benefits to be communicated to governments, business, and the community.The combination of end-user assessment informed by expert advice has made it possible to verify that the underpinning research has contributed to the outcomes.
30 Conclusions of the EIAIt is possible to assess research impact across a broad range of disciplines.The case study approach can provide a compelling narrative of the impact of research.Research impact could be assessed against an outcomes based system of classification such as the ABS SEO codes – with provisosExpert Panels comprising a majority of end-user stakeholders are able to assess research impact.Development of an impact component of any broader research assessment exercise would require further consideration of the number of case studies to be submitted.
32 Review of EIA BY RAND (Europe) Review of Trial documentationA critical review of 10 case studiesAn assessment of Panel scoringSurveys of ATN universitiesFindingsA success but could be improvedConsider “contribution” as an assessment criteriaA need to create a culture of impactScale up issuesAmount of money needed for national assessmentLiked concise documentation, SEO CodesEngage with non-Trial universities
33 Review of EIA BY RAND (Europe) Positive outcomes from Trial (researchers’ perspective)Understanding the impact/effect of their researchAbility to adapt material for other purposesAppreciating need to collect ongoing evidenceConsolidating networks through re-establishing contactswith clients and end-users
34 Future of Impact Assessment in Australia discussion paper on assessing the wider benefits arising from university-based research released June 2013.The National Research Investment Plan actions to maximise public benefits that arise from publicly funded researchCase studies vs metricsSome universities starting Impact stocktakesNew GovernmentWould it result in extra funding or help maintain existing funding?
35 Director, Research Group of Eight Thank YouIan McMahonDirector, Research Group of Eight