Presentation is loading. Please wait.

Presentation is loading. Please wait.

Assessing Research Impact

Similar presentations

Presentation on theme: "Assessing Research Impact"— Presentation transcript:

1 Assessing Research Impact
Dr Ian McMahon

2 What is Impact? Definition for the Research Excellence Framework [UK]
An effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia Includes an effect, change or benefit to: The activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding Of an audience, beneficiary, community, constituency, organisation or individuals In any geographic location whether locally, regionally, nationally or internationally Excludes impacts on research or the advancement of academic knowledge within HE; and impacts on teaching or other activities within the submitting HEI

3 Types of Impact Economic Social Public Policy and services Health
Cultural Quality of Life Environment

4 Why Assess Impact? Advocacy Accountability to taxpayers
Funding allocation

5 Advocacy Demonstrate the importance of university research
increasing productivity and wellbeing engaging effectively with current and future national and global challenges.   focus on showcasing or measuring the societal benefits from research,

6 Accountability to taxpayers
Provide assurance to Government and general public of the return from their investment in research   Increasingly important in a tight fiscal government environment where returns on investment in research will need to be demonstrated in terms of environmental, economic and social impact.

7 Funding Allocation UK Research Evaluation Fund Australia possibility of funding some research block grants

8 Concerns Measuring impact is not easy Attribution or contribution
Time lags between research and impact Research user engagement Transaction costs of impact assessment That some impact is negative That all research must have impact That impact is only economic Demonstrating impact in the arts and humanities

9 Concerns That impact cannot be ‘measured’
That the requirement to demonstrate it distorts academic behaviour That it takes time for impact to happen That the expectation of impact is a threat to academic freedom That impact will become an industry That measures will become targets

10 Impact Context Research Quality Framework preparation in Australia Excellence in Research for Australia 2010, 2012 Impact and UK REF – pilot in 2010, full REF in 2014 RAND Corporation: Capturing Research Impacts – A review of international practice. (for HEFCE) OECD projects – 5% funding to demonstrate impacts Economic Impact of Research & Commercialisation at Leiden University.

11 UK RESEARCH AND IMPACT Research Excellence Framework (REF) requires impact case studies (1 per every 10 academics submitted) and an impact strategy for each unit of assessment submitted. Impact judged on significance and reach. REF will be used at allocate UK Quality-related research block grant funding. Research Councils UK (including Arts and Humanities Research Council) requires applicants to submit a ‘pathways to impact’ statement

12 Australian Context Australian Government 2011 Focusing Australia’s Publicly Funded Research Review recommended a feasibility study on possible approaches for assessing the wider benefits arising from publicly funded research. 2013 New Government with greater emphasis on research with immediate benefit

Options for university-based research impact assessment To provide evidence for public investment in research To increase incentives for universities/end-user engagement improve understanding of pathways between research and innovation to inform policy. Findings: feasible to introduce two new research impact mechanisms University assessment based on case studies and metric data, integrating data from government, research agencies and universities.

14 AUSTRALIAN REPORTS Research skills for an innovative future
Smarter Manufacturing for a smarter Australia Review of Collaborations between the Public and Private Sectors Review of Higher Education Access and Outcomes for Aboriginal and Torres Strait Islander People The Place of Science in Policy Development in the Public Service

15 RELATED STUDIES Excellence in Innovation for Australia
The Conversation: public engagement metrics IP Australia: patent-based metrics University of Melbourne: STAR metrics Higher Education Funding Council of England: Research Excellence Framework Research Councils UK: pathways to impact. UNSW: income from Technology Transfer Cooperative Research Centres: Impact Study CSIRO: Impact 2020

16 Australian Research Council Definitions and Principles
Research impact is the demonstrable contribution that research makes to the economy, society, culture, national security, public policy or services, health, the environment, or quality of life, beyond contributions to academia. Impact Measurement and operational principles. Respect the diversity in research disciplines/sectors in demonstrating research impact.

17 ATN – Go8 Excellence in Innovation for Australia (EIA) Trial
There are compelling stories that need to be told of research impact arising from research at Australian universities. EIA arose out of a commitment shared by the ATN and the Go8 – in the national interest - to produce research that is both excellent in and of itself and also benefits the broader Australian community. The excellence of research produced by Australian universities has been assessed by the Excellence in Research for Australia (ERA) EIA seen as being potentially complementary to ERA in assessing the impact of research

18 The EIA in a nutshell A trial assessing the impact of university research by case studies Conducted over 2nd half of 2012 – finished yesterday 12 universities, 162 case studies across Defence, Economic Development, Society, and Environment 7 assessment panels – 70% non-academic membership Main conclusions: Can assess impact across a broad range of disciplines using case studies Universities produce high-impact research

19 Context of the Trial RQF preparation in Australia 2005 - 2007
ERA: Trial 2009, v , v Recognise work in UK REF – pilot in 2010, full REF in 2014 RAND Corporation: Capturing Research Impacts – A review of international practice. (for HEFCE) OECD projects – 5% funding to demonstrate impacts DIISRTE feasibility study into assessment of impact ATN/Go8 Symposium in November, 2011

20 Who was involved 12 universities ATN: UniSA, Curtin, UTS, RMIT, QUT
Go8: UQ, UNSW, Melbourne, UWA Newcastle, UTas, CDU VCs, DVCs-R, Research Directors + many more Advisory Board: Phil Clark, Robin Batterham, Patricia Kelly, Laurie Hammond, Craig Roy, David Sweeney + uni representatives 75 assessment panellists – 53 non-academics including from Rio Tinto, Chevron, IBM, Microsoft, Salvation Army, CSIRO, Deloitte, KPMG, … Australian Government – supporting assessment panel process Lots of researchers

21 General principles of the EIA
That the development of the EIA should look to leverage the work performed in developing the UK REF where appropriate That the EIA examine retrospective impact rather than prospective impact That there be no explicit excellence threshold for submissions, but the impact should show a link to underpinning research That the EIA take a verifiable approach to evidence in submissions That impact will be primarily assessed by means of case- studies for all disciplines. That research underpinning a submitted impact should be attributed to the university (ies) at which the research was generated

22 Guidelines for the Trial
Four broad sectors, Defence, Economic Development, Society, Environment Time period – impacts from 1st January 2007, underpinning research from 1st January, 1992 Unit of evaluation – the impact itself, not discipline, academic unit or individual A maximum of five case studies for each of the four Sectors (i.e. 20 in total) A minimum of two case studies in three of the four Sectors (i.e. 6 in total)

23 The mechanics of submissions
Case Studies on prescribed templates ≤ 15 pages Information sought on Context Details of the impact Underpinning research and research publications – submissions to draw the link to research quality Income and other relevant metrics Validation material available People involved in research and impact

24 The assessment of submissions
Each case study rated A-E based on reach & significance of impact A = outstanding impact B = very considerable impact C = considerable impact D = recognised impact E = limited impact 7 assessment panels based on broad SEO code categories 1 x Defence 2 x Economic Development 2 x Society 2 x Environment

25 Submissions

26 EIA outcomes

27 The 9 Key Learnings of the EIA
There are compelling stories that need to be told of research impact arising from research at Australian universities. Using a case study methodology with SEO codes to assess research impact is applicable as a way forward to a national assessment of research impact – with provisos The use of expert Panels for the impact assessment process, with significant input from representative external panel members, is applicable for a national impact assessment process. The range of impact ratings was wider than anticipated. Many case studies put forward cases based on prospective impact rather than demonstrated impact

28 The 9 Key Learnings of the EIA
The link between impact and good quality research was able to be identified. It was appropriate to assess impact using the industry- based grey literature. Panel Chairs all commented on the time involved assessing case studies Universities reported challenges in putting together case studies.

29 What the EIA has demonstrated
High quality research carried out in Australian universities has had enormous benefits for the health, security, prosperity, cultural and environmental wellbeing of Australia, the region and the world. Approximately 87% of the case-study assessments demonstrated at least ‘considerable’ impact. The case-study method adopted in the Trial enables these benefits to be communicated to governments, business, and the community. The combination of end-user assessment informed by expert advice has made it possible to verify that the underpinning research has contributed to the outcomes.

30 Conclusions of the EIA It is possible to assess research impact across a broad range of disciplines. The case study approach can provide a compelling narrative of the impact of research. Research impact could be assessed against an outcomes based system of classification such as the ABS SEO codes – with provisos Expert Panels comprising a majority of end-user stakeholders are able to assess research impact. Development of an impact component of any broader research assessment exercise would require further consideration of the number of case studies to be submitted.

31 Further information including final report

32 Review of EIA BY RAND (Europe)
Review of Trial documentation A critical review of 10 case studies An assessment of Panel scoring Surveys of ATN universities Findings A success but could be improved Consider “contribution” as an assessment criteria A need to create a culture of impact Scale up issues Amount of money needed for national assessment Liked concise documentation, SEO Codes Engage with non-Trial universities

33 Review of EIA BY RAND (Europe)
Positive outcomes from Trial (researchers’ perspective) Understanding the impact/effect of their research Ability to adapt material for other purposes Appreciating need to collect ongoing evidence Consolidating networks through re-establishing contacts with clients and end-users

34 Future of Impact Assessment in Australia
discussion paper on assessing the wider benefits arising from university-based research released June 2013. The National Research Investment Plan actions to maximise public benefits that arise from publicly funded research Case studies vs metrics Some universities starting Impact stocktakes New Government Would it result in extra funding or help maintain existing funding?

35 Director, Research Group of Eight
Thank You Ian McMahon Director, Research Group of Eight

Download ppt "Assessing Research Impact"

Similar presentations

Ads by Google