page 1 RESEARCH EXCELLENCE FRAMEWORK : RESEARCH IMPACT ASESSMENT LESSONS FROM THE PILOT EXERCISE Professor John Marshall Director Academic Research Development

page 2 “We live in a moment of history where change is so speeded up that we begin to see the present only when it is already disappearing” R D Laing, Scottish Psychiatrist ( ) Thought for the day:

page 3 “Research Impact is as big a change in REF as the introduction of the original RAE was for HEIs”.

page 4 REF2014 – What we know 36 REF sub Panels (UoAs) down from 68 in RAE2008 REF2014 guidance document to be published in July 2011 Sub Panel Guidance document to be published in January 2012 Panel Chairs and Sub Panel members have been announced The method of assessment has been changed to include the a new measure relating to the impact of excellent research (2* or above)

page 5 Research Excellence Framework (REF2014) Assessment Criteria and Weightings Outputs (65%) Environment (20%) Impact (20%) REF emphasis is on Research Excellence. Only highest quality levels (3* and 4*) will attract funding post Contribution of research group activity over 15 year timescale important for impact Impact case studies : 1 per UoA plus 1 per 10 FTE

page 6 REF Research Impact Decisions In the REF there will be an explicit element to assess the 'impact' arising from excellent research, alongside the 'outputs' and 'environment' elements. Submissions will also include information about how the unit has supported and enabled impact during the assessment period.

page 7 The assessment of impact will be based on expert review of case studies submitted by higher education institutions. Case studies may include any social, economic or cultural impact or benefit beyond academia that has taken place during the assessment period, and was underpinned by excellent research produced by the submitting institution within a given timeframe. Requires auditable evidence of impact. REF Research Impact Decisions

page 8 REF2014 Research Impact Pilot Study The REF team ran a pilot exercise which aimed to test the feasibility of assessing research impact, and to develop the method of assessment for use in the REF. The pilot exercise was undertaken in five units of assessment (UOAs): Clinical Medicine Physics Earth Systems and Environmental Sciences Social Work and Social Policy English Language and Literature.

page 9 All material required to make a judgement about the impact was contained within the case study and no additional information gathering was required by panel members. Clear identification of who the beneficiaries were or which groups/organisations had changed something as a result of the research. This could include ‘intermediary’ organisations as well as ‘end users’ or audiences.) A coherent narrative which explained clearly the relationship between the research and the impact, and the nature of the impact or benefits arising. Identified Good practice in REF Impact Case studies (1)

page 10 Identified Good practice in REF Impact Case studies (2) Indicators that were meaningful and contextualised to support the case being made, not used as a substitute for a clear narrative. The indicators included precise information that was relevant to the case study. A brief explanation of what was original or distinctive about the research insights that contributed to the impacts. Specific details about the names of researchers, their position in the HEI and dates and locations of the research activity.

page 11 Specific and appropriate independent sources of corroborating information were provided. Focused and concise evidence, without generalised or overblown statements or unexplained lists of publications or references. Selected examples of case studies that scored highly and were considered good practice in the pilot exercise are available on the REF website at under ‘Impact pilot exercise’. Identified Good practice in REF Impact Case studies (3)

page 12 There was a clear link between the department’s research and the engagement activity. Evidence was provided about dissemination (such as audience or visitor numbers) as well as a clear explanation of the significance or benefits to the audiences. The activity went beyond ‘business as usual’ engagement (such as public lectures), and attracted widespread interest or involved a programme of activity which was often innovative and was sustainable or created ‘legacy’ resources. In some cases there were additional, clearly identified benefits to ‘intermediaries’ (such as cultural institutions or curators), local communities or to the economy. REF impact case Studies involving Engagement with the Public

page 13 Weaknesses in REF impact case studies: Generalised, vague statements provided about key claims without sufficient evidence, requiring panel members to undertake further investigation to gather information or rely on expert or prior knowledge. Lack of clarity about the link between the underpinning research and the specific impacts claimed.

page 14 Excessively long lists of unexplained publications or web references, making it laborious for panels to identify salient information. Reliance on indicators that lacked context or meaning, for example, numbers of hits on a website, without benchmarks or further contextual explanation; or relying on anecdotal evidence such as personal correspondence or quotes from individual members of the public as evidence of impact. Weaknesses in REF impact case studies (2):

page 15 1.REF guidance/preparation of REF2014 submission 2.REF research impact case study development 3.New Current Research Information System (CRIS) to support the development of the REF submission 4.New Research repository (REF publications visibility/citations) 5.New REF Researcher staff development (via CREDO) GCU PREPARATIONS FOR REF2014

page 16 Where can I find more information about REF2014 ? The timetable for REF2014 with links to other material is available at : REF link on University Research Website: nuniversity/ref2014/

page 17 PART 2: WORKSHOP EXERCISE Evaluating Impact Cases Studies from the 5 REF pilot areas: Clinical Medicine Physics Earth Systems and Environmental Sciences Social Work and Social Policy English Language and Literature Split into groups and discuss one area of interest to you (20 min) Which case studies do you think are better and why ? How would we develop case studies like these? Rejoin larger group for close out discussion reporting on best practice relevant to GCU