1. Wednesday 27th June 2018
12pm - 1.30pm
Venue: Kedleston Road-Sports Centre- Field Meeting Room
REF Impact Case Studies: A session for researchers and research leaders
Dr Tracey Wond
In this session, we will begin to explore how impact can be understood, managed and measured in the REF case study context. Drawing upon prior case studies, impact evaluation approaches and participants’ prior experiences, we will examine the nature of impact and approaches to capturing it.
The session should appeal to any researchers looking to develop their own research impact as well as research leaders developing (and supporting others to develop) REF case studies.
Refreshments will be provided.
Limited to 20 places.
REF Impact Case Studies For researchers and research leaders Dr Tracey Wond

2. DR TRACEY WOND
Joint UoA lead 17 REF2021
UoA lead 19 REF2014
TODAY PART 1 - Understand REF impact requirements PART 2 - Exploring impact PART 3 - Consider how we might evaluate and evidence REF Impact - Learn from prior case studies (REF2014)

3. Introducing REF REF = Research Excellence Framework
Subjects are divided into Units of Assessment (UoAs).
UoAs are clustered into panels and each panel issues its own guidance on what research excellence assessment looks like (including impact).
Sub-panels representing UoAs work into these main panels.
REF supports institutions to draw down QR funding (quality related funding). Provide accountability.
Impact introduced in REF2014, to assess wider impact of research.
Impact presented through case studies. These are then expertly reviewed.
The case study is not about showcasing all of the research in a unit.

4. Introducing REF Impact
Impact increased from REF2014
Reflects growing government priority for research that makes a difference. E.g. UKRI ambition – ‘economic impact’, ‘social impact’. Steven Hill, Head of Research Policy, HEFCE – ‘Delivering benefits to society must remain at the heart of the research endeavour’.
In REF2014, panel guidance further outlined and exemplified impact. For REF2021, update on impact expected in summer 2018.
Impact eligible in the institution(s) it was conducted.
Impact within and beyond the submitting institution (teaching impact)
Underpinned with at least 2 star ‘excellent’ research produced between 01/01/00 and 31/12/20.
Impacts to have occurred 01/08/13 and 31/07/20.
Assessed by research and significance (further update expected)
Able to continue or develop a REF2014 case study (but show additionality)
In REF2014 this was Impact: 20%; Outputs: 65%: Env: 15%
UK Research and Innovation (UKRI) explicitly prioritises research as supporting economic and social impact.
The REF2021 methodology is still emerging and updates and clarifications have been released gradually. In relation to impact we will know more about impact in summer 2018.
See Initial Decisions document: :
Impact
A key recommendation of the Stern review was to ensure the REF could better capture the multiple and diverse pathways and mechanisms through which impact arises from a body of work, and through which real benefits to the UK and wider world are delivered. This aim was widely supported by respondents to the consultation, and the funding bodies will seek to implement this in the exercise, as follows:
We will work with the panels to provide additional guidance on:
The criteria for impact of ‘reach and significance’.
Impact arising from public engagement.
The guidance on submitting impacts on teaching will be widened to include impacts within, as well as beyond, the submitting institution. We will also work with the panels to develop appropriate guidance on demonstrating evidence against the criteria for this type of impact.
Also decision to ‘harmonise’ definition of impact with research councils.

5. Defining REF Impact
In REF 2014, Impact was defined as:
“an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia”
Panel criteria further articulate.
In REF2014 this was Impact: 20%; Outputs: 65%: Env: 15%
UK Research and Innovation (UKRI) explicitly prioritises research as supporting economic and social impact.
The REF2021 methodology is still emerging and updates and clarifications have been released gradually. In relation to impact we will know more about impact in summer 2018.
See Initial Decisions document: :
Impact
A key recommendation of the Stern review was to ensure the REF could better capture the multiple and diverse pathways and mechanisms through which impact arises from a body of work, and through which real benefits to the UK and wider world are delivered. This aim was widely supported by respondents to the consultation, and the funding bodies will seek to implement this in the exercise, as follows:
We will work with the panels to provide additional guidance on:
The criteria for impact of ‘reach and significance’.
Impact arising from public engagement.
The guidance on submitting impacts on teaching will be widened to include impacts within, as well as beyond, the submitting institution. We will also work with the panels to develop appropriate guidance on demonstrating evidence against the criteria for this type of impact.
Also decision to ‘harmonise’ definition of impact with research councils.

6. ‘Underpinning Research’
Extracts from REF2014.
REF2021 not available yet!

7. Nature of Impact Witnessed UoAs getting this wrong in REF2014
Outputs usually come after activities. Here it appears they are trying to demonstrate outputs as actual research outputs – i.e. underpinning research.
Diagram source: in Collecting Research Impact Evidence: Best Practice Guidance for the Research Community
Some sub panel chairs have reported that they didn’t see impact in some cases. Exploring past case studies, available publicly this is also observed.
Kelloggs’ Foundation Logic Model is commonly recommended to understand more about logic modelling and distinguish inputs, outputs, activity, outcomes and impact.
Number of tweets indicates activity as opposed to impact.
Witnessed UoAs getting this wrong in REF2014
Note doing things (activities) is not the same as impact.
REF panel chairs want to see distinction shown between activities and impact.

8. Logic Modelling Plots how your activity is meant to work
Within evaluation we explore logic models a lot, and tend to start off with them.
Logic models are popular in evaluation practice;
It plots how your activity is meant to work and can look similar to the impact arrow on the previous page;
With the addition of context and rationale a theory can be built on how impact should happen
If your case study only covers up to here it may be worth reviewing

9. Attribution – Making the Link
Growing demand for quantitative data.
Make the link between the underpinning research, the impact, and the impact evidence
Clarity about how the impact occurred.
Specific evidence – clearly support the impact and narrative.

10. CMO – A Realist Evaluation Approach
CONTEXT
MECHANISM
OUTCOMES
What is the context in which the impact occurs and what considerations does this give rise to?
Consideration for setting (e.g. entrepreneurs, SMEs, regional business support provision, policy etc.)
Consideration for other features of the context of the impact – reach, novelty, type of organisation, location.
Consideration for intention for change/impact which will influence the evaluative research questions, e.g.:
To what extent did team y’s work on entrepreneurial learning influence the way in which business support was delivered (in the regions targeted, and beyond)?
How was the change effected? Using what mechanisms?
This might be planned or unplanned. If planned then it may be worth revisiting the ‘theory of change’.
Mechanisms might include outputs such as:
Satisfaction with new approach
No. attending session
No. of retweets a post gained.
What impact was created and how can this be evidenced?
REF usually provide guidance on what impact may look like.
Evidence that the research led to the change.
Where possible with consideration for causality – linking the research intervention with the impact.
Outcomes
Mechanism
Context
Flexibility acknowledged in REF documentation;

11. Impact Evidence- Types
Collecting Research Impact Evidence: Best Practice Guidance for the Research Community
Flexibility acknowledged in REF documentation;
See handout – ‘Guidance on the types of evidence that could be collected’
Source: Vertigo Ventures and Digital Science (2016)

12. Impact Evidence – REF2014 analysis
Collecting Research Impact Evidence: Best Practice Guidance for the Research Community
Recognise variations across subject areas/Panels.
Analysis by Vertigo Ventures and Digital Science (2016)

13. Tips for Impact See handout – ‘Research Impact Process’
See handout – ‘Research Impact Process’

14. Further Resources
University of East Anglia: Impact Case Study Packs available at:
tudy%20pack&gsc.sort=
University of Bath: Impact toolkit: