1. Demonstrating research impact in the REF Graeme Rosenberg REF Manager

2. Outline:
What is the REF?
What is impact?
How should it be demonstrated?
How will it be assessed?

3. What is the REF?
The REF is the new framework for assessing research in all disciplines in all UK higher education institutions
It replaces the RAE
Its purpose is:
To inform research funding allocations by the four UK HE funding bodies (approximately £2 billion per year)
Provide accountability for public funding of research and demonstrate its benefits
To provide benchmarks and reputational yardsticks

4. The assessment framework
Overall quality
Outputs
Maximum of 4 outputs per researcher
Impact
Impact template and case studies
Environment
Environment data and template
65%
20%
15%

5. Submissions There are 36 discipline-based units of assessment (UOAs)
Each institution decides which UOAs to submit in
Each submission in a UOA must contain a standard set of information:
Staff details (REF1a/b/c)
Research outputs (REF2)
Impact template and case studies (REF3a/b)
Environment data (REF4a/b/c)
Environment template (REF5)

6. Assessment
Submissions will be assessed by 36 expert sub-panels working under the guidance of 4 main panels
Panels will carry out the assessment according to the published criteria and working methods
Sub-panel responsibilities
Contributing to the panel criteria and working methods
Assessing submissions and recommending the outcomes
Main panel responsibilities
Developing the panel criteria and working methods
Ensuring adherence to the criteria/procedures and consistent application of the overall assessment standards
Signing off the outcomes

7. Example of a quality profile

8. Timetable Criteria phase 2011 Submissions phase 2012-13
Develop and publish Guidance on submissions (Jul)
Develop, consult on and publish Panel criteria (Jan 2012)
Submissions phase
HEIs submit Codes of practice
Launch the REF submissions system
Submission deadline Nov 2013
Assessment phase 2014
Panels assess submissions
Publish outcomes Dec 2014

9. What is impact? Impact is defined broadly for the REF:
an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia
The REF definition of impact is retrospective – benefits or changes that have already occurred, not future potential
Impacts can be manifest in a wide variety of ways, may take many forms and occur in a wide range of spheres, in any geographic location

10. Examples of types of impact
Enhanced professional standards, ethics, guidelines or training
Improved health or welfare outcomes
Improved risk management
Public debate has been shaped or informed by research
More effective management or workplace practices
Improved business performance
Improved quality, accessibility or efficiency of a public service
A social enterprise initiative has been created
Research has enabled stakeholders to challenge conventional wisdom
Production costs have reduced
Changes to the design or delivery of the school curriculum
Improved forensic methods or expert systems
Enhanced preservation, conservation or presentation of cultural heritage
Improved access to justice, employment or education
Policy debate or decisions have been influenced or shaped by research
Improved management or conservation of natural resources
Jobs have been created or protected
Research has informed public understanding, values, attitudes or behaviours
Organisations have adapted to changing cultural values
The policies or activities of NGOs or charities have been informed by research
Changes to legislation or regulations
Enhanced corporate social responsibility policies
Levels of waste have reduced
Changes in professional practice
Enhanced technical standards or protocols
A new product has been commercialised
New forms of artistic expression or changes to creative practice

11. Examples from the pilot exercise
Improved treatment and quality of life for patients with type 1 diabetes
Detector technology used to screen liquids at airports
Conservation of bumblebees
Influencing policy on child maintenance and better service for separated parents
Enriched public appreciation of heritage at Hampton Court Palace
Pilot examples are published at:
an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia
Impacts can be manifest in a wide variety of ways, may take many forms and occur in a wide range of spheres, in any geographic location
Panels provide examples of impact relevant to their disciplines, intended to be helpful - not as exhaustive or prescriptive lists

12. Demonstrating impact Specific examples of impacts already achieved
Impact template
Sets out the submi tted unit’s appro ach to impac t
Case studies
Specific examples of impacts already achieved
20% of the impact sub-profile
80% of the impact sub-profile

13. Case studies
One case study must be submitted per 10 FTE staff (with a minimum of two cases in a submission)
Impacts that took place during 2008 to 2013; underpinned by research since 1993
Submitted case studies need not be representative of activity across the unit: pick the strongest examples
Case studies must be completed on a template - maximum 4 pages

14. The case study template
Summary of the impact
Underpinning research:
What was it? Who undertook it and when?
What were the key insights or findings?
References to the research
Details of the impact:
How did the research lead/contribute to the impact?
Who were the beneficiaries?
How did they benefit?
Sources of corroboration

15. Lessons from the pilot exercise
All the material required to make a judgement should be included
The narrative should be coherent and clearly explain the ‘links in the chain’
Clear definition of who benefitted, and what had changed
Indicators should be meaningful and contextualised
Key claims should be capable of verification
Where the impact arises from public engagement:
How was the engagement activity based on the research?
Dissemination in itself is not impact – what was the benefit?

16. Preparing impact submissions
HEIs seem to be at varying stages of:
Briefing staff
Identifying potential cases
Drafting case studies
Some issues seem to be:
How to write compelling narrative, and who should be involved
Evidence gathering – retrospective and from many sources
Institutional memory – what if the staff have left?
Second guessing the panels to decide on the strongest cases

17. How will impacts be assessed?
Sub-panels will assess impacts against the criteria of reach and significance:
A holistic judgement about each case study
‘Reach’ is not a geographic scale.
Each case study will be examined by at least one academic and one ‘user’ on the panel
Panels will make judgements on the merits of the evidence provided – not further investigations
A sample of case studies will be audited to corroborate or verify information

18. Underpinning research
Panels will assess the impact, not the underpinning research
But each case study must be underpinned by research that satisfies the following criteria:
produced by staff while working in the submitting HEI
generated outputs published between 1 Jan 1993 and 31 Dec 2013
meets the quality threshold of at least equivalent to 2*
made a material and distinct contribution to the impact

19. The criteria for assessing impacts are reach and significance*
Assessment criteria
The criteria for assessing impacts are reach and significance*
Four star
Outstanding impacts in terms of their reach and significance
Three star
Very considerable impacts in terms of their reach and significance
Two star
Considerable impacts in terms of their reach and significance
One star
Recognised but modest impacts in terms of their reach and significance
Unclassified
The impact is of little or no reach and significance; or the impact was not eligible; or the impact was not underpinned by excellent research produced by the submitted unit
* Each main panel provides further description of the criteria

20. Further information www.ref.ac.uk (includes all relevant documents)
Enquiries from staff at HEIs should be directed to their nominated institutional contact (see for a list)
Other enquiries to