1. REF 2021
What we know and thought we knew, in preparation for the next Research Excellence Framework
Dr. Tim Brooks, Research Policy & REF Manager, RDCS
VFI, 19 January 2016

2. About the REF
A periodic national assessment of research activity which:
Provides benchmarking information
Ensures public accountability for investment in research and its benefit
Enables the selective allocation of recurrent research grant (QR)

3. What was the REF?
A process of expert review which assesses research activity – outputs, impact, environment – over a period of c. 5 years
Organised by discipline, into 36 ‘sub panels’ covering a ‘Unit of Assessment’ and four ‘Main Panels’ to ensure consistency
Quality profiles rating research

4. Submissions
Institutions decided if to participate, and which UoA(s) to submit to.
Academic staff in post on the census date the key building block of each submission
Employment details
Any relevant personal circumstances

5. Research outputs
Outputs of research published during the REF assessment period
Selected for submission on the basis of quality
Up to four outputs for every member of staff, reduced where personal circumstances justify it
Worth 65% of the final mark

6. Impact
Impacts realised during the assessment period, underpinned by research done by staff in the submitting institution
Worth 20% of the final mark

7. Environment Data about doctoral awards and research grant income
Narrative statement describing the research environment, strategies and plans, structures and support.
Worth 15% of the final score

8. Confidentiality
Panel members and other staff bound by confidentiality agreements
Arrangements in place to allow institutions to partly or fully redact any element of any submission including case studies
Arrangements to have case studies assessed by individuals with national security clearance

9. The next REF? Three weeks in November 2015: Green Paper
Comprehensive Spending Review
Nurse Review
And then the Stern Review …

10. The next REF?
Rules and regulations not likely to appear before spring 2017, and more likely summer 2018.
In most cases little or no change e.g. UoA organization, staff eligibility, output requirements, environment markers.
But – possibility of submitting all eligible staff, and this might change links to outputs.

11. The next REF? Open Access
All journal articles and conference contributions accepted for publication on or after 1 April 2016 must be made available via an open access repository (e.g. ARRO) between acceptance and 3 months post publication.
From 1 April 2017, policy tightens to within 3 months of acceptance.
Additional credit for exceeding, and supporting the exceeding, of the basic requirements.

12. The next REF? Metrics
Recent independent review of metrics usage in research assessment
Confirms primacy of peer review BUT encourages judicious increase in metric usage where appropriate
Green Paper proposes an interim metrics-based assessment?

13. The next REF? Timetable
Spring 2017/Summer 2018: first guidance published
31 July 2020: end of environment & impact assessment periods (impact of research from 1 January 2000)
31 October 2020: staff census date
27 November 2020: submission deadline
31 December 2020: last date for publications

14. The next REF? Institutional strategy
Go forward with the 15 UoAs submitted to REF2014
Add additional UoAs where appropriate
Submit more staff (CP - 55% eligible staff auditable by 2017)
Take steps to support quality
First ‘stocktake’ during 2016 including impact cases
Events and other activities to drive preparations forward

15. Impact for REF 2014 Worth 20% of the final score
An impact template (20%) describing how the submitting unit had enabled impact from research during the assessment period
Case studies (80%) giving examples of specific impacts realised during the assessment period, underpinned by research, at least 2* in quality, produced by the submitting unit in the previous 21 years
Number of case studies determined by FTE of staff in the submission.

16. The next REF? Impact Seen as a resounding success in REF 2014
Worth 25% next time? Or 40%?
Abandon ‘impact template’ document or add into research environment requirements?
Increase number of case studies required per FTE?
‘Reuse’ of previously submitted case studies?

17. Defining Impact
“an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia”
Impacts within HE excluded unless they extend significantly beyond the submitting HEI.

18. Case Studies (1)
Summary: a brief introduction to the specific impact(s) being described
Underpinning research: Details of the key research insights or findings, and details of what research was undertaken, when and by whom
References to the research, demonstrating ≥ 2* quality

19. Case Studies (2)
Details of the impact: and explanation of how the research made a distinct, material contribution to the impact; and the nature and extent of that impact.
Sources to corroborate the impact: details of reports, reviews, individual users or other beneficiaries supporting the claims made

20. Assessment
Threshold test: is the quality of underpinning research at least 2*?
Impact case studies scored on their significance and reach
Context of the impact all important.

21. RAND findings
Links between underpinning research and its contribution to the impact must be very clear
It shows if the author doesn’t believe in the value of what they’re saying
Assessors had to take a lot of the content on face value; audit requests more likely if they were unconvinced by language or had prior knowledge of the impact claimed
The case study says everything it must – assessors will not always follow links

22. Challenges Understanding what constitutes eligible impact
Identifying good examples of impact arising from our research
Locating sufficiently robust evidence retrospectively
… but challenges that can be overcome

23. Enabling Impact Who might benefit from the research?
How might they benefit from the research?
What can be done to communicate the outcomes of the research and engage with potential beneficiaries?
What resources will be needed to support this?

24. Next steps Do excellent research, and publish it
Embed impact in your activities
Identify synergies, and collaborate, internally and externally, inside and across disciplines
Develop and follow strategies to support impact and develop the research environment
Supervise doctoral students to completion, and win grants

25. More information REF 2014 rules & regulations: www.ref.ac.uk/pubs/
REF 2014 results & submissions: results.ref.ac.uk
REF impact case study database: impact.ref.ac.uk
REF evaluation reports:
RCUK Pathways to Impact guide:

26. Thanks for your attention
Any questions?
Dr. Tim Brooks, Research Policy & REF Manager, RDCS