REF 2021 What we know and thought we knew, in preparation for the next Research Excellence Framework Dr. Tim Brooks, Research Policy & REF Manager, RDCS VFI, 19 January 2016

About the REF A periodic national assessment of research activity which: Provides benchmarking information Ensures public accountability for investment in research and its benefit Enables the selective allocation of recurrent research grant (QR)

What was the REF? A process of expert review which assesses research activity – outputs, impact, environment – over a period of c. 5 years Organised by discipline, into 36 ‘sub panels’ covering a ‘Unit of Assessment’ and four ‘Main Panels’ to ensure consistency Quality profiles rating research

Submissions Institutions decided if to participate, and which UoA(s) to submit to. Academic staff in post on the census date the key building block of each submission Employment details Any relevant personal circumstances

Research outputs Outputs of research published during the REF assessment period Selected for submission on the basis of quality Up to four outputs for every member of staff, reduced where personal circumstances justify it Worth 65% of the final mark

Impact Impacts realised during the assessment period, underpinned by research done by staff in the submitting institution Worth 20% of the final mark

Environment Data about doctoral awards and research grant income Narrative statement describing the research environment, strategies and plans, structures and support. Worth 15% of the final score

Confidentiality Panel members and other staff bound by confidentiality agreements Arrangements in place to allow institutions to partly or fully redact any element of any submission including case studies Arrangements to have case studies assessed by individuals with national security clearance

The next REF? Three weeks in November 2015: Green Paper Comprehensive Spending Review Nurse Review And then the Stern Review …

The next REF? Rules and regulations not likely to appear before spring 2017, and more likely summer 2018. In most cases little or no change e.g. UoA organization, staff eligibility, output requirements, environment markers. But – possibility of submitting all eligible staff, and this might change links to outputs.

The next REF? Open Access All journal articles and conference contributions accepted for publication on or after 1 April 2016 must be made available via an open access repository (e.g. ARRO) between acceptance and 3 months post publication. From 1 April 2017, policy tightens to within 3 months of acceptance. Additional credit for exceeding, and supporting the exceeding, of the basic requirements.

The next REF? Metrics Recent independent review of metrics usage in research assessment Confirms primacy of peer review BUT encourages judicious increase in metric usage where appropriate Green Paper proposes an interim metrics-based assessment?

The next REF? Timetable Spring 2017/Summer 2018: first guidance published 31 July 2020: end of environment & impact assessment periods (impact of research from 1 January 2000) 31 October 2020: staff census date 27 November 2020: submission deadline 31 December 2020: last date for publications

The next REF? Institutional strategy Go forward with the 15 UoAs submitted to REF2014 Add additional UoAs where appropriate Submit more staff (CP - 55% eligible staff auditable by 2017) Take steps to support quality First ‘stocktake’ during 2016 including impact cases Events and other activities to drive preparations forward

Impact for REF 2014 Worth 20% of the final score An impact template (20%) describing how the submitting unit had enabled impact from research during the assessment period Case studies (80%) giving examples of specific impacts realised during the assessment period, underpinned by research, at least 2* in quality, produced by the submitting unit in the previous 21 years Number of case studies determined by FTE of staff in the submission.

The next REF? Impact Seen as a resounding success in REF 2014 Worth 25% next time? Or 40%? Abandon ‘impact template’ document or add into research environment requirements? Increase number of case studies required per FTE? ‘Reuse’ of previously submitted case studies?

Defining Impact “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia” Impacts within HE excluded unless they extend significantly beyond the submitting HEI.

Case Studies (1) Summary: a brief introduction to the specific impact(s) being described Underpinning research: Details of the key research insights or findings, and details of what research was undertaken, when and by whom References to the research, demonstrating ≥ 2* quality

Case Studies (2) Details of the impact: and explanation of how the research made a distinct, material contribution to the impact; and the nature and extent of that impact. Sources to corroborate the impact: details of reports, reviews, individual users or other beneficiaries supporting the claims made

Assessment Threshold test: is the quality of underpinning research at least 2*? Impact case studies scored on their significance and reach Context of the impact all important.

RAND findings Links between underpinning research and its contribution to the impact must be very clear It shows if the author doesn’t believe in the value of what they’re saying Assessors had to take a lot of the content on face value; audit requests more likely if they were unconvinced by language or had prior knowledge of the impact claimed The case study says everything it must – assessors will not always follow links

Challenges Understanding what constitutes eligible impact Identifying good examples of impact arising from our research Locating sufficiently robust evidence retrospectively … but challenges that can be overcome

Enabling Impact Who might benefit from the research? How might they benefit from the research? What can be done to communicate the outcomes of the research and engage with potential beneficiaries? What resources will be needed to support this?

Next steps Do excellent research, and publish it Embed impact in your activities Identify synergies, and collaborate, internally and externally, inside and across disciplines Develop and follow strategies to support impact and develop the research environment Supervise doctoral students to completion, and win grants

More information REF 2014 rules & regulations: www.ref.ac.uk/pubs/ REF 2014 results & submissions: results.ref.ac.uk REF impact case study database: impact.ref.ac.uk REF evaluation reports: http://www.hefce.ac.uk/rsrch/REFreview/ RCUK Pathways to Impact guide: www.rcuk.ac.uk/innovation/impacts/

Thanks for your attention Any questions? Dr. Tim Brooks, Research Policy & REF Manager, RDCS tim.brooks@anglia.ac.uk