1. Email us: info@ref.ac.uk
UCML, London
18 January 2019
REF 2021
Susan Hodgett (D25)
Charles Forsdick (D26)
Follow us on Twitter
@REF_2021
us:

2. REF - overview
Undertaken by the REF team on behalf of the four UK higher education funding bodies
Key purposes:
To provide accountability for public investment in research and produce evidence of the benefits of this investment.
To provide benchmarking information and establish reputational yardsticks, for use within the HE sector and for public information.
To inform the selective allocation of funding for research –ca. £2bn per year.
Assessed across three areas:
Outputs (e.g. publications, performance, prototypes) – 60%
Impact (the benefit of research for wider society) – 25%
Research environment (resources, strategies to support research) – 15%
Assessed by expert sub-panels (of senior academics and research users) in 34 disciplined-based ‘units of assessment’, overseen by 4 main panels

3. The Stern Review
Independent review of REF commissioned by Minister and led by Lord Nicholas Stern. Reported in July 2016.
Key principles
Lower burden
Less game-playing
Less personalisation, more institutionally focused
Recognition for investment
More rounded view of research activity
Interdisciplinary emphasis
Broaden impact

4. Key changes since REF 2014
Submission of all staff with significant responsibility for research
Transitional approach to non-portability of outputs
Decoupling of staff from outputs
Additional measures to support interdisciplinary research
Broadening and deepening definitions of impact
Open access requirements
More structured environment statement
Weightings

5. Guidance on submissions
Specific consultation areas, Summer-Autumn 2018
clarity, usefulness and coverage of the list of independent research fellowships
proposed eligibility arrangements for seconded staff
proposed ineligibility of staff based in a discrete department or unit outside the UK
proposed approach for taking account of staff circumstances
clarity and usefulness of the glossary of output types
proposal to make ineligible the outputs of former staff who have been made redundant (except where the staff member has taken voluntary redundancy)
proposed intention to permit the submission of co-authored outputs only once within the same submission
PLUS comments on overall clarity of the guidance, including annexes

6. 2021 framework Overall quality Outputs Impact Environment 60% 25% 15%
FTE x 2.5 = number of outputs required
Impact
Impact case studies
Environment
Environment data and template -
60%
25%
15%

7. Expert panels
34 sub-panels working under the guidance of four main panels with advice from Equality and Diversity and Interdisciplinary Research advisory panels (EDAP and IDAP)
Two-stage appointment process (via nominations):
Criteria-setting phase – sufficient members appointed to ensure each sub-panel has appropriate expertise
Assessment phase – recruitment in 2020 of additional panel members and assessors to ensure appropriate breadth of expertise and number of panel members necessary for the assessment phase, informed by the survey of institutions’ submission intentions in 2019.

8. Submissions – summary
Each submission in a UOA provides evidence about the activity and achievements of a ‘submitted unit’
Responsibility for mapping staff into UOAs with institutions – guided by UOA descriptors
Institutions will normally make one submission in each UOA they elect to submit in
Joint submissions are encouraged where this is an appropriate way of describing collaborative research
Consistency with 2014 process for multiple submissions – only by exception and with permission from the REF manager

9. Submissions – staff
All staff with significant responsibility for research should be returned to the REF
Category A eligible
Category A submitted
- ‘Teaching and Research’ or ‘Research only’
- Independent researcher
- Minimum of 0.2 FTE
- Substantive connection
Accurately identifies staff with significant responsibility for research
100 per cent returned
Some T&R staff do not have significant responsibility for research
Staff with significant responsibility returned, following process developed, consulted on and documented
Approach may vary by UOA where employment practices vary at this level

10. Submissions – staff Significant responsibility for research
‘those for whom explicit time and resources are made available to engage actively in independent research, and that is an expectation of their job role.’
Guidance will set out a ‘menu’ of what we consider may be appropriate indicators of significant responsibility.
Independent research
Variation in key attributes also identified in the consultation
Guidance will build on generic definition used in REF 2014 (undertaking ‘independent research, leading or acting as principal investigator or equivalent on a research grant or significant piece of research work’.)

11. Submissions – decoupling
Number of outputs per submission
Output pool to include
FTE of Cat A submitted
2.5
Number of outputs
Submitted outputs
Min of 1 per Cat A submitted
Max of 5 attributed to individuals
May include outputs of staff that have left

12. Submissions - outputs Transitional approach to non-portability
Outputs may be submitted by:
the institution employing eligible staff member when the output was demonstrably generated; and
the institution employing the staff member on the census date
‘Demonstrably generated’ – date when the output was first made publicly available
Full eligibility criteria to be defined by panels

13. Outputs Rigour Originality Significance
Assessed against three criteria:
Originality
the extent to which the output makes an important and innovative contribution to understanding and knowledge in the field
Significance
the extent to which the work has influenced, or has the capacity to influence, knowledge and scholarly thought, or the development and understanding of policy and/or practice
Rigour
the extent to which the work demonstrates intellectual coherence and integrity, and adopts robust and appropriate concepts, analyses, theories and methodologies
Scored one to four star (or unclassified)
Each main panel sets out its own understanding of the starred quality levels

14. Outputs
Open access
Outputs deposited as soon after the point of acceptance as possible, and no later than three months after this date from 1 April 2018.
Deposit exception from 1 April 2018 – outputs remain compliant if they are deposited up to three months after the date of publication.
Assessment metrics
Quantitative data may be used to inform the assessment of outputs, where panels consider this appropriate for the discipline

15. Outputs – citation data
All sub-panels will use citation data (where available), as potential indicator of academic significance
Main Panel A
Sub-panels 7, 8, 9 and 11 will receive citation data (where available), as part of indicator of academic significance
Main Panel B
Sub-panel 16 will receive citation data (where available), and will use where it is considered appropriate as an additional piece of supplementary evidence
Main Panel C
No sub-panels will receive or use citation data
Main Panel D

16. Outputs – interdisciplinary research
For the purposes of the REF, interdisciplinary research is understood to achieve outcomes (including new approaches) that could not be achieved within the framework of a single discipline. Interdisciplinary research features significant interaction between two or more disciplines and / or moves beyond established disciplinary foundations in applying or integrating research approaches from other disciplines.
HEIs are invited to identify outputs that meet this definition. This process is distinct from a request for cross-referral.
There will be no advantage or disadvantage in the assessment in identifying outputs as interdisciplinary.

17. Submissions – interdisciplinary research
Developed with advice from the Interdisciplinary Research Advisory Panel
Interdisciplinary advisers
Oversee and participate in the assessment of IDR
Interdisciplinary identifier
Identify IDR outputs, clearer guidance on use
Section in environment
Unit’s structures in support of IDR

18. Submissions – codes of practice
Code of practice to cover:
Process for ensuring a fair approach to selecting outputs
Process(es) for identifying Category A submitted staff in any UOAs where not submitting 100 per cent
Timetable for submission is summer 2019.
Publication intended by end of 2019.

19. Impact – submission Submission:
Impact remains eligible for submission by institution(s) where research was generated (i.e. non-portable)
Impact must be underpinned by research of minimum 2* quality
Timeframe:
1 January December 2020 for underpinning research
1 August July 2020 for impacts
Case studies continued from examples submitted in 2014 will be eligible for submission in REF 2021, provided they meet the same eligibility criteria

20. Impact – criteria Reach
Assessed against two criteria:
Reach
the extent and/or diversity of the beneficiaries of the impact, as relevant to the nature of the impact. (It will not be assessed in geographic terms, nor in terms of absolute numbers of beneficiaries.)
Significance
the degree to which the impact has enabled, enriched, influenced, informed or changed the performance, policies, practices, products, services, understanding, awareness or well-being of the beneficiaries.

21. Environment template Sections
Unit context, research and impact strategy.
People, including:
staffing strategy and staff development
research students
equality and diversity.
Income, infrastructure and facilities.
Collaboration and contribution to the research base, economy and society.
Information about the unit’s support for impact to be included across the four sections

22. Environment data Research income, degrees awarded and income-in-kind
Use of more quantitative data – advice from working group of Forum for Responsible Research Metrics

23. Environment
Assessment criteria:
Vitality
the extent to which a unit supports a thriving and inclusive research culture for all staff and research students, that is based on a clearly articulated strategy for research and enabling its impact, is engaged with the national and international research and user communities and is able to attract excellent postgraduate and postdoctoral researchers.
Sustainability
the extent to which the research environment ensures the future health, diversity, well-being and wider contribution of the unit and the discipline(s), including investment in people and in infrastructure.

24. Institutional level assessment of environment
Institutional-level information will be included in the UOA-level environment template and will be assessed by the relevant sub-panel in REF 2021.
Pilot of the standalone assessment of the institutional- level environment will draw on this submitted information.
Outcomes from the separate pilot exercise will not be included in REF 2021.

25. Timetable Spring 2018 Panels met to develop criteria
Summer to Autumn 2018
Consultation on draft guidance and criteria
Draft guidance on codes of practice
Early 2019
Publish final guidance and criteria
2019
Complete preparation of submission systems
Submission deadline for codes of practice: noon, 7 June 2019
2020
Submission phase
Submission deadline: noon, 27 November 2020
2021
Assessment phase
Publication of results: December 2021

26. Further information
(includes all relevant documents and FAQs)
Enquiries from staff at HEIs should be directed to their nominated institutional contact (will make available on
Other enquiries to