1. REF 2021 Briefing Consultation on the draft guidance
19 September 2018

2. REF 2021 Briefing What was published in July?
Draft guidance on submissions
Draft panel criteria and working methods
Draft guidance on codes of practice
A list of independent research fellowships
Guidance for standardising quantitative indicators of impact
Guidance for using quantitative indicators in the environment statement
Guidance to panels (on preparing the panel criteria and working methods)

3. REF 2021 Briefing Principles underlying the draft guidance
Continuity from REF 2014 – notwithstanding the changes that follow on from the Stern review, to maintain a consistent approach with REF 2014 where it was logical to do so
Consistency – the main panels have agreed a more consistent approach which has reduced the extent of panel specific criteria and working methods

4. REF 2021 Briefing Core assumptions remain unchanged…
Staff - 100% of eligible staff submitted (assumed to be all ART staff, Grade 8 research only staff and research only staff meeting the independent researcher definition)
Outputs (weighted 60%) - a portfolio of outputs which equals 2.5 * the eligible staff FTE on the census date, with a minimum of 1 output per staff member and a maximum of 5
Impact (weighted 25%) - a number of impact case studies based on FTE thresholds
Environment (weighted 15%) – a template with supporting data on research income and PhD awards, contextualised by an institutional environment template

5. REF 2021 Briefing Staff - what did we learn?
Updated definitions of ‘significant responsibility for research’ and ‘independent researcher’
Substantive connection – the need for a statement for staff with an FTE between 0.2 – 0.29
Staff circumstances – as in REF 2014, a series of measures are proposed which would allow for a reduction in the number of outputs required to recognise individual circumstances e.g. ECR status, family leave
Use of HESA staff returns for audit

6. REF 2021 Briefing Outputs - what did we learn?
Updated open access policy, including the proposal to allow a maximum of 5% of in scope outputs that do not meet the open access policy
Rules on submitting co-authored outputs
A definition of interdisciplinary research and the criteria and process for assessment
Citation data to be in all main panel A UoAs plus 7 (Earth and Environmental Sciences), 8 (Chemistry), 9 (Physics), 11 (Computer Science and Informatics) and 16 (Economics & Econometrics) (*“Neither journal impact factors or hierarchies of journals will be used in the assessment of outputs”)

7. REF 2021 Briefing Outputs - what did we learn (by panel)?
A pilot on capturing research activity costs in UoA4 – Psychology, Psychiatry and Neuroscience
A revised list of output types and definitions for main panel D
Additional information requirements on certain multi authored outputs to demonstrate substantial research contribution (main panels A & B only)
Double-weighting – expectation that this will be more routine for certain output types (main panels C & D only)

8. REF 2021 Briefing Impact - what did we learn?
Clarity on the definition of a continued case study i.e. where the underpinning research is the same as in REF 2014 and the types of impact are the same
Guidelines on standardising quantitative indicators (we are invited not required to use these)
Main panel A encourages new rather than continued case studies. B, C&D make no distinction
Further advice on eligible types of impact (e.g. on teaching within the HEI) and the expectations for supporting evidence

9. REF 2021 Briefing Environment - what did we learn? (1)
What the institutional level environment statement will include i.e. Context & mission; Strategy; People and Income, infrastructure and facilities
What the UoA environment template will include i.e. Unit context, research and impact strategy; People; Income, infrastructure and facilities and; Collaboration and contribution
Information on the additional data required by each main panel for these sections

10. REF 2021 Briefing Environment - what did we learn? (2)
Unit section weightings – 25% of the score will be attributed to each of the 4 sections except panel D (30% People; 20% Income)
Guidelines on standardising quantitative indicators (we are encouraged not required to use these)

11. REF 2021 Briefing Code of practice
This must cover the institution’s approach to:
Identifying staff with significant responsibility for research (where a HEI is not submitting 100% of category A eligible staff)
Determining research independence
The selection of outputs
Deadline for submission: 7 June 2019

12. REF 2021 Briefing Consultation process:
Draft guidance and consultation questions circulated to Departments (complete)
Draft consultation response circulated to REF Strategy Group for initial comment (complete)
Draft consultation response discussed at Faculty Research Groups (complete)
Further consideration at today’s briefing
REF Strategy Group to agree final response (2nd Oct.) with final comments from URC (4th Oct.)
Deadline for submission: noon, 15th October

13. REF 2021 Briefing Guidance - selected consultation questions:
Question 4: Possible indicators of research independence are set out at paragraph 130, including a reference to a list of independent fellowships. This list is intended to guide institutions on determining independence for staff holding fellowships from major research funders. The list is not intended to be comprehensive. Do you have any comments on the clarity, usefulness, or coverage of this list?
Proposed response: Yes, the list is helpful in providing more clarity on fellowships. Further clarification on the level of expectation on HEIs to determine the independence of research only staff?

14. REF 2021 Briefing Guidance - selected consultation questions:
Question 7a&b: a. The proposed approach for taking account of circumstances will achieve the aim of promoting equality and diversity in REF b. The potential advantages of the proposed approach outweigh the potential drawbacks identified.
Proposed response: Disagree. We share the aim of the funding bodies to ensure that equality and diversity is promoted throughout the REF. However, it is not clear that the proposed approach will deliver this aim and as acknowledged in the consultation question, there are a series of associated drawbacks e.g. potential for gaming; potential for outputs from staff with individual circumstances being excluded rather (as in REF 2014) being included. Strengthen E&D in the REF5?

15. REF 2021 Briefing
Staff circumstances – worked examples where a reduction of 2 outputs is granted to a portfolio of 25 outputs
Scenario 1 - Potential for gaming where the reduction is applied to other staff
Optimised return without reduction applied
ECR – 1 output
Lecturers – 10 outputs
Professors – 14 outputs
With reduction applied
Lecturers – 9 outputs
Professors – 13 outputs
Scenario 2 - Potential for underrepresentation where the reduction is applied to the individual
Optimised return without reduction applied
ECR – 3 outputs
Lecturers – 9 outputs
Professors – 13 outputs
With reduction applied
ECR – 1 output
Lecturers – 10 outputs
Professors – 14 outputs

16. REF 2021 Briefing Guidance - selected consultation questions:
Question 11a: Do you agree with the proposed intention to permit the submission of co-authored outputs only once within the same submission?
Proposed response: Given the changes to the number of unique outputs required per person the flexibility for co-authors to double weight large scale outputs, this approach is sensible. Further flexibility could be introduced, for example, allowing more than 1 member of staff in a UoA to cite the same output to meet their minimum of 1 requirement, but to only submit the output once.

17. REF 2021 Briefing Guidance – response on other areas:
Open access – tolerance on non compliance – “It is welcome that a degree of non-compliance has been introduced into the guidance. However, it is clear from the recent Research England report that significant challenges remain in terms of achieving compliance with the open access policy, with 19% of outputs studied not compliant or without a known exception... 5% non-compliance provides additional challenges for i) smaller departments and/or disciplines where ii) the journal articles are not the standard publication outlet (therefore likely to be common on Panel D, for example). On the basis of existing data, it may be worth increasing the %/number of non-compliant outputs (e.g. 1 or 2 per UoA) or 10%(?) of in scope outputs (whichever is greater)”.

18. REF 2021 Briefing
Panel criteria and working methods - overall questions
Do the UOA descriptors provide a clear and appropriate description of the disciplines covered by the UOAs?
Key questions throughout:
Are the criteria appropriate?
Are the criteria clear?
Could more consistency across panels be achieved?
Are there differences between disciplines that would justify further differentiation between main panels?

19. REF 2021 Briefing
Panel criteria and working methods - specific questions
Outputs - the proposed criteria for double-weighting outputs in Main Panels C and D, and on whether requests to double-weight books should automatically be accepted
Outputs - whether Annex C ‘Main Panel D – outputs types and submission guidance’ is helpful and clear
Environment - Whether the difference in section weightings across main panels is sufficiently justified by disciplinary difference
Environment - whether the list of quantitative indicators provided is clear and helpful

20. REF 2021 Briefing Any further views?
Any additional comments should be directed to the REF manager by Friday 21 September in the first instance.

21. REF 2021 Briefing
Dates
Details
2 October 2018
REF Check Exercise 2018 – completion of exercise including feedback to Departments
15 October 2018
Deadline for responses to the consultation on the draft guidance
31 December 2018
Preparation of draft Code of Practice & Institutional environment template
Winter (expected January 2019)
Publish final guidance and criteria
1 February 2019
Publication of requirements for Mock REF Exercise
29 March 2019
Mock REF Exercise Deadline for submission
7 June 2019
Deadline for the submission of the Code of Practice
28 June 2019
Mock REF Exercise 2019 – completion of exercise including feedback to Departments
n.b. full REF timeline available on the REF 2021 webpages…

22. REF deadline for submission…
REF 2021 Briefing
REF deadline for submission…
27 November 2020