REF 2021 Briefing Consultation on the draft guidance 19 September 2018

REF 2021 Briefing What was published in July? Draft guidance on submissions Draft panel criteria and working methods Draft guidance on codes of practice A list of independent research fellowships Guidance for standardising quantitative indicators of impact Guidance for using quantitative indicators in the environment statement Guidance to panels (on preparing the panel criteria and working methods)

REF 2021 Briefing Principles underlying the draft guidance Continuity from REF 2014 – notwithstanding the changes that follow on from the Stern review, to maintain a consistent approach with REF 2014 where it was logical to do so Consistency – the main panels have agreed a more consistent approach which has reduced the extent of panel specific criteria and working methods

REF 2021 Briefing Core assumptions remain unchanged… Staff - 100% of eligible staff submitted (assumed to be all ART staff, Grade 8 research only staff and research only staff meeting the independent researcher definition) Outputs (weighted 60%) - a portfolio of outputs which equals 2.5 * the eligible staff FTE on the census date, with a minimum of 1 output per staff member and a maximum of 5 Impact (weighted 25%) - a number of impact case studies based on FTE thresholds Environment (weighted 15%) – a template with supporting data on research income and PhD awards, contextualised by an institutional environment template

REF 2021 Briefing Staff - what did we learn? Updated definitions of ‘significant responsibility for research’ and ‘independent researcher’ Substantive connection – the need for a statement for staff with an FTE between 0.2 – 0.29 Staff circumstances – as in REF 2014, a series of measures are proposed which would allow for a reduction in the number of outputs required to recognise individual circumstances e.g. ECR status, family leave Use of HESA staff returns for audit

REF 2021 Briefing Outputs - what did we learn? Updated open access policy, including the proposal to allow a maximum of 5% of in scope outputs that do not meet the open access policy Rules on submitting co-authored outputs A definition of interdisciplinary research and the criteria and process for assessment Citation data to be in all main panel A UoAs plus 7 (Earth and Environmental Sciences), 8 (Chemistry), 9 (Physics), 11 (Computer Science and Informatics) and 16 (Economics & Econometrics) (*“Neither journal impact factors or hierarchies of journals will be used in the assessment of outputs”)

REF 2021 Briefing Outputs - what did we learn (by panel)? A pilot on capturing research activity costs in UoA4 – Psychology, Psychiatry and Neuroscience A revised list of output types and definitions for main panel D Additional information requirements on certain multi authored outputs to demonstrate substantial research contribution (main panels A & B only) Double-weighting – expectation that this will be more routine for certain output types (main panels C & D only)

REF 2021 Briefing Impact - what did we learn? Clarity on the definition of a continued case study i.e. where the underpinning research is the same as in REF 2014 and the types of impact are the same Guidelines on standardising quantitative indicators (we are invited not required to use these) Main panel A encourages new rather than continued case studies. B, C&D make no distinction Further advice on eligible types of impact (e.g. on teaching within the HEI) and the expectations for supporting evidence

REF 2021 Briefing Environment - what did we learn? (1) What the institutional level environment statement will include i.e. Context & mission; Strategy; People and Income, infrastructure and facilities What the UoA environment template will include i.e. Unit context, research and impact strategy; People; Income, infrastructure and facilities and; Collaboration and contribution Information on the additional data required by each main panel for these sections

REF 2021 Briefing Environment - what did we learn? (2) Unit section weightings – 25% of the score will be attributed to each of the 4 sections except panel D (30% People; 20% Income) Guidelines on standardising quantitative indicators (we are encouraged not required to use these)

REF 2021 Briefing Code of practice This must cover the institution’s approach to: Identifying staff with significant responsibility for research (where a HEI is not submitting 100% of category A eligible staff) Determining research independence The selection of outputs Deadline for submission: 7 June 2019

REF 2021 Briefing Consultation process: Draft guidance and consultation questions circulated to Departments (complete) Draft consultation response circulated to REF Strategy Group for initial comment (complete) Draft consultation response discussed at Faculty Research Groups (complete) Further consideration at today’s briefing REF Strategy Group to agree final response (2nd Oct.) with final comments from URC (4th Oct.) Deadline for submission: noon, 15th October

REF 2021 Briefing Guidance - selected consultation questions: Question 4: Possible indicators of research independence are set out at paragraph 130, including a reference to a list of independent fellowships. This list is intended to guide institutions on determining independence for staff holding fellowships from major research funders. The list is not intended to be comprehensive. Do you have any comments on the clarity, usefulness, or coverage of this list? Proposed response: Yes, the list is helpful in providing more clarity on fellowships. Further clarification on the level of expectation on HEIs to determine the independence of research only staff?

REF 2021 Briefing Guidance - selected consultation questions: Question 7a&b: a. The proposed approach for taking account of circumstances will achieve the aim of promoting equality and diversity in REF 2021. b. The potential advantages of the proposed approach outweigh the potential drawbacks identified. Proposed response: Disagree. We share the aim of the funding bodies to ensure that equality and diversity is promoted throughout the REF. However, it is not clear that the proposed approach will deliver this aim and as acknowledged in the consultation question, there are a series of associated drawbacks e.g. potential for gaming; potential for outputs from staff with individual circumstances being excluded rather (as in REF 2014) being included. Strengthen E&D in the REF5?

REF 2021 Briefing Staff circumstances – worked examples where a reduction of 2 outputs is granted to a portfolio of 25 outputs Scenario 1 - Potential for gaming where the reduction is applied to other staff Optimised return without reduction applied ECR – 1 output Lecturers – 10 outputs Professors – 14 outputs With reduction applied Lecturers – 9 outputs Professors – 13 outputs Scenario 2 - Potential for underrepresentation where the reduction is applied to the individual Optimised return without reduction applied ECR – 3 outputs Lecturers – 9 outputs Professors – 13 outputs With reduction applied ECR – 1 output Lecturers – 10 outputs Professors – 14 outputs

REF 2021 Briefing Guidance - selected consultation questions: Question 11a: Do you agree with the proposed intention to permit the submission of co-authored outputs only once within the same submission? Proposed response: Given the changes to the number of unique outputs required per person the flexibility for co-authors to double weight large scale outputs, this approach is sensible. Further flexibility could be introduced, for example, allowing more than 1 member of staff in a UoA to cite the same output to meet their minimum of 1 requirement, but to only submit the output once.

REF 2021 Briefing Guidance – response on other areas: Open access – tolerance on non compliance – “It is welcome that a degree of non-compliance has been introduced into the guidance. However, it is clear from the recent Research England report that significant challenges remain in terms of achieving compliance with the open access policy, with 19% of outputs studied not compliant or without a known exception... 5% non-compliance provides additional challenges for i) smaller departments and/or disciplines where ii) the journal articles are not the standard publication outlet (therefore likely to be common on Panel D, for example). On the basis of existing data, it may be worth increasing the %/number of non-compliant outputs (e.g. 1 or 2 per UoA) or 10%(?) of in scope outputs (whichever is greater)”.

REF 2021 Briefing Panel criteria and working methods - overall questions Do the UOA descriptors provide a clear and appropriate description of the disciplines covered by the UOAs? Key questions throughout: Are the criteria appropriate? Are the criteria clear? Could more consistency across panels be achieved? Are there differences between disciplines that would justify further differentiation between main panels?

REF 2021 Briefing Panel criteria and working methods - specific questions Outputs - the proposed criteria for double-weighting outputs in Main Panels C and D, and on whether requests to double-weight books should automatically be accepted Outputs - whether Annex C ‘Main Panel D – outputs types and submission guidance’ is helpful and clear Environment - Whether the difference in section weightings across main panels is sufficiently justified by disciplinary difference Environment - whether the list of quantitative indicators provided is clear and helpful

REF 2021 Briefing Any further views? Any additional comments should be directed to the REF manager by Friday 21 September (ed.kirby@york.ac.uk) in the first instance.

REF 2021 Briefing Dates Details 2 October 2018 REF Check Exercise 2018 – completion of exercise including feedback to Departments 15 October 2018 Deadline for responses to the consultation on the draft guidance 31 December 2018 Preparation of draft Code of Practice & Institutional environment template Winter 2018-19 (expected January 2019) Publish final guidance and criteria 1 February 2019 Publication of requirements for Mock REF Exercise 29 March 2019 Mock REF Exercise 2019 - Deadline for submission 7 June 2019 Deadline for the submission of the Code of Practice 28 June 2019 Mock REF Exercise 2019 – completion of exercise including feedback to Departments n.b. full REF timeline available on the REF 2021 webpages…

REF deadline for submission… REF 2021 Briefing REF deadline for submission… 27 November 2020