The Research Excellence Framework Assessment framework and guidance on submissions Graeme Rosenberg, REF Manager

The Research Excellence Framework The REF replaces the RAE as the UK-wide framework for assessing research in all disciplines. Its purpose is: To inform research funding allocations by the four UK HE funding bodies Provide accountability for public funding of research and demonstrate its benefits To provide benchmarks and reputational yardsticks

Principles of the REF The REF is a process of expert review: Assessing all types of research on an equal basis HEIs are encouraged to submit the work of all their excellent researchers Transparent criteria and procedures

Key changes since RAE Inclusion of assessment of impact Fewer UOA’s/panels operating more consistently Strengthened equality and diversity measures Revised eligibility criteria for staff Limited use of citation data in some UOAs Removal of ‘esteem’ as a distinct element Revised approach to ‘environment’ Role of users and additional assessors on panels Publication of overall quality profiles in 1% steps

The assessment framework: overview Overall quality Outputs Maximum of 4 outputs per researcher Impact Template and case studies Environment Template and data 65% 20% 15%

Timetable 2011 Impact decisions (Feb) Panel membership (Feb) Guidance on submissions (Jul) Draft panel criteria for consultation (Jul) 2012 Panel criteria and methods (Jan) HEIs submit codes of practice (Jul) Test submissions system Survey of submission intentions and requests for multiple submissions (Oct- Dec) 2013 Launch submissions system Recruit additional assessors Staff census date (31 Oct) Submissions deadline(29 Nov) 2014 Assessment of submissions Publish outcomes (Dec)

Guidance and criteria Comprehensive information on preparing submissions and on how panels will assess them will be set out in: ‘Assessment framework and guidance on submissions’ (July 2011) ‘Panel criteria and working methods’: - July 2011: published in draft form for consultation - Jan 2012: in final form. Supplementary guidance at later stages will clarify points of detail, but no new types of information will be requested.

Submissions Each submission in UOA provides evidence about the activity and achievements of a ‘submitted unit’, including: - Staff details (REF1a/b/c) - Research outputs (REF2) - Impact template and case studies (REF3a/b) - Environment data (REF 4a/b/c) - Environment template (REF5) A submitted unit may, but need not, comprise staff who work within a single ‘department’ or organisational unit.

Submissions (2) Institutions will normally make one submission in each UOA they elect to submit in Joint submissions are encouraged where this is an appropriate way of describing collaborative research Multiple submissions only by exception and with permission from REF manager: - Where an HEI also makes a joint submission in that UOA - Due to merger - In SP 28 where on submission is in Celtic studies - Where a sub-panel considers there is a disciplinary case

Submissions: process We are developing web-based application to collect submissions: Pilot in autumn 2012 final Launch in January 2013 Survey of submissions intentions and requests for multiple submissions: October - December 2012 All information provided in submissions must be capable of verification – a proportion of all submitted information will be routinely checked and panel members can identify areas for verification

Staff (REF1a) HEIs choose who to submit in two categories: Category A staff: Academic staff with a contract of at least 0.2FTE, on payroll of the HEI on 31 October 2013, with a primary employment function of either ‘research only’ or ‘teaching and research’ Category C staff: Employed by an organisation other than an HEI with a contract or job role including research, and whose research is primarily focused in the submitting unit on 31 October 2013 Research assistants are not generally eligible

Individual staff circumstances (REF1b) Up to four outputs must be listed against each member of staff This can be reduced without penalty where circumstances have constrained an individuals ability to work productively or produce four outputs during the assessment period Consistent approach across panels: Clearly defined circumstances will have a ‘tariff’ based on time ‘absent’ Complex circumstances requiring judgement will be referred to an expert panel

Individual staff circumstances (REF1b) Clearly defined circumstances Qualifying as an ECR Part-time working Maternity, paternity or adoption leave Secondments or career breaks outside of HE Complex circumstances Disability Ill health or injury Mental health conditions Constraints relating to pregnancy or maternity (in addition to a clearly defined period of maternity leave) Childcare or other caring responsibilities Gender reassignment Other circumstances relating to protected characteristics defined in the Equality Act 2010

Research outputs (REF2) Outputs must be: - a product of research (as defined for the REF) - first brought into the public domain between 1 January 2008 and 31 December Authored/co-authored by the member of staff against whom it is listed (regardless of where they were employed prior to the census date) All outputs assessed on a fair and equal basis in terms of ‘rigour, originality and significance’

Research outputs (REF2) Co-authored outputs can be listed against more than one co-author (‘Panel criteria’ will provide further guidance) Institutions may request ‘double-weighting’ of outputs of extended scale and scope (‘Panel criteria’ will provide further guidance)

Research outputs (REF2) Some sub panels will consider the number of times outputs are cited but: will rely on expert review to make rounded judgements based on the full set of criteria will assess all outputs on an equal basis regardless of availability of citation data will recognise the limitations of citation data – especially for recently published outputs – and will have due regard to potential equality implications The funding bodies do not sanction or recommend that HEIs rely on citation data to inform the selection of staff or outputs for their REF submissions

Impact: definition for the REF An effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia It includes an effect, change or benefit to: - The activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding - Of an audience, beneficiary, community, constituency, organisation or individuals - In any geographic location whether locally, regionally, nationally or internationally It excludes impacts on research or the advancement of academic knowledge within HE; and impact on teaching or other activities within the submitting HEI

Impact submissions Panels will provide (not restrictive) guidance on kinds of impact they would anticipate in their UOA, and on appropriate forms of evidence Submissions will comprise: specific examples of impacts underpinned by the submitted unit’s research (case studies) the unit’s general approach to enabling impact from its research (the impact template)

Impact template (REF 3a) Statement to describe the units approach to supporting and enabling impact during the assessment period including: Context Approach during Strategy and plans for supporting impact Relationship between the approach and the submitted case studies

Case studies (REF 3b) Number required is determined by the FTE of Category A staff in the submission Need not be representative of activity across the unit: pick the strongest examples. The impact described must: Meet the REF definition of impact Have occurred between 1 Jan 2008 and 31 July 2013 (can be at any stage of maturity) Be underpinned by excellent research produced by the submitting unit between 1 January 1993 to 31 December 2013

Case studies (REF 3b) Case studies must: Describe the underpinning research Reference one or more key outputs Evidence the quality of the research Explain how the research made a ‘material and distinct’ contribution to the impact Explain and provide evidence of the nature and extent of the impact: who/what was affected? how were they affected? Onus is on submitting units to provide evidence appropriate to the impact.

Environment data (REF 4a/b/c) Environment sub-profile based on narrative template (REF5) informed by standard analysis of submitted data on: Research doctoral degrees awarded (REF4a) Research income (REF4b) Research income in-kind (REF4c) Definitions are aligned as far as possible with HESA. We will provide HESA data to institutions to assist in preparing submissions, and will use it for verification purposes

Environment template (REF5) Narrative information required about the research environment for each submitting unit relating to the assessment period: Overview Research strategy People (including staffing strategy, staff development and research students) Income, infrastructure, and facilities Collaboration and contribution to the discipline Panel criteria will provide further guidance on appropriate forms of evidence

Codes of practice on the selection of staff Each institution is required to develop, document and apply a code of practice on selecting staff to include in their REF submissions and submit to REF team by 31 July 2012 Equality and Diversity Panel will examine in advance of submission deadline and all will be published at end of assessment process Guidance on submissions provides further guidance on developing a code of practice and further support will be on the ECU website from September 2011

Further information From our website: From your nominated institutional contact or data contact Or