1. These slides have been produced by the REF team, and were last updated on 30 January 2012
They provide a summary of the assessment framework and guidance on submissions and the REF panel criteria
They may be used freely for the purposes of briefing any interested parties on the REF
Should anyone other than the REF team edit these slides, then any slides that you add or amend with your own interpretation of the REF guidelines, should not be presented with the REF logo and design. It should be clear to the audience that such slides have not been produced by the REF team.

2. The Research Excellence Framework Assessment framework, guidance on submissions and panel criteria

3. Presentation outline Overview REF panels Staff Outputs Impact
Environment

4. Overview

5. Purpose of the REF Overview:
The REF replaces the RAE as the UK-wide framework for assessing research in all disciplines. Its purpose is:
To inform research funding allocations by the four UK HE funding bodies (approximately £2 billion per year)
Provide accountability for public funding of research and demonstrate its benefits
To provide benchmarks and reputational yardsticks

6. Principles of the REF Overview:
The REF is a process of expert review. The assessment is founded on the professional judgement of discipline- based expert panels.
The conduct of the REF is guided by the principle of:
Equity: All types of research and forms of output in all disciplines shall be assessed on an equal basis
Equality: HEIs are encouraged to submit the work of all their excellent researchers
Transparency: The assessment criteria, procedures and outcomes to be published in full

7. Key changes since 2008 RAE Overview: Inclusion of assessment of impact
Fewer UOAs/panels, operating more consistently
Strengthened equality and diversity measures
Revised eligibility criteria for staff
Addition of (limited) use of citation data in some UOAs
Removal of ‘esteem’ as a distinct element
Revised approach to ‘environment’ and data collection
Increased ‘user’ input on panels; and an integrated role for additional assessors
Publication of overall quality profiles in 1% steps

8. Guidance and criteria Overview:
Comprehensive information and guidance is set out in:
Assessment framework and guidance on submissions (July 2011):
Sets out the information required in submissions and the definitions used
Panel criteria and working methods (Jan 2012):
Sets out how panels will assess submissions
Further supplementary guidance will be published during on technical and procedural matters

9. The assessment framework
Overview:
The assessment framework
Overall quality
Outputs
Maximum of 4 outputs per researcher
Impact
Impact template and case studies
Environment
Environment data and template
65%
20%
15%

10. Submissions Overview:
Each submission in a UOA provides evidence about the activity and achievements of a ‘submitted unit’ including:
Staff details (REF1a/b/c)
Research outputs (REF2)
Impact template and case studies (REF3a/b)
Environment data (REF4a/b/c)
Environment template (REF5)
A submitted unit may, but need not, comprise staff who work within a single ‘department’ or organisational unit

11. Multiple and joint submissions
Overview:
Multiple and joint submissions
Institutions will normally make one submission in each UOA they elect to submit in
Joint submissions are encouraged where this is an appropriate way of describing collaborative research
Multiple submissions only by exception and with permission from the REF manager:
Where an HEI also makes a joint submission in that UOA
Where HEIs have merged
In Sub-panel 28 where one submission is in Celtic studies
Where a sub-panel considers there is a case given the nature of the disciplines covered. These sub-panels are listed in the panel criteria statements.

12. Timetable Overview: 2011 2013 2014 2012 Panels appointed (Feb)
Guidance on submissions (Jul)
Draft panel criteria for consultation (Jul)
Close of consultation (5 Oct)
2012
Panel criteria (Jan)
HEIs submit codes of practice (final deadline Jul)
Pilot of submissions system (Sep)
Requests for multiple submissions (final deadline Dec)
Survey of submission intentions (Dec)
2013
Launch REF submissions system (Jan)
Recruit additional assessors
Staff census date (31 Oct)
Submissions deadline (29 Nov)
2014
Panels assess submissions
Publish outcomes (Dec)

13. Publication of results
Overview:
Publication of results
The primary outcome of the REF is an ‘overall quality profile’ to be awarded to each submission:
Using the same scale as RAE2008, but in steps of 1%
To be published in Dec 2014
Further reports and feedback during 2015:
An overview report by each main panel, including observations by each of their sub-panels
Concise feedback on submissions, to the heads of HEIs
The output, impact and environment sub-profile for each submission
A report by the Equality and Diversity Advisory Panel
Submissions will be published (except for confidential or sensitive information)

14. Example of a quality profile
Overview:
Example of a quality profile

15. REF panels

16. Main and sub-panel roles
REF panels:
Main and sub-panel roles
There are 36 sub-panels working under the guidance of 4 main panels. Membership is published at
Sub-panel responsibilities
Contributing to the main panel criteria and working methods
Assessing submissions and recommending the outcomes
Main panel responsibilities
Developing the panel criteria and working methods
Ensuring adherence to the criteria/procedures and consistent application of the overall assessment standards
Signing off the outcomes

17. Main Panel A REF panels: 1 Clinical Medicine 2
Public Health, Health Services and Primary Care 3
Allied Health Professions, Dentistry, Nursing and Pharmacy 4
Psychology, Psychiatry and Neuroscience 5
Biological Sciences 6
Agriculture, Veterinary and Food Science

18. Main Panel B REF panels: 7 Earth Systems and Environmental Sciences 8
Chemistry 9
Physics 10
Mathematical Sciences 11
Computer Sciences and Informatics 12
Aeronautical, Mechanical, Chemical and Manufacturing Engineering 13
Electrical and Electronic Engineering, Metallurgy and Materials 14
Civil and Construction Engineering 15
General Engineering

19. Main Panel C REF panels: 16
Architecture, Built Environment and Planning 17
Geography, Environment Studies and Archaeology 18
Economics and Econometrics 19
Business and Management Studies 20
Law 21
Politics and International Studies 22
Social Work and Social Policy 23
Sociology 24
Anthropology and Development Studies 25
Education 26
Sports-related Studies

20. Main Panel D REF panels: 27 Area Studies 28 Modern Languages 29
English Literature and Language 30
History 31
Classics 32
Philosophy 33
Theology and Religious Studies 34
Art and Design: History, Practice and Theory 35
Music, Drama, Dance and Performing Arts 36
Communications, Cultural and Media Studies, Library and Information Management

21. Additional assessors REF panels:
Additional assessors will be appointed to extend the breadth and depth of panels’ expertise:
Both ‘academic’ assessors (to assess outputs) and ‘user’ assessors (to assess impacts) will be appointed
Assessors will play a full and equal role to panel members, in developing either the outputs or impact sub-profiles. They will be fully briefed, take part in calibration exercises and attend meetings:
Some appointments in 2012, where the sub-panel has already identified a clear gap in expertise required to assess outputs or impact
Further appointments in 2013, in the light of the survey of institutions’ submission intentions

22. UOA descriptors and boundaries
REF panels:
UOA descriptors and boundaries
UOA boundaries are not rigidly defined and all panels expect submissions to cross boundaries with other UOAs
Sub-panel membership is broad and includes members with interdisciplinary expertise
Assessors will be appointed to extend their breadth and depth of expertise. Where there is a significant overlap between UOAs, ‘joint’ assessors may be appointed to work with more than one sub-panel.
The sub-panels prefer to assess submitted work within the sub-panel but may, exceptionally, cross-refer parts of submissions to other sub-panels for advice. The original sub-panel remains responsible for recommending the outcome

23. Main panel working methods
REF panels:
Main panel working methods
Each main panel will work with its sub-panels to apply common assessment criteria
Panels will undertake calibration exercises and reviews of emergent outcomes to make sure overall assessment standards are applied consistently

24. Sub-panel working methods
REF panels:
Sub-panel working methods
The sub-panel chair will allocate work to members/assessors with appropriate expertise
All outputs will be reviewed in sufficient detail to contribute to the outputs sub-profile
Each case study will normally be assessed by at least one academic member and one user member/assessor
Sub-panel members may raise audit queries with the REF team for investigation
The panel secretariat will minute panel procedures

25. Staff

26. Category A and C staff Staff:
HEIs select which staff to include in submissions:
Category A staff: Academic staff with a contract of at least 0.2 FTE, on the payroll of the HEI on 31 Oct 2013, with a primary employment function of either ‘research only’ or ‘teaching and research’
Category C staff: Staff employed by an organisation other than an HEI with a contract or job role including research, and whose research is primarily focused in the submitting unit on 31 Oct 2013
(Category C staff will contribute to the quality profile but not the volume measure for funding purposes)
Research assistants are eligible only by exception

27. Codes of practice on staff selection
The funding bodies encourage HEIs to submit the work of all their excellent researchers, and HEIs have legal obligations affecting their staff selection procedures
Each HEI is required to develop, document and apply a code of practice on the fair and transparent selection of staff for the REF:
Must be signed off by the head of the institution and submitted to the REF team by 31 July 2012 at the latest
The REF Equality and Diversity Advisory Panel will examine these for adherence to the guidance
They will be published at the end of the assessment process

28. Codes of practice on staff selection
Codes should demonstrate fairness to staff by addressing the principles of:
Transparency: clearly setting out the procedures for staff selection, and communicating these to all eligible staff
Consistency: applying consistent procedures across the institution
Accountability: clearly defining responsibilities for decisions, with appropriate training for those involved
Inclusivity: promoting an inclusive environment, with robust procedures for staff to disclose individual circumstances
An Equality Impact Assessment should inform the code and be kept under review at key stages of the selection process

29. Individual staff circumstances
Up to four outputs must be listed against each member of staff
This can be reduced without penalty where circumstances have constrained an individual’s ability to work productively or produce four outputs during the assessment period:
A wide range of circumstances will be taken into account
With as much clarity as possible about how many outputs may be reduced without penalty
To be treated consistently across the exercise
The allowances for maternity, paternity and adoption leave have been revised following consultation

30. Individual staff circumstances
Clearly defined circumstances
Early Career Researchers
Part-time working, career breaks and secondments outside of HE
Periods of maternity, adoption and additional paternity leave
Complex circumstances
Disability
Ill health or injury
Mental health conditions
Additional constraints related to bringing a child into the family
Other caring responsibilities
Gender reassignment
Other circumstances related to the protected characteristics or employment legislation

31. Clearly defined circumstances
Staff:
Clearly defined circumstances
‘Tariffs’ define the number of outputs that may be reduced without penalty
These will be applied consistently by all REF sub- panels
Circumstances may be combined up to a maximum reduction of three outputs
Where an individual has a combination of clearly defined and complex circumstances, these should be submitted collectively as ‘complex’

32. Clearly defined circumstances: Early career researchers
Staff:
Clearly defined circumstances: Early career researchers
Staff eligible for selection who started their careers as independent researchers on or after 1 August 2009
Date at which the individual first met the REF definition of an early career researcher:
Number of outputs may be reduced by up to:
On or before 31 Jul 2009
Between 1 Aug 2009 and 31 Jul inclusive 1
Between 1 Aug 2010 and 31 Jul inclusive 2
On or after 1 Aug 2011 3

33. Staff:
Clearly defined circumstances: Part-time working, secondments and career breaks
Total months absent between Jan 2008 and 31 Oct 2013:
Number of outputs may be reduced by up to: 1 2
46 or more 3
For part-time working, the equivalent ‘months absent’ should be calculated

34. Clearly defined circumstances: Maternity, paternity and adoption leave
Staff:
Clearly defined circumstances: Maternity, paternity and adoption leave
Changes were made to these arrangements, following consultation
Individuals may reduce the number of outputs by one, for each discrete period of:
Statutory maternity leave or statutory adoption leave taken substantially during the period 1 January 2008 to 31 October 2013, regardless of the length of the leave. 
Additional paternity or adoption leave lasting for four months or more, taken substantially during the period 1 January 2008 to 31 October 2013.
Constraints relating to pregnancy, maternity, paternity, adoption or childcare that fall outside of – or justify the reduction of further outputs in addition to – these allowances may be submitted using the arrangements for complex circumstances.

35. Clearly defined circumstances: Other circumstances in UOAs 01-06
Staff:
Clearly defined circumstances: Other circumstances in UOAs 01-06
Outputs may be reduced by up to two for the following:
Category A staff who are junior clinical academics (and have not gained a CCT or equivalent prior to 31 October 2013.
Category C staff who are employed primarily as clinical, health or veterinary professionals

36. Complex circumstances
Staff:
Complex circumstances
EDAP will consider all cases of complex circumstances
EDAP will make recommendations to the Main Panel Chairs, who will decide
Sub-panels will be informed of the decisions and will assess the remaining outputs without penalty
Information will be kept confidential to EDAP, Main Panel Chairs and the REF team
ECU will publish a range of worked examples – including EDAP’s rationale

37. Outputs

38. Eligibility Outputs: Outputs must be:
a product of research (as defined for the REF)
first brought into the public domain between 1 January and 31 December 2013
Authored/co-authored by the member of staff against whom it is listed (regardless of where they were employed prior to the census date)

39. Outputs ‘pre-published’ before 2008
A change in the guidance on pre-publication of outputs has been made, following consultation:
An output first published in its final form during the REF publication period, but ‘pre-published’ in – is eligible for submission to the REF, provided that the ‘pre-published’ output was not submitted to the 2008 RAE. Examples may include:
An online first article or preprint
A preliminary version or working paper

40. Range of output types Outputs:
Outputs may include but are not limited to: printed or electronic publications, materials, devices, images, artefacts, products, buildings, confidential or technical reports, patents, performances, exhibits or events
All forms of outputs shall be assessed on a fair and equal basis

41. Co-authorship Outputs:
Co-authored outputs can be listed against more than one co-author when returned in different submissions
An output can be listed no more than twice within the same submission and must be accompanied by a statement explaining the substantial and distinctive contribution of each of the submitting authors

42. Double-weighting Outputs:
Institutions may request ‘double-weighting’ for outputs of extended scale and scope
Requests for double weighting should be accompanied by a supporting statement
Sub-panels will assess the claim for double-weighting separately from assessing the quality of the output
If a sub-panel accepts a request for double weighting, the output will count as two outputs in both a submission and in the calculation of the outputs sub- profile

43. Additional information on outputs
Panels may make use of additional information in the form of citation data and details provided by the HEI
Sub-panels have requested further information on the research process or content where this is not evident from the output
Sub-panel will not make use of journal impact factors, rankings or lists, or the perceived standing of the publisher, in assessing the quality of research outputs

44. Citation data Outputs:
The following sub-panels will make use of citation data:
Main Panel A: Sub-panels 1-6
Main Panel B: Sub-panels 7-11
Main Panel C: Sub-panel 18
These sub-panels will use only the citation data provided by the REF team, sourced from Scopus (except for Sub- panel 11 that will, in addition, use Google Scholar)
All other sub-panels will neither receive nor make use of citation data
None of the sub-panels will use journal impact factors, journal rankings or other forms of bibliometric analysis

45. Citation data Outputs:
Where used, sub-panels will consider citation data as follows:
As one indicator of ‘academic significance’. Expert review remains the primary means of assessing outputs
Panels will assess all outputs on an equal basis regardless of the availability of such data. They recognise the limitations of citation data – especially for recently published outputs – and will have due regard to potential equality implications
In the relevant UOAs only, citation counts from Scopus will be provided to panels on a consistent basis. HEIs will be able to verify the outputs have been matched correctly, and view citation counts on the submission system
The funding bodies do not sanction or recommend that HEIs rely on citation data to inform the selection of staff or outputs for their REF submissions

46. originality, significance and rigour*
Outputs:
Assessment criteria
The criteria for assessing the quality of outputs are
originality, significance and rigour*
Four star
Quality that is world-leading in terms of originality, significance and rigour
Three star
Quality that is internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence
Two star
Quality that is recognised internationally in terms of originality, significance and rigour
One star
Quality that is recognised nationally in terms of originality, significance and rigour
Unclassified
Quality that falls below the standard of nationally recognised work. Or work which does not meet the published definition of research for the purposes of this assessment
* Each main panel provides a descriptive account of the criteria

47. Impact

48. Definition of impact for the REF
An effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia
Impact includes an effect, change or benefit to:
The activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding
Of an audience, beneficiary, community, constituency, organisation or individuals
In any geographic location whether locally, regionally, nationally or internationally
It excludes impacts on research or the advancement of academic knowledge within HE; and impacts on teaching or other activities within the submitting HEI

49. Range of impacts Impact:
Panels recognise that Impacts can manifest in a wide variety of different ways, may take many forms and occur in a wide range of spheres
Examples of impact may include:
Impacts on public policy and services,
Impacts on society, culture and creativity,
Impacts on practitioners and services,
Impacts on the environment,
Impacts on the economy

50. Examples of impacts: Main Panel A
Impacts on:
For example:
Health and welfare
Patient outcomes have improved
Society, culture and creativity
Public debate has been stimulated
The economy
Costs of healthcare have changed
Commerce
Business performance has improved
Public policy and services
Policy decisions informed by research
Production
Waste levels have reduced
Practitioners and services
Professional standards influenced by research
Environment
Changes to environmental risk management
International development
Quality of life improvements

51. Range of impacts – Main Panel B
Impacts on:
For example:
The economy
Patient outcomes have improved
Public policy and services
Public debate has been stimulated
Society, culture and creativity
Public engagement in science has been stimulated
Health
Business performance has improved
Practitioners and professional services
Policy decisions informed by research
Environment
Waste levels have reduced

52. Examples of impacts: Main Panel C
Impacts on:
Examples:
Creativity, culture and society
Enhancements to heritage preservation, conservation and presentation
Shaping or informing public attitudes and values
Economy, commerce or organisations
Improved products, processes or workplace practices
Enhanced corporate social responsibility policies
Environment
Changes in public awareness or behaviour
Business operations have been changed to achieve environmental objectives
Health and welfare
Development of policy or practice with regard to health services or social care provision
Practitioners and professional services
Influence on professional standards, guidelines or training
Practitioner debate has been stimulated by research findings
Public policy, law and services
Legislative change or effect on legal practice
Influence on policy (by government, NGO or private organisation)
Impact on democratic participation or access to justice

53. Examples of impacts: Main Panel D
Impacts on:
Examples:
Civil society
Cultural life
Economic prosperity
Design of new products or services
Education
Policy making
Public discourse
Public services

54. Submissions Impact: Impact template (REF3a)
Sets out the submitted unit’s general approach to enabling impact from its research
One template per submission – with a page limit depending on the number of staff submitted
Covers the period 1 Jan 2008 to 31 Jul 2013
Contributes 20% to the impact sub-profile
Case studies (REF3b)
Specific examples of impacts already achieved, that were underpinned by the submitted unit’s research
The number of case studies required depends on the number of staff submitted (1 plus 1 per 10 FTE)
Impacts during 1 Jan 2008 to 31 Jul 2013; underpinned by research since 1 Jan 1993
Contributes 80% to the impact sub-profile

55. Case studies (REF3b) Impact:
In each case study, the impact described must:
Meet the REF definition of impact
Have occurred between 1 Jan 2008 and 31 July (can be at any stage of maturity)
Be underpinned by excellent research (of at least 2* quality) produced by the submitting unit between 1 January 1993 to 31 December 2013
Submitted case studies need not be representative of activity across the unit: pick the strongest examples

56. Underpinning research
Impact:
Underpinning research
Case studies must be underpinned by research produced by the submitted unit that has made a material and distinct contribution to the impact
Case studies must provide evidence that the underpinning research meets the quality threshold of at least two star
HEIs may submit case studies of a confidential nature, these may be redacted for publication or not published at all
HEIs may request advance permission to submit case studies containing sensitive material that could only be assessed by individuals with national security clearance

57. Types of evidence Impact:
Evidence of the nature and extent of the impact, in terms of its reach and significance, should be provided and may include:
Critical reviews in the media,
Citation by international bodies,
Visitor or audience numbers and feedback
Business performance measures, sales, turnover,
Documented evidence of change to public policy/legislation/regulations/guidelines
Verifiable influence on particular projects or processes which bring benefits

58. Case studies (REF3b) Impact:
Each case study is limited to 4 pages and must:
Describe the underpinning research produced by the submitting unit
Reference one or more key outputs and provide evidence of the quality of the research
Explain how the research made a ‘material and distinct’ contribution to the impact (there are many ways in which this may have taken place)
Explain and provide evidence of the nature and extent of the impact: Who/what was affected? How were they affected? When?
Provide independent sources that could be used to verify claims about the impact (on a sample audit basis)

59. Template (REF3a) Impact:
The unit’s approach to enabling impact from its research:
Context for the approach
The unit’s approach during
Strategy and plans for supporting impact
Relationship to the submitted case studies
Provides additional information and context for the case studies, and can take account of particular circumstances that may have constrained a unit’s selection of case studies
To be assessed in terms of the extent to which the unit’s approach is conducive to achieving impact of ‘reach and significance’

60. The criteria for assessing impacts are reach and significance*
Assessment criteria
The criteria for assessing impacts are reach and significance*
Four star
Outstanding impacts in terms of their reach and significance
Three star
Very considerable impacts in terms of their reach and significance
Two star
Considerable impacts in terms of their reach and significance
One star
Recognised but modest impacts in terms of their reach and significance
Unclassified
The impact is of little or no reach and significance; or the impact was not eligible; or the impact was not underpinned by excellent research produced by the submitted unit
* Each main panel provides a descriptive account of the criteria

61. Environment

62. Data (REF4) Environment: All submissions to include data on:
Research doctoral degrees awarded (REF4a)
Research income (REF4b)
Research income in-kind (REF4c)
Definitions are aligned with HESA returns. We will provide HESA data to institutions to assist in preparing submissions, and will use it for verification purposes
Sub-panels 8, 9, 19, 25 and 26 request specific additional data, to be included within the environment template (REF5)
Data will be considered by panels alongside the qualitative information provided in REF5

63. Environment template (REF5)
Each submission to include a completed template (with page limits depending on the number of staff submitted):
Overview
Research strategy
People (including staffing strategy and staff development; and research students)
Income, infrastructure, and facilities
Collaboration and contribution to the discipline
Panel criteria indicate appropriate forms of evidence, and the relative importance of each section
No expectation that the environment relates to a single department or organisational unit

64. vitality and sustainability*
Environment:
Assessment criteria
The criteria for assessing the environment are
vitality and sustainability*
Four star
An environment that is conducive to producing research of world-leading quality, in terms of its vitality and sustainability
Three star
An environment that is conducive to producing research of internationally excellent quality, in terms of its vitality and sustainability
Two star
An environment that is conducive to producing research of internationally recognised quality, in terms of its vitality and sustainability
One star
An environment that is conducive to producing research of nationally recognised quality, in terms of its vitality and sustainability
Unclassified
An environment that is not conducive to producing research of nationally recognised quality
* Each main panel provides a descriptive account of the criteria

65. Further information www.ref.ac.uk (includes all relevant documents)
Enquiries from staff at HEIs should be directed to their nominated institutional contact (see for a list)
Other enquiries to