Achieving and Demonstrating Research Impact John Scott

Workshop Outline What is ‘impact’ and how can it be achieved? This part of the workshop will consider the REF definition of impact, the various types of impact, and the criteria for scoring impact. It will consider the broad range of types of impact and their varying advantages and disadvantages. It will also consider how the achievement of impact might be built into research projects and the support mechanisms that Universities can establish to promote the achievement of impact. Coffee break How can impact be demonstrated? In this part of the workshop I will consider the issue of producing evidence on impact and of constructing a plausible account of the impact achieved. We will consider the issues involved in constructing and presenting appropriate case studies. The Future of the REF. This final session will be an open discussion of wider issues in the REF: panel structure, assessment mechanisms, bibliometrics, etc. It is an opportunity to explore issues and potentialities in the approach to REF Final Questions

What is ‘impact’ and how can it be achieved?

The Development of Research Assessment 1986 UGC Review Subject areas rated as ’outstanding’, ‘above average’, ‘average’ or ‘below average’ UFC Research Selectivity Exercise 1992 HEFCE (S&W) Research Assessment Exercise Subject areas rated on scale from 1 to 5. Grade 4 means ‘some evidence of international excellence’ RAE Grade 3 split into 3a and 3b. 3a ‘ possibly showing evidence of international excellence.’ 2001 RAE 2008 RAE Each subject area given a ‘profile’ with % of activity at levels from 1 to 4. Grade 2 means ‘Quality that is recognised internationally in terms of originality, significance and rigour 2014 Research Excellence Framework (REF)

All exercises involved, to a greater or lesser degree, an assessment of publications and the research environment. Later exercises added assessment of ‘esteem’ REF introduced the assessment of ‘impact’, replacing ‘esteem’.

Subject Panels Assessments made by ‘expert’ panels, generally discipline-based, but have become broader in composition. Subject panels coordinated by ‘Main Panels’ covering large areas. Universities can choose how to group their staff in subject areas and which panels to submit them to.

ABCD 1 Clinical Medicine 2 Public Health, Health Services and Primary Care 3 Allied Health Professions, Dentistry, Nursing and Pharmacy 4 Psychology, Psychiatry and Neuroscience 5 Biological Sciences 6 Agriculture, Veterinary and Food Science 7 Earth Systems and Environmental Sciences 8 Chemistry 9 Physics 10 Mathematical Sciences 11 Computer Science and Informatics 12 Aeronautical, Mechanical, Chemical and Manufacturing Engineering 13 Electrical and Electronic Engineering, Metallurgy and Materials 14 Civil and Construction Engineering 15 General Engineering 16 Architecture, Built Environment and Planning 17 Geography, Environmental Studies and Archaeology 18 Economics and Econometrics 19 Business and Management Studies 20 Law 21 Politics and International Studies 22 Social Work and Social Policy 23 Sociology 24 Anthropology and Development Studies 25 Education 26 Sport and Exercise Sciences, Leisure and Tourism 27 Area Studies 28 Modern Languages and Linguistics 29 English Language and Literature 30 History 31 Classics 32 Philosophy 33 Theology and Religious Studies 34 Art and Design: History, Practice and Theory 35 Music, Drama, Dance and Performing Arts 36 Communication, Cultural and Media Studies, Library and Information Management

The ‘Profile’ system Each element of the assessment (outputs, environment, and esteem/impact) assessed separately. Panels judge the % of Each element that is rated at the quality levels. Profiles for each element combined into an overall profile using a ‘weighting’. Weightings used in REF: Outputs 65% Environment 15% Impact 20%

Structure of the Impact Assessment Universities must submit one case study for about every eight staff submitted, with a minimum of two case studies. Must also submit an Impact Statement – strategy and approach to impact. Case Studies and Statement are equal constituents of the assessment.

Where did ‘impact’ come from? Government demand for ‘value for money’. Public reaction to ‘ivory towers’ HEFCE pre-empted Government action by introducing ‘impact assessment’. Consultation and debate led to broadening of meaning of ‘impact’.

Definition of impact for the REF (From Assessment Framework and Guidance on Submissions) 140. For the purposes of the REF, impact is defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia (as set out in paragraph 143) Impact includes, but is not limited to, an effect on, change or benefit to: the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding of an audience, beneficiary, community, constituency, organisation or individuals in any geographic location whether locally, regionally, nationally or internationally Impact includes the reduction or prevention of harm, risk, cost or other negative effects. Paragraph 143 excludes ‘academic impact’ and impact on students.

This definition is specified by each main panel. There are slight differences, so it is important to know which panel is to be submitted to. I will illustrate with the criteria for Main Panel C: ‘The main panel acknowledges that impact within its remit may take many forms and occur in a wide range of spheres. These may include (but are not restricted to): creativity, culture and society; the economy, commerce or organisations; the environment; health and welfare; practitioners and professional services; public policy, law and services. ‘The categories used to define spheres of impact, for the purpose of this document, inevitably overlap and should not be taken as restrictive. Case studies may describe impacts which have affected more than one sphere.’

The criteria for assessing impacts are ‘reach and significance’.: ‘Reach will be understood in terms of the extent and diversity of the communities, environments, individuals, organisations or any other beneficiaries that have benefited or been affected. ‘Significance will be understood in terms of the degree to which the impact has enriched, influenced, informed or changed policies, opportunities, perspectives or practices of communities, individuals or organisations.’ (Panel C Criteria) These are assessed on the scale of starred levels: Four star Outstanding impacts in terms of their reach and significance. Three star Very considerable impacts in terms of their reach and significance. Two star Considerable impacts in terms of their reach and significance. One star Recognised but modest impacts in terms of their reach and significance. Unclassified The impact is of little or no reach and significance; or the impact was not eligible; or the impact was not underpinned by excellent research produced by the submitted unit.’

Key considerations: Eligibility of impact Research must have been carried out at the institution between 1 st Jan 1993 and 31 st December i.e., 20 year period up to submission date. For REF 2020, probably research from 1 st Jan Impact must be apparent between 1 st January 2008 and 31 st July For REF 2020, probably impact after 1 st January Presentation of evidence Must put in place mechanisms for collecting evidence on impact. This needs to be an integral part of the research and of Department/Faculty practice. Support at institution/unit level Must have mechanisms and strategy in place that support project-level activities.

How can impact be demonstrated?

What is the panel looking for? Clear Evidence. Assessment must be based on what is on the paper. This involves three key aspects: Quality of underpinning research: the 2* threshold Significance and reach of impact: the evidence Compelling narrative linking research to impact

Key questions considered by panel in assessing case studies: What was the research activity: a person, a project, or a programme? Was research actually carried out at the university? Was the research carried out within the time period allowed? Did the impact occur within the time period allowed? Did the research actually contribute to the impact? Is there supporting evidence for the impact?

Issues in considering Impact Statements Is an impact strategy demonstrated? Are institutional support mechanisms actually taken up? Relation of case studies to Impact Statement. Are they examples of a successful application of strategy, examples that have informed the development of a strategy, or serendipitous successes from which something can be learned.

Problems apparent in the assessment Impact assessment was based on a small number of elements, so results were ‘lumpy’ compared with outputs. This affected the overall assessment profile What may change for 2020 Impact weighting unlikely to go down More metrics to help avoid the ‘lumpy’ distribution Panel configuration – key issues about which panel to submit to.