Impact workshop Phil Hannaford VP Research and Knowledge Exchange

Agenda Introduction Overview HEFC Case Study Template Impact case studies- examples of best practice Next steps

Overview

Purpose of the REF The REF replaces the RAE as the UK-wide framework for assessing research in all disciplines. Its purpose is: To inform research funding allocations by the four UK HE funding bodies (approximately £2 billion per year) Provide accountability for public funding of research and demonstrate its benefits To provide benchmarks and reputational yardsticks

Key changes since the 2008 RAE Inclusion of assessment of impact Fewer UOAs/panels, operating more consistently Strengthened equality and diversity measures Revised eligibility criteria for staff Addition of (limited) use of citation data in some UOAs Removal of esteem as a distinct element Revised approach to environment and data collection Increased user input on panels; and an integrated role for additional assessors Publication of overall quality profiles in 1% steps

The assessment framework: Overview Overall quality Outputs Maximum of 4 outputs per researcher Impact Impact template and case studies Environment Environment data and template 65% 20% 15%

Timetable 2011 Panels appointed (Feb) Guidance on submissions (Jul) Draft panel criteria for consultation (Jul) Close of consultation (5 Oct) 2012 Final panel criteria and methods (Jan) HEIs submit codes of practice (final deadline Jul) Requests for multiple submissions (final deadline Dec) Survey of submission intentions complete (Dec) 2013 Launch REF submissions system (Jan) Recruit additional assessors Staff census date (31 Oct) Submissions deadline (29 Nov) 2014 Panels assess submissions Publish outcomes (Dec)

Impact- the big unknown

Impact- the big unknown

Impact- the big unknown

Impact- the big unknown

Impact: Submissions Impact template (REF3a) Sets out the submitted units general approach to enabling impact from its research One template per submission – with a page limit depending on the number of staff submitted Covers the period 1 Jan 2008 to 31 Jul 2013 Contributes 20% to the impact sub- profile Case studies (REF3b) Specific examples of impacts already achieved, that were underpinned by the submitted units excellent research conducted between 1 Jan 1993 to 31 Dec 2013 Number required depends on the number of staff submitted (2 up to FTE, plus 1 for every extra 10 FTEs (or part thereof)) Impacts during 1 Jan 2008 to 31 Jul 2013; underpinned by research since 1 Jan 1993 Contributes 80% to the impact sub- profile

Impact: Definition for the REF (1) An effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia Impact includes an effect, change or benefit to: - The activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding - Of an audience, beneficiary, community, constituency, organisation or individuals - In any geographic location whether locally, regionally, nationally or internationally

Impact: Definition for the REF (2) Impact includes reduction or prevention of harm, risk, cost or other negative effects It excludes impacts on research or the advancement of academic knowledge within HE; and impacts on teaching or other activities within the submitting HEI Other impacts within the HE sector, including teaching or students, are included where they extend significantly beyond the submitting HEI

Types of impact EconomicSocial Public policy & services HealthCulturalEnvironmentQuality of life Impacts: Definition for the REF

Impact: Criteria The criteria for assessing impacts are reach and significance* Four star Outstanding impacts in terms of their reach and significance Three star Very considerable impacts in terms of their reach and significance Two star Considerable impacts in terms of their reach and significance One star Recognised but modest impacts in terms of their reach and significance Unclassified The impact is of little or no reach and significance; or the impact was not eligible; or the impact was not underpinned by excellent research produced by the submitted unit * Each main panel provides a descriptive account of the criteria

Reach – How widely felt it was Significance – How much difference it made to beneficiaries

Impact: Template (REF3a) The units approach to enabling impact from its research: - Context for the approach - The units approach during Strategy and plans for supporting impact - Relationship to the submitted case studies Provides additional information and context for the case studies, and can take account of particular circumstances that may have constrained a units selection of case studies To be assessed in terms of the extent to which the units approach is conducive to achieving impact of reach and significance

Impact: Case studies (REF3b) In each case study, the impact described must: - Meet the REF definition of impact - Have occurred between 1 Jan 2008 and 31 July 2013 (can be at any stage of maturity) - Be underpinned by excellent research (of at least 2* quality) produced by the submitting unit between 1 January 1993 to 31 December 2013 Submitted case studies need not be representative of activity across the unit: pick the strongest examples NB. Its not about the esteem or influence of an individual or unit

Impact: Case studies (REF3b) Each case study is limited to 4 pages and must: - Describe the underpinning research produced by the submitting unit - Reference one or more key outputs and provide evidence of the quality of the research - Explain how the research made a material and distinct contribution to the impact (there are many ways in which this may have taken place) - Explain and provide evidence of the nature and extent of the impact: Who/what was affected? How were they affected? When? - Provide independent sources that could be used to verify claims about the impact (on a sample audit basis)

Impacts may be at any stage of development or maturity Impacts stay with the institution (unlike publications)- so cannot buy-in impacts Can be shared with other insitutions (each has to show their disticntice contribution to the imapct) Impacts must have taken place during the assessment period (not future or potential impacts) Impacts or benefits arising from engaging the public with the submitted units research will be eligible (but not dissemination activity unless there is evidence of its benefits) Impacts arising from public engagement must show that that the engagement activity was at least in part based on the submitted units research AND drew materially and distinctly upon it Impact: other key points

Clinical Medicine (17 case studies) + Earth Systems (4 case studies) Writing team Steering Group (SG) of senior academics chaired by VP R&E Process was: Trawl for stories Interviewed researchers Glasgows pilot experience

Labour intensive for the staff involved collection and collation of the material we need to submit the iterative nature of the drafting process Glasgows pilot experience

External supportive evidence Challenging to engage external contacts: - who? - Sorry they left a few months ago... - S – t – r – e – t – c – h – i – n – g Goodwill! Glasgows pilot experience

External supportive evidence Challenging to engage external contacts: - who? - Sorry they left a few months ago... - S – t – r – e – t – c – h – i – n – g Goodwill! Glasgows pilot experience NB. We need to think when to engage with external supporters

Glasgows pilot experience Possible types of evidence Testimonials from named individuals Press coverage Guidelines/Documents/Reports Training materials Details of conference/invitations to speak Links to relevant background information Public engagement – speak to the organiser of the event Details of grants Publications – highlighted where peer-reviewe

Glasgows pilot experience Avoid subjectivity Striking the balance Once upon a time...

Glasgows pilot experience Reading group External reading team from user community Clinical medicine UoA Users and academics Extremely helpful Internal reading group

Glasgows pilot experience Selecting our case studies Pipeline – extra stories Look into the past Consider reach and significance Pilot panel reports Refer to panel criteria Furthest along the pathway Believable Choose your strongest

Glasgows pilot experience Our results Very pleased with the overall result – there were no surprises Managed to avoid: Generalised, vague claims Excessive publication lists or web references Lack of coherence Claiming potential impact Lack of necessary information

Glasgows pilot experience Top four tips... Watch the template limits Make it easy for the reader So what? Start with the impact

Glasgows preparations Mini-REF - best examples of impact UoA Pipelines created (Colleges/R&E) Prioritising pipelines (UoA Champions/R&E) Developing case studies with academics Horizon scanning

Useful resources HEFCE REF2014 site Pilot panel reports and best practice examples HEFCE REF FAQs Public Engagement – NCCPE Materials Your colleagues (in Aberdeen and outside)

Read the guidance !!!!! Assessment framework and guidance on submissions Panel criteria and working methods

Case study template (REF 3b)

Examples of good practice

Preparing an impact case study

Preparing an impact case study Suggested questions to help clarify impact: What user groups outwith academia did you work with? What was the purpose of the interaction? What has been the effect on the users/audience? Did it change something for them? How did they benefit from the interaction? How did your contribution effect the impact/benefit?