1. The REF impact pilot findings Chris Taylor, Deputy REF manager

2. … maintaining the capacity of higher education to undertake world-leading research across a range of academic disciplines, promoting economic growth and national well-being and the expansion and dissemination of knowledge REF Consultation September 2009/38 REF: A UK-wide framework for assessing research quality ______________________________________________________________________________________________________________________________

3. The purpose of the REF ______________________________________________________________________________________________________________________________  to secure the continuation of a world-class dynamic and responsive research base in the UK through -funding: selective funding allocations informed by quality assessment -benchmarking and information: establishing reputational yardsticks -accountability: demonstrating that public investment in research is effective and delivers public benefit

4. Timetable ______________________________________________________________________________________________________________________________ 2010 Initial decisions (Mar 2010) Impact pilot Recruit panels 2011 Guidance on submissions (Jul 2011) Panel criteria and methods (Jan 2012) 2013 Submissions (Nov 2013) Recruit additional assessors 2014 Assessment Publish outcomes (Dec 2014)

5. The REF Framework ______________________________________________________________________________________________________________________________ Overall excellence profile Outputs (60%?) Maximum of 4 outputs per researcher Impact (25%?) Case studies (1 per 10 staff?) Environment (15%?) Narrative template + income and student data

6. Impact ______________________________________________________________________________________________________________________________  The aim is to identify and reward the contribution that high quality research has made to the economy and society:  Making these explicit to the government and wider society  Creating a level playing field  Encouraging institutions to achieve the full potential contribution of their research in future

7. Our initial proposals  Assessment through expert review  Historical impact, not predicting future impact Impact of the unit’s research, not individual researchers Underpinned by high quality research of all types Minimum burden necessary to enable robust assessment A wide view of ‘impact’ beyond academia, inclusive of all disciplines

8. Types of impact EconomicSocial Public policy & services HealthCulturalQuality of lifeInternationalEnvironment A wide view of impact ______________________________________________________________________________________________________________________________

9. The impact pilot exercise ______________________________________________________________________________________________________________________________  The pilot tested and developed a case study approach to assessing the impact of research  Five units of assessment (UOAs)  29 UK higher education institutions each submitting to 2 UOAs  Each submission included: -An ‘impact statement’ for the submitted unit as a whole -Case studies illustrating examples of impacts achieved (a total of one case study per 10 research staff)  Impacts that occurred during 2005-09, underpinned by research since 1993

10. The pilot panels ______________________________________________________________________________________________________________________________  Membership drawn from academia and research users from the private, public and third sectors  Tested the methodology by:  Assessing the case studies in terms of ‘reach and significance’ of the impacts  Considering the wider ‘impact statements’  Producing impact profiles  Identifying issues and how to improve the process

11. Pilot reports ______________________________________________________________________________________________________________________________  Publications on www.ref.ac.uk:www.ref.ac.uk  The findings of the five pilot panels  Feedback from the 29 pilot HEIs (by Technopolis)  Examples of good practice case studies  A summary of workshops to explore impact in the arts, humanities and social science  Guidance documents used in the pilot exercise

12. Benefits of research ______________________________________________________________________________________________________________________________ Impacts on patient outcomes, health policy and practice, medical technology and the pharmaceutical industry Clinical Medicine Impacts on high-tech products and services, public engagement with science and defence and energy policy Physics Impacts on environmental policy, conservation, managing the environmental, utilities, risks and hazards, exploration of resources, public health Earth systems & environmental sciences Impacts on social policy, public services, third sector, practitioners and public debate Social work & social policy Impacts on creative industries, cultural enrichment, civil society, English as a global product, policy development English language & literature

13. Clinical Medicine 4*3*2*1*U UOA average17253412 Institution A04035250 Institution B2510301520 Institution C040 10 Institution D0554500 Institution E204525010 Institution F25 0 Institution G25304500 Institution H2025 1020 Institution I0075250 Impact profiles ______________________________________________________________________________________________________________________________

14. Key findings ______________________________________________________________________________________________________________________________  The process makes explicit the benefits that research in each discipline brings to society  It is possible to assess the impact of research, through expert review of case studies  A number of refinements are needed for full implementation  A generic approach is workable, with scope for REF panels to tailor the criteria as appropriate to their disciplines  The weighting should be significant to be taken seriously by all stakeholders

15. Defining impact ______________________________________________________________________________________________________________________________  There should be a wide definition of (non academic) impact  Interim or early stage impacts should count, assessed on the same criteria and not by anticipating future potential impact  Impacts through public engagement should count so long as they are based on the unit’s research and the benefits are articulated  REF panels should develop further guidance about what constitutes impact from research in their disciplines (this should not be restrictive)

16. Case studies ______________________________________________________________________________________________________________________________  Panels were able to differentiate between case studies using the 5-point scale to produce impact profiles  Improvements would help panels to make robust judgements: –Revising the template to encourage a more natural narrative –Case studies should contain all the information needed by panels to make judgements –References should be for verification purposes only –Supporting indicators should be contextualised and meaningful to the particular case –The focus could be broad or narrow, so long as the narrative is coherent and there is evidence of specific impacts

17. Submissions ______________________________________________________________________________________________________________________________  1 case per 10 staff provided an appropriate range of evidence, though further consideration of very small units is needed  As well as assessing case studies, panels want to know how the unit/institution supports impact. This should be a distinct section of the environment element, replacing the ‘impact statement’

18. Assessment criteria ______________________________________________________________________________________________________________________________  The criteria and level definitions (with some refinements) should be widely applicable, with flexibility for REF panels to interpret them as appropriate  Panels gave credit wherever the HEI’s research made a distinctive contribution to the impact: –It should not be essential for the HEI to be involved in exploitation –Where the impact depended on a wider body of research the submission should acknowledge this  The quality threshold for the pilot was appropriate and it will be vital ensure high quality for the REF. Cases should cite only directly relevant research and justify their quality.

19. Other issues ______________________________________________________________________________________________________________________________  A timeframe of 15 years from research to impact is broadly appropriate – some disciplines may need to extend this  Cases should provide independent sources that could verify specific claims, to be followed up on a sample basis  Cross referral should remain an option but it is preferable for panels to assess the material submitted to them  It is essential to include research users on the panels and their time will need to be used effectively  Scaling up from the pilot will have implications for the panels that will need to be managed

20. Timetable ______________________________________________________________________________________________________________________________ Nov 2010Publish reports Nov-Dec 2010Discussions Mar 2011Publish decisions on broad framework for assessing impact Jul 2011Guidance on submissions Autumn 2011Panels consult on criteria Jan 2012Publish panel criteria 2012-2013HEIs make submissions 2013Recruit additional research users to assess impact 2014Assessment

21. Art and design – workshop/issues ______________________________________________________________________________________________________________________________  Workshop on practice-based creative and performing arts held on 1 June 2010.  Attendees nominated by subject bodies  Short case studies looked at included…  Film research in Columbia informing policy.  Commercial and cultural benefits from a partnership between performance academics and a digital arts company.  Improved visitor and audience experiences from the use of music research at a museum exhibition.

22. Art and design - workshop ______________________________________________________________________________________________________________________________  The case study approach seems workable but subject- specific guidance crucial and there will be behavioural consequences  Audiences?, users?, beneficiaries?  The nature and range of benefits and impacts (cultural, quality of life, policy?)  Evidence and indicators: some challenges….. but look at English annexe of impact report: Stimulating creativity and cultural excellence, Promoting education and learning, Advancing the UK’s nations, regions and communities, Bringing the UK to the world and the world to the UK, Sustaining citizenship and civil society.