1. Demonstrating the impact of UK e-Research; a research council perspective Drs. Astrid Wissenburg, ESRC Workshop 11: Profiling UK e-Research: Mapping Communities and Measuring Impacts UK e-Science ALL HANDS MEETING 2008

2. Demonstrating the impact of UK e-Research; a research council perspective Definitions Context and drivers Current approaches - examples Outstanding issues

3. What is e-Research? “e-Research refers to the development of, and the support for, information and computing technologies to facilitate all phases of research processes. The term e-Research originates from the term e-Science but expands its remit to all research domains not just the sciences. It's concerned with technologies that support all the processes involved in research including (but not limited to) creating and sustaining research collaborations and discovering, analysing, processing, publishing, storing and sharing research data and information. “ (JISC)

4. What is (economic) impact? (1) “An action or activity has an economic impact when it affects the welfare of consumers, the profits of firms and/or the revenue of government. Economic impacts range from those that are readily quantifiable, in terms of greater wealth, cheaper prices and more revenue, to those less easily quantifiable, such as effects on the environment, public health and quality of life. “ (Treasury) “improvements for UK society and the economy” (RCUK)

5. What is (economic) impact? (2) “The concept of 'impact' in the social sciences applies to all sectors: public, private and voluntary. It embraces economic and societal impact in the sense of direct and often quantifiable economic benefits; wider social impacts that will benefit society more generally such as effects on the environment, public health or quality of life; and impacts on government policy, the third sector and professional practice. (….) Economic and societal impact builds on academic impact: developing social science knowledge and methods.” (ESRC)

7. Impact aspects Academic and non-academic On all sectors: public, private, voluntary – and general public Quantifiable and non-quantifiable Direct and indirect Policy, practice, products

8. Strategic drivers for impact agenda Government’s Science and Innovation investment frameworks emphasize the maximisation of public investment in science on the economy Research councils have been challenged to demonstrate an increase in their economic impact Major research challenges are interdisciplinary, and require collaboration and co-production to ensure quality research with impact

9. RCUK Impact Mission* To advance knowledge, understanding and technology, and provide trained researchers; To build partnerships that enhance take-up and impact, thereby contributing to the: economic competitiveness of the United Kingdom, effectiveness of public services and policy, and enhancement of the quality of life and creative output of the nation. *derived from the Royal Charters of the Research Councils

10. RCUK expectations for societal and economic impact (1) Include: identify potential benefits and beneficiaries from the outset, and through the full life cycle of the project(s) maintain professional networks that extend beyond discipline and research community publish results widely – considering the academics, user and public audiences for research outcomes

11. RCUK expectations for societal and economic impact (2) exploit results where appropriate, in order to secure social and economic return to the UK manage collaborations professionally, in order to secure maximum impact without restricting the future progression of research take responsibility for the curation, management and exploitation of data for future use For full document see: www.rcuk.ac.uk/cmsweb/downloads/rcuk/innovation/expectationssei.pdf

12. Purposes of identifying and measuring impact To learn lessons, specifically to understand, improve and support the processes which create impact To make the case for the science budget To inform funding decisions

13. Identifying and measuring impact Growing portfolio of evaluation methodologies to ‘measure’ economic and societal impact: Across all types of ‘research activities’ – including e- Research At all levels: national science budget, activity type, funding mode, programmes, individual projects Across the whole life-cycle of activities, including pre and post

14. Example 1: RCUK Economic Impact study (SQW/PA, 2007) Series of 18 case studies across training, research and facilities investments from all research councils. Methodology: stakeholder interviews was supplemented by data and secondary research Used a classification of impact types: Development of human capital Business and commercial Policy Quality of life

15. Example 1: RCUK study - impacts Examples of impacts from case studies include: Use of scientific facilities by industry: direct income through selling access indirect impact through businesses' exploitation of knowledge that is developed by using the facilities. Software development and licensing revenue Spin-off companies

16. Example 1: RCUK study - findings A wide diversity of impacts emerging from research funding, many not part of the original rationale for the specific investment Not possible to robustly extrapolate from the case study findings to total spend of research councils, partly because the methodology is resource intensive and expensive Lack of ‘management information’ about outputs from investments, especially post project.

17. Example 2: DIUS study on large scale facilities (2008) Review of economic impacts relating to the location of large-scale science facilities in the UK Five facilities covered Analysis of economic impacts arising from employment, expenditure and knowledge transfer and scientific benefits Focus on benefit of the location of facilities in the UK as opposed to access to similar facilities abroad.

18. Example 2: DIUS study on large scale facilities (2008) The major economic impacts arise from: employment of relatively highly paid staff, most of whom reside close to the facility the awarding of contracts to UK-based suppliers Plus: contributing to local technology clusters but generally a relatively small component Some examples of transferring knowledge and technologies to suppliers, but no evidence that this is on a substantial scale

19. Example 3: ESRC policy and practice impact evaluations Case study approach intended to cover different investments and different methodologies, including: “Payback” method: this method examines the stages of a project throughout its life, and categorises its “paybacks”. "tracking forward" from research outputs to the way in which these have been incorporated into practice in the public and private sectors. an analysis of end of award reports to assess the potential for impact, followed by an analysis based on "unobtrusive" or "non- reactive" measures involving web-based searches, and databases of publications a logic chain, mapping resource inputs to shorter term research outputs and to longer term outcomes.

20. Example 3: ESRC policy and practice impact evaluations qualitative studies with mixed methods captures the complexities and allows for triangulation models that anticipate the communication flows between researchers and potential users provide a framework for analysis, but linear models assume a trail of evidence which is seldom present

21. Example 3: ESRC policy and practice impact evaluations – next steps Economic impact study of 2 Research Centres: Identify achieved economic impacts (through research, dissemination, networking, etc); Apply appropriate techniques to calculate values for the economic impact of selected aspects Present the results within a broader analysis of the Centres’ policy and practice impacts; Identify lessons for future economic evaluations

22. Methodological issues How can we address the problems of Attribution Traceability Measurability Time lags? How can we move beyond inputs and outputs as proxies for actual impact? How do we represent the receptiveness of users?

23. Research funders issues Can/should we establish any robust baseline? What is the relationship between individual projects to programmes to investment portfolios to the science budget? What we can do: More systematic data gathering of output data and ‘impact’ stories, during and post projects More testing of economic impact valuations Extract the lessons about good pre-conditions and practice for impact creation

24. E-Research Issues Are the current approaches applicable to assess the impact of e-Research? Understand pathways to impact, both direct and indirect, from e-Research Impact of e-Research on the research process itself and the implications for impact measurement Can some e-Research tools be sued as part of a portfolio to collect and analyse ‘impact’ information

25. Thank you astrid.wissenburg@esrc.ac.uk