1. Collection and analysis of research outputs and impacts Dr Ian Viney Medical Research Council

2. Provide evidence for communicating the outcomes and impacts of medical research (economic, societal and academic) Quantitative and qualitative information used to develop/strengthen strategic decisions (what worked, what didn’t) Understand better how research leads to impact (how can we maximise the opportunities for impact?) MRC Evaluation Programme

3. InputOutput/OutcomeImpact Improvements to health (living longer and with better quality of life) Attracting R&D investment (from global business and non-UK funding sources) Improving the performance of existing businesses Creating new businesses (that contribute to economic growth and further R&D) Delivering highly skilled people to the labour market Academic impact (effects on further research including other disciplines) Improving public policy and public services Evaluation Framework Funding for Research and Training Generation of new knowledge Leveraging funding Dissemination of research Intellectual property/ licensing Development of new products/processes Changes to policy Development of collaborative networks Research materials

4. MRC Vision for increasing impact MRC Delivery plan 2011/12 – 2014/15

5. Compiling the evidence Tracing the interactions between the producers and users of research, establishing evidenced links across as many steps as possible Publications may have a minor impact in transferring knowledge to industry, people exchanges and problem solving (collaboration) are more important Similarly focussing on hard commercialisation metrics (spin outs, patents etc.) miss a large range of activities (e.g. product development, policy influences) Need a holistic approach to output, challenge is to capture these “softer” processes effectively Failures (absence of output, closure/re-direction of projects) also have to be factored in Advantage to this “micro-economic” approach is that casual links are established, disadvantage is that these links are rarely “generalisable” Has to be gathered from researchers, either by employer or funder – there are no repositories for much of this data Need to keep administrative burden as low as possible For medical research in particular, research programmes benefit from a variety of funding sources, are highly collaborative, and progress over the long-term. Return on investment studies are extremely difficult

6. MRC e-Val MRC e-Val was an online database used to gather outputs/outcomes/impacts arising from MRC research, it was launched in 2008, and we asked researchers to record output realised since 2006 Over six years we have learnt a lot about asking the right questions, validating, analysing, and presenting the data MRC has built a robust dataset, covering now 7 years of output, which allows progress/productivity and quality of MRC research to be analysed, and this evidence to be utilised in decision making Need for both quantitative and qualitative information, e.g. to “set case studies in context” of the productivity and quality of all MRC output A requirement for comprehensive data across the whole MRC portfolio against which progress and trends can be prospectively tracked over time

7. “Federated” e-Val – multiple funding agencies, one route for gathering output data MRC licensed e-Val approach to Researchfish Ltd. Built a single portal for researchers to record the output from their research across multiple funders (Researchfish.com), latest web technology and cloud hosting e-Val provides the structure to capture outputs systematically, using a proven approach, each funder receives the information they need and there is the opportunity to collaborate on cross-funder evaluation System is flexible and appears to be equally applicable to disciplines outside of medicine (Science and Technology Facilities Council also use the approach) 20 funding organisations initially subscribed to Researchfish with a combined annual research spend of £1.8bn 12 Research organisations (11 Universities) have also subscribed to Researchfish, obtaining enhanced real-time reports on progress across multiple funding organisations

8. Progress to date? MRC dataset now covers outputs realised 2006 – 2012, from 98% of our portfolio (£2.5bn expenditure across 4,500 awards, 3,500 researchers) 6,500 researchers are now using the federated system and this is rapidly growing. Scottish Health Department, STFC, BHF, Arthritis Research UK are examples of funders that have accumulated significant datasets, some over multiple data gathering exercises. A further 50 UK organisations are now preparing to gather data, this will increase the investment tracked to £3bn per year (almost all medical research funding in the UK) We have used the data to easily compile case studies, and to set these in the context of the progress, productivity and quality of the whole MRC portfolio Researchfish now exchanges data with Europe PubMed Central, enhancing information about the links between funding and papers – useful for open access monitoring

9. Characteristics of the Researchfish system Attribute outputs to multiple funders, large and growing user base Format data into CVs Enter data once (Re-use widely) Researchers guided through a highly structured form Forms have standard data to select from Extensive “in form” validation High quality data “Delegate” colleagues to assist with data entry Work in research teams Researchers work collaboratively Comprehensive dashboard reports Real time reporting Complete service Funders obtain timely data Easily share Researchfish data between organisations (other funders or universities) Joint analysis Any organisation size, research discipline, number of users Customisation of questions Flexibility

10. Output types Guidance and form for entering data Grant portfolio

11. Predictive lookup function within Researchfish, driven by large datasets of funding agencies and collaborators – easy for researchers to fill in and data is highly standardised

12. Reporting new products/interventions in development Categorisation for products and interventions based on NCI translational research flowcharts and US DoD technology readiness levels Simple flow charts, unambiguous categories, allow the data to be structured and reported consistently Automated “lookups” suggest categories to researchers – no keying in of organisation/funder names etc.

13. MRC use of Researchfish data spending review submissions to Government (2010, 2013), medical research treated positively published and returned data each year to Universities and MRC Units, some of which are ingesting this into their research information systems used to adjust/make the case for new funding initiatives (e.g. a further £20m per year investment in experimental medicine - funding underway) 2012 quinquennial review of all three MRC Institutes included bibliometric data, providing an international comparison with other research organisations. The UK Government has provided additional science funding (e.g. £90m of the £180m biomedical catalyst fund announced in 2011) referring directly to the MRC data. MRC has monitored progress against the objectives set out in its Strategic Plan (Research Changes Lives) using outcomes compiled from Researchfish data. These convey good progress since 2009, http://www.rclprogress.mrc.ac.uk/ http://www.rclprogress.mrc.ac.uk/

14. Headline Quantitative Results (2012 update) MRC research produces high quality output. Annual analysis of citations from MRC publications has shown that MRC reported papers have consistently more than twice the world average impact. 110 new products or interventions have been launched onto the market (or are undergoing market authorisation), as a result of MRC research, since 2006. The dataset now includes reports of 686 projects developing such products at all stages. Reports of more than 300 influences on clinical guidelines worldwide, including 50 NICE guidelines. In the main these involve impacts on recommendations in these guidelines, and are a subset of 3,300 reports of all types of policy influence. 579 new patents have been reported arising in part as a result of MRC research since 2006, 180 of these (31%) have already been licensed. We have compiled evidence of significant impact on the establishment or growth of 104 companies, representing at least 535 new skilled UK jobs. 56 spin out companies were launched since 2006 with significant input from MRC research. There are substantially more productive links to established UK and global companies. http://www.mrc.ac.uk/Achievementsimpact/Outputsoutcomes/Researchfish2012/index.htm

15. Wellcome Trust Project Grants 2006 Developed Professor Edward Watkins research independence. NIHR South-West CLAHRC (2007) Provision of institutional programmatic support to support the further translation of this work into a viable treatment. Total value of Peninsula CLAHRC £10m Basic Research Applied Research Healthcare Practice & Policy 1st translational gap2nd translational gap MRC Experimental Medicine Grant (2007) “Cognitive training as a facilitated self-help intervention for depression” Allowed for growth of research team and shift in focus from experimental research to evaluation of therapy via RCT. £464k MRC Brain Sciences Trial Platform Grant (2005) “Preventing depression relapse in NHS practice using Mindfulness-Based Cognitive Therapy" £237k Mood Disorders Centre (University of Exeter): Cognitive therapy for depression Outcomes Proof-of-principle evidence that training depressed individuals to think about their problems in a more concrete way reduces their symptoms of depression. Pilot work in a multiple case series has demonstrated in patients with major depression that is safe, tolerable, and acceptable. The final stages of the current MRC grant have tested out the efficacy of the intervention in a small scale controlled trial (n =121). The results of the main outcome analysis indicate that the cognitive training guided self-help treatment adds significant clinical benefit in terms of reducing depression relative to treatment-as-usual alone, and that it is of equivalent efficacy to a relaxation based guided self-help treatment in reducing depression, but superior in changing depressogenic thinking such as rumination. The project has therefore developed a novel treatment derived from experimental findings that is easily accessible, cost-effective and able to be disseminated, and could be made widely available (for example via Improving Access to Psychological Treatments IAPT initiative). This treatment material has been copyrighted. The report of the main outcome of the trial has now been prepared as a paper and submitted to a high impact peer-reviewed journal for publication. Collaborations with researchers in the Netherlands (2009) has extended this work to a new population (adolescents at risk) and to a prevention treatment (rather than an acute treatment). The work has been cited in the NICE guideline update on depression (2009) Wellcome Trust Capital Grant to build the new Sir Henry Wellcome Mood Disorders Centre at Exeter (due to open in 2011) £3.6m Health Technology Assessment (2010) “Preventing depressive relapse in NHS Practice through mindfulness-based cognitive therapy (MBCT)” £2.1m Mood Discorders Centre Founded 2003 by Professor Edward Watkins and Professor Willem Kuyken Discovery Clinical trial New treatment High impact paper Policy influence New collaboration Intellectual property Overall, the total cost of health services for depression in England in 2007 was estimated to be £1.7 billion whilst lost employment increased this total to £7.5 billion. By 2020 it is expected that depression will be the second leading cause of disability in the world, and by 2026 the condition is projected to cost the UK health service £3 billion (£12.2 billion cost to the UK with lost employment). Pipeline analysis One of 40 therapeutic behavioural interventions in development 10 of which are in early “clinical assessment” 3 of which have relevance to treating depression Prospective tracking provides information on the speed of translation into practice

16. Case studies easily compiled Screening by the Cytosponge followed by treatment of patients with dysplasia or intramucosal cancer would reduce the number of cases of incident symptomatic esophageal adenocarcinoma by 19%, compared with 17% for screening by endoscopy. 2012 paper

17. Co-publication with the private sector - Bibliometrics The MRC e-Val dataset includes reports of all publications arising as a result of MRC funded research since 2006, currently this covers 50,000 reports In collaboration with Thompson Reuters the MRC has analysed the available information on the location of all authors on these papers (currently 500,000 data points) Our preliminary analysis shows co-authorship with industry increasing from 2006 – 2011 from 5% to nearly 7% of papers, and suggest that this is the result of MRC policies which have encouraged interactions with the private sector. We expect this to rise further as data from 2011 is partial. In the “International comparative performance of the UK research base (2011)” it was stated that UK academic/private sector co-authorship across all disciplines was low (approximately 1.3%) The data to the left corresponds to MRC researchers co-authoring with at least 720 organisations, approximately 30% of which are UK-based

18. Location of co-authors - Europe

19. Location of co-authors - UK

20. Challenges Researchfish successfully draws information from other datasets, such as PubMed (IPO and clinical trial registry data is in test). Where there are unique identifiers data quality can often be validated, and importantly outputs can be de-duplicated. Publishing and exporting data is straightforward. More Researchfish information will be displayed in the RCUK gateway to research, Europe PMC displays Researchfish data, some research organisations are ingesting the data etc. Challenge is to “synchronise” data with other systems, identifying the same/similar output and if appropriate merging records – clearly a role for CERIF to continue to be developed to encompass all output types

21. Next Steps Continuing to capture information on outcomes in an efficient way across all scientific disciplines, from medicine to the arts/humanities Address areas of under-reporting e.g clinical trials Maintaining the support of the research community (good communications, regular feedback, build on success), allow re-use of the data e.g. in CVs and most importantly grant applications Continue to extend the approach to acknowledge and involve other research team members (“research teams” functionality now live). Collaborators/co-investigators now enter output once and share it. Outputs are uniquely identified. In 2013 7,000 MRC co-investigator records have been uploaded. Continue to improve reporting function

22. Thank you Email – evaluation@headoffice.mrc.ac.ukevaluation@headoffice.mrc.ac.uk Web - http://www.mrc.ac.uk/Achievementsimpact/Outputsoutco mes/e-Val/index.htm http://www.mrc.ac.uk/Achievementsimpact/Outputsoutco mes/e-Val/index.htm Twitter – @MRCeVal

23. Funding organisations Universities Academy of Medical Sciences University of Aberdeen Alzheimers Research UK Cardiff University Arthritis Research UK University of Dundee Asthma UK University of Glasgow British Heart Foundation Imperial College London Cancer Research UK Newcastle University Chief Scientist Office (Scotland) University of Nottingham Epilepsy Research UK University of Oxford Great Ormond Street Hospital Children’s Charity University College London Kidney Research UK Medical Research Council NISCHR (Wales Health Department R&D) HSC Public Health Agency (Northern Ireland Health Department R&D) Parkinson’s UK Science and Technology Facilities Council Stroke Association Tenovus Organisations subscribing to Researchfish (February 2013) Total annual research expenditure of funding organisations = £1.8bn

24. Researchfish “output types” Section Output type 1Publications 2Collaborations 3Further funding 4Next destination 5Engagement activities 6Influence on Policy, Patients and the Public 7Research Materials (Tools, Methods, Databases) 8Intellectual property and licensing 9Development of products or interventions 10Impacts on the private sector 11Awards and recognition 12Use of Facilities 13Other outputs/future steps