1. Making an Impact A Preferred Framework and Indicators to Measure Returns on Investment in Health Research Full Report available at http://www.cahs-acss.ca/e/assessments/completedprojects.php http://www.cahs-acss.ca/e/assessments/completedprojects.php Canadian Academy of Health Sciences Académie canadienne des sciences de la santé scientific advice for a healthy Canada

2. Table of Contents Background The Report  Impact categories and Impacts  Three case examples  Recommendations 2

3. Background 3 scientific advice for a healthy Canada

4. CAHS: Who We Are  Created in 2004  Non-profit charitable organization  One of three founding member academies  Unique collaboration of 6 health disciplines and the full spectrum of academic health sciences  Elected Board (13) plus >260 Fellows 4

5. scientific advice for a healthy Canada What is CAHS?  A new collaborative body  Multidisciplinary, accomplished health scientists  Not an advocacy group  “Scientific advice for a healthy Canada”  Publications including this report on CAHS web site: http://www.cahs-acss.ca 5

6. scientific advice for a healthy Canada Scientific Advice for a Healthy Canada  Tackle urgent and complex problems  Use knowledge to address apprehension  Congregate the best minds  Assemble the best science  Listen, deliberate, debate  Provide the best advice Unbiased Unbiased Non-vested Non-vested Balanced Balanced Feasible Feasible 6

7. Dec-06 Jan-07 Feb-07 Mar-07 Apr-07 May-07 Jun-07 Jul-07 Aug-07 Sep-07 Oct-07 Nov-07 Dec-07 Jan-08 Feb-08 Mar-08 Apr-08 May-08 Jun-08 Jul-08 Aug-08 Sep-08 Oct-08 Nov-08 Dec-08 Jan-09 CAHS Executive or Board Assessment Panel and staff Assessment Standing Committee Initial literature and expertise searches, drafting prospectus, fundraising, communication to sponsors, forum planning, financial management Select & appoint chair Refinement of prospectus, approval of question, define expertise appoint panelists Sponsor’s input 7 commissioned papers Progress report sponsors Approval, publication preparation, translation, meeting planning, dissemination External review Reviewed recommended approval Recommend & Recruit External Reviewers Sponsors External Reviewers Others Meeting Invite panelists, refine question, comprehensive literature and expertise searches, panel meetings, draft assessment and related information, interview experts, identify areas where information is lacking, commission papers to fill gaps, obtain consensus on recommendations, address issues raised by external reviewers, help disseminate report Evolution of Return on Investment Assessment 7

8. scientific advice for a healthy Canada ROI Sponsors Major Sponsors  Canadian Health Services Research Foundation (CHSRF)  Canadian Institutes of Health Research (CIHR)  Canada’s Research-Based Pharmaceutical Companies (Rx&D)  Public Health Agency of Canada (PHAC)Sponsors  Alberta Heritage Foundation for Medical Research (AHFMR)  Association of Canadian Academic Healthcare Organizations (ACAHO)  Association of Faculties of Medicine of Canada (AFMC)  BIOTECanada  Canadian Agency for Drugs and Technologies in Health (CADTH)  Fonds de la recherche en santé du Québec (FRSQ)  Government of Ontario, Ministry of Research and Innovation; Ministry of Health and Long- Term Care  Heart & Stroke Foundation of Canada (HSFC) Sponsors (continued)  Manitoba Health Research Council (MHRC)  Michael Smith Foundation for Health Research (MSFHR)  National Cancer Institute of Canada (NCIC)  Nova Scotia Health Research Foundation (NSHRF)  Ontario Neurotrauma Foundation (ONF)  Saskatchewan Health Research Foundation (SHRF)  Western Economic Diversification Canada (WD)Contributors  Canada Foundation for Innovation (CFI)  Canadian Association of Schools of Nursing (CASN)  Canadian Medical Association (CMA)  Canadian Nurses Association (CNA)  Canadian Nurses Foundation (CNF)  Newfoundland & Labrador Centre for Applied Health Research (NLCAHR)  Research Canada 8

9. scientific advice for a healthy Canada Why ROI in Health Research? (1) Why ROI in Health Research? (1)  Lack of public understanding of the value of research applicability to current issues in health  Concern about accessible, affordable, high quality health care in a publicly funded system  Need to adequately measure & meaningfully convey benefits of health research to policy-makers & public  Increasingly common view that health care / health research) is a cost-driver consuming an ever greater share of resources at expense of other sectors  Concern about expenditure accountability in both the public and private sectors in Canada and abroad 9

10. scientific advice for a healthy Canada  Lack of consensus on how and when to best evaluate return on research expenditures  Questions from policy makers about tangible results attributable to recent increases in public investment in health research e.g. CIHR, CFI, CRC programs  Uncertainty about appropriateness of Canada’s health research expenditures versus those of analogous contributions in other industrialized countries  Need to acquire appropriate evidence to strike right funding balance between investigator-initiated “discovery” & targeted “strategic” health research Why ROI in Health Research? (2) Why ROI in Health Research? (2) 10

11. Mobilizing Science and Technology to Canada’s Advantage 2007 The Science and Technology Framework Vision: We will build a sustainable national competitive advantage based on science and technology and the skilled workers whose aspirations, ambitions, and talents bring innovations to life. To achieve this vision, we will create three S&T Advantages for Canada: Entrepreneurial Advantage Canada must translate knowledge into practical applications to improve our wealth, wellness, and well-being. Entrepreneurial Advantage Canada must translate knowledge into practical applications to improve our wealth, wellness, and well-being. Knowledge Advantage Canada must build upon our research and engineering strengths, generate new ideas and innovations, and achieve excellence by global standards. People Advantage Canada must grow its base of knowledge workers by developing, attracting, and retaining the highly skilled people we need to thrive in the modern global economy.

12. Mobilizing Science and Technology to Canada’s Advantage 2007 Policy Commitments Canada’s federal government will increase its accountability to Canadians by: Improving its ability to measure and report on the impact of S&T expenditures. The government will improve its understanding of Canadian S&T developments and the impact of federally performed S&T, and will work with the OCED and other countries to develop metrics that will enable comparisons against international benchmarks of success. 12

13. CAHS Standing Committee on Assessments  Andreas Laupacis, MD (Chair), Executive Director, Li Ka Shing Knowledge Institute of St. Michael'sHospital; Professor, Faculty of Medicine, University of Toronto  John A. Cairns, MD, Professor of Medicine and Dean Emeritus, UBC  Timothy Caulfield, LLM, CRC in Health Law and Policy; Professor, Faculty of Law and School of Public Health; Senior Health Scholar AHFMR and Research Director, Health Law Institute, University of Alberta  André-Pierre Contandriopoulos, PhD, Professeur Titulaire, Département d'Administration de la santé, Université de Montréal  Alastair Cribb, DVM, PhD, Dean, Faculty of Veterinary Medicine, U of Calgary  Jean Gray, CM,LLD, DSc, Professor Emeritus, Dalhousie University  Pavel Hamet, MD, PhD, CRC, Predictive Genomics; Chief, Gene Medicine Services, Centre de recherche Centre hospitalier de l’Université de Montreal  Dorothy Pringle, OC, RN, PhD, Professor Emeritus, Faculty of Nursing, U of Toronto  Matthew Spence, OC, MD, PhD, Retired President and CEO, AHFMR  Peter S. L. Tugwell, MD, CRC in Health Equity; Director, Centre for Global Health (Institute of Population Health); Professor of Medicine & Epidemiology, U Ottawa  Sharon L. Wood Dauphinee, PhD, PT, Professor, McGill University, Montreal, QC 13

14. scientific advice for a healthy Canada Steps in a CAHS Assessment  Choice of topic  Choice of chair  Choice of assessment panel  Panel’s independent work  Review of draft report by external reviewers and CAHS Assessment Committee  Revision of report  Approval by CAHS Board 14

15. scientific advice for a healthy Canada External Reviewers   Joseph B. Martin. MD, PhD, Edward R. and Anne G. Lefler Professor of Neurobiology and former Dean of the Harvard Faculty of Medicine, Harvard Medical School, Boston, MA, USA   The Honourable Michael J. L. Kirby, MA, PhD, LLD(Hon), Chair, Mental Health Commission of Canada, Senator (former), Ottawa, ON, Canada   John W. Frank, MD, CCFP, MSc, FRCP(C ), Director, Scottish Collaboration for Public Health Research andPolicy, MRC Human Genetics Unit, Edinburgh, United Kingdom 15

16. The Report 16 scientific advice for a healthy Canada

17. The Panel (1)  Cyril Frank, MD (Chair), McCaig Professor of Joint Injury and Arthritis Research; Professor, Division of Orthopaedics University of Calgary  Renaldo Battista, MD, MPH, ScD, Professor and Director of the Department of Health Administration, Université de Montréal  Linda Butler, Fellow and Head, Research Evaluation and Policy Project, Australian National University  Martin Buxton, BA, Professor, Health Economics, Brunel University, UK  Neena Chappell, PhD, Canada Research Chair; Social Gerontology, Professor of Sociology and Centre on Aging, University of Victoria  Sally C. Davies, Director General, Research and Development, Department of Health and National Health Service, UK  Aled Edwards, PhD, Banbury Professor, Banting and Best Department of Medical Research, University of Toronto  Chris Henshall, PhD, Pro-Vice-Chancellor, External Relations, University of York, UK 17

18. scientific advice for a healthy Canada The Panel (2)  Yann Joly, LLB, LLM, project manager, Centre de recherche en droit public Université de Montréal  Gretchen Jordan, PhD, Principal Member of Technical Staff, Science and Technology Strategic Management Unit, Department of Energy, Washington, DC  Terence Kealey, MB, BS, PhD, Vice Chancellor & Clinical Bio-Chemist, University of Buckingham, UK; author The Economic Laws of Scientific Research  Michael C. Wolfson, PhD, Assistant Chief Statistician, Analysis and Development, Statistics Canada  Steven H. Woolf, MD, MPH, Professor, Departments of Family Medicine, Epidemiology, and Community Health, Virginia Commonwealth University, US 18

19. scientific advice for a healthy Canada The Staff  Eddy Nason – Health Research Evaluation Analyst and Writer (Toronto) –  Larissa Sommerfeld – Research Assistant (Calgary)  Linda Marchuk – Research Associate, Admin Support and Finance Officer (Calgary)  Rhonda Kennedee – Meeting and Events Coordinator (Calgary) 19

20. scientific advice for a healthy Canada The Remit  Is there a “best way” (method) to evaluate the impacts of health research in Canada & are there ‘best metrics’ for assessing those impacts (or improving them)?  Useful to a full range of funders/research types  Compatible with what is already in place in Canada  Transferrable to international comparisons  Able to identify the full spectrum of potential impacts 20

21. scientific advice for a healthy Canada The Approach  Sponsor interviews by panelists to be certain of needs  Literature review  Expert interviews  Seven commissioned papers in areas of special interest and/or perceived gaps: Public perspective Public perspective Ethics Ethics International frameworks International frameworks Pillar II, Pillar III, and Pillar IV research Pillar II, Pillar III, and Pillar IV research ‘Meso-level’ metrics for impact ‘Meso-level’ metrics for impact  Working Groups on panel  Face-to-face meetings x 3 21

22. scientific advice for a healthy Canada Sponsor Expectations 22

23. scientific advice for a healthy Canada Different Evaluation Needs Evaluation for Accountability Evaluation for Advocacy Evaluation for Learning Comprehensive Evaluation = Our Target External audience Mission linked Identify ‘best’ 23

24. scientific advice for a healthy Canada Many Complexities For example: the (nearly) infinite number of potential evaluation questions  Evaluation questions from one funder and one program only: Have we increased the skill set of Canadian health research? Have we increased the skill set of Canadian health research? Have we increased the number of skilled researchers working in Canada? Have we increased the number of skilled researchers working in Canada? Are our trainees producing high quality research? Are our trainees producing high quality research? Are our trainees disseminating their findings to a variety of appropriate stakeholders? Are our trainees disseminating their findings to a variety of appropriate stakeholders? 24

25. scientific advice for a healthy Canada Experts Interviewed  Douglas Barber  Stefan Ellenbroek  Greg Webster  Wendy Baldwin  Egon Jonsson  Noralou P.Roos  John Cairns  Ilse Treurnicht  Sharon Manson Singer  Teren Clarke  Cheryl L. Koehn  Greg Tassey  Muhajarine Nazeem  Carol Dahl 25

26. scientific advice for a healthy Canada Selected Literature >260 articles, books, web links, etc  Buxton MJ, Hanney SR. How can payback from health services research be assessed? J Health Serv Res Pol. 1996;1(1):35-43. ******  US Senate. Joint Economic Committee. The benefits of medical research and the role of the NIH. Washington, D.C. 2000.  Nason E, Janta B, Hastings G, Hanney S, O'Driscoll M, Wooding S. 2008. Health research: Making an Impact. The Economic and Social Benefits of HRB Funded Research.Dublin:Ireland.  Oortwijn W, Hanney S, Ligtvoet A, Hoorens S, Wooding S, Grant J, et al. Assessing the impact of health technology assessment in the Netherlands. Int J Technol Assess Health Care. 2008;24(3):259-69.  Lavis J, Ross S, McLeod C, Gildiner A. Measuring the impact of health research. J Health Services Res Pol. 2003;8(3):165-170.  Funding First.; 2000. Exceptional Returns: The economic value of America's investment in medical research. New York, New York: The Lasker Foundation. ******  Cutler DM, Kadiyala S.; 1999. The Economics of Better Health: The Case of Cardiovascular Disease. New York, New York: The Lasker Foundation.  Access Economics; 2003. Exceptional Returns: the value of investing in health R&D in Australia. 2003.  Access Economics; 2008.Exceptional returns: The value of investing in health R&D in Australia II.Canberra.AccessEconomics;2008 ********  Buxton M, Hanney S, Morris S, Sundmacher L, Metre-Ferrandiz J, Garau M, et al. Medical Research – What’s it worth? Estimating the economic benefits from medical research in the UK. Report to the UK evaluation forum 2008. London, UK. ******** 26

27. scientific advice for a healthy Canada Many Options for Methods  Econometric approaches and models  Performance measurement systems  Logic models and frameworks Implementation evaluation models Implementation evaluation models Balanced scorecards Balanced scorecards 27

28. scientific advice for a healthy Canada Many ‘Technical Complexities’ Issues in Determining Health Research Impacts  Attribution issues (effects of factors other than research) and the ‘counterfactual’ (what would have happened without the research being done)? Need collaboration Need collaboration Need research on those topics Need research on those topics  Time lags to impact are very long Need indicators that can track longitudinally Need indicators that can track longitudinally  Double-counting of health research impacts Need to determine contributions wherever possible Need to determine contributions wherever possible  The ‘Halo effect’ (only consider positive impacts of research) Need to consider negative impacts of research Need to consider negative impacts of research 28

29. scientific advice for a healthy Canada Big Challenge  To address the diverse needs of the sponsors Identify a method that can work for all Identify a method that can work for all  To identify a method that can help resolve the technical challenges noted above  To synthesize all available information on this topic  To not ‘reinvent the wheel’ as many other groups are also working in this area in the world right now  Avoid simply advocating ‘more research’ on the topic 29

30. scientific advice for a healthy Canada Health Research Evaluation Frameworks – An International Comparison Philipp-Bastian Brutscher, Steven Wooding, Jonathan Grant RAND EUROPE 30

31. scientific advice for a healthy Canada Reviewed 8 evaluation frameworks FrameworksCountryDescription Leiden University Medical Center (LUMC) NL  Looks at “societal impact” (rather than scientific quality)  Can be seen as part of a broader movement to correct for the “serious imbalance in the research portfolio”. Measure of Research Impact and Achievement (MORIA) AUS  Was developed at the Australian NHMRC as an analytic instrument in the peer review process for grant applications.  It builds on the Record of Research Achievement framework. Program Assessment Rating Tool (PART) US  Was introduced in 2001, as part of the Bush administration’s agenda to improve government management.  PART is used to assess the effectiveness of around 800 federal programmes. Vinnova (Swedish Govern. Agency for Innovation Systems) S  Consists of two main parts: an ongoing evaluation process and an impact analysis. 31

32. scientific advice for a healthy Canada Reviewed 8 evaluation frameworks (Cont’d) FrameworksCountryDescription Payback UK + Canada + Ireland  Has been applied in a number of contexts.  Is an input-process-output-outcome frame-work. It typically comprises two parts: evaluation criteria - (potential impact criteria) and a logic model. UK Department for Innovation, Universities and Skills (DIUS) UK  Aims to “assess the overall health of the science and innovation system…”.  Is the latest stage in a process of developing S&I performance appraisal methods. European Union Framework Programme (EU) EU  Is meant as a system for tracking the results of research programmes  Intended as a way to identify what needs to be improved for these programmes to be more effective. Congr. Directed Medical Research Programs (CDMRP) US  Part of the US Army Medical Research and Material Command (USAMRMC).  Consists of a grants management system, a product database and an Award Survey. 32

33. scientific advice for a healthy Canada Payback Logic Model “Flow” Adapted from: Hanney S, Gonzalez-Block M, Buxton M and Kogan M. The Utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Research Policy Systems 2003, 1:2 33

34. Original Payback Payback1996 CIHR 2005 CIHR 2008 Knowledge Production Advancing Knowledge Research Targeting, Capacity and Absorption Research Targeting and Research Capacity ResearchCapacity Informing Policies and Product Development Informing Policy Informing Decision Making Decision Making Health and health sector benefits Health and Health Sector Benefits Health Benefits Broader Economic Benefits Economic Benefits Five Categories of Health Research Impact

35. scientific advice for a healthy Canada Payback Model Advantages (for Canada)  Built on a logic model and framework  Strong for accountability and management  In use in Canada (CIHR and AHFMR) Similar to some others in use in Canada (eg FRSQ) and builds on elements of others (Lavis, etc) Similar to some others in use in Canada (eg FRSQ) and builds on elements of others (Lavis, etc)  Can allow adequate time scale (longitudinal evaluation)  Can track from individual level upward  Can potentially track from inputs to outputs and onwards to outcomes  Considers social and economic impacts 35

36. CAHS Logic Model Framework built on payback logic model (across bottom) Topic Identification Selection Inputs Process Dissemination Secondary Outputs Adoption Final Outcomes PAYBACK FRAMEWORK Primary Outputs Canadian Health Research Biomedical Clinical Health Services Population and Public Health Cross-Pillar Research Health Industry Economic and Social Prosperity Determinants of Health Public Information, Groups Improvements in Health and Well-being Health care Appropriateness, Access, etc. Prevention and Treatment Health Status, Function, Well-being, Economic Conditions Initiation and Diffusion of Health Research Impacts Global Research Research Capacity Impacts feed back into inputs for future research Other Industries Government Research Agenda Interactions/Feedback 36

37. Canadian Health Research BiomedicalBiomedical ClinicalClinical Health servicesHealth services Population and public healthPopulation and public health Cross-pillar researchCross-pillar research Health Industry -Products/drugs -Services, databases -Practitioners’ behaviour -Clinical/manager’s guidelines -Institutional policies -Social care practices Government (multiple levels) -Resource allocation -Regulation-Policy -Intervention programs -Taxes and subsidies Economic and social prosperity Determinants of health -Personal behaviour -Social/cultural determinants -Environmental determinants -Living and working conditions Public Information, Groups - Advocacy groups - Media coverage - General knowledge - Confidence in data Knowledge Pool Improvements in health and well-being (disease prevalence and burden) Health care -Appropriateness-Acceptability-Accessibility-Competence-Continuity -Effectiveness -Safety Occur through prevention and treatment For disease, illness, injury, or progressive condition -Prevention -Diagnosis/prognosis -Treatment/palliation -Post-treatment Health status and function, well-being, economic conditions Global Research Research Capacity -Increased understanding -Methodological advances -Larger, more comprehensive data sets -Human capital (absorptive capacity) -Student and faculty career paths -Reputation -Research revenues -Cross-fertilization of ideas/research -Education curriculum Research Decision Making -R&D agendas/investment (industry/gov’t/foundations) -Identify issues, gaps -Identify issues, gaps -Evidence problems are being addressed -Tackle harder problems That influence decision making in… Consultation/ Collaborations That affect healthcare, health risk factors, and other health determinants Research activity External Influences: Interests, Traditions Technical limitations, Political dynamics OtherIndustries -Products/services -Built infrastructure -Work environment That produces results That contribute to changing health, well -being and economic and social prosperity Initiation and Diffusion of Health Research Impacts Impacts feed back into inputs for future research Topic IdentificationTopic Identification SelectionSelection InputsInputs ProcessProcess Dissemination Secondary Outputs Adoption Final Outcomes PAYBACK FRAMEWORK Primary Outputs 37

38. Impact Categories and Impacts 38 scientific advice for a healthy Canada

39. Original Payback Payback1996 CIHR 2005 CIHR 2008 CAHS 2009 (Impacts) Knowledge Production Advancing Knowledge Research Targeting, Capacity and Absorption Research Targeting and Research Capacity ResearchCapacityBuildingCapacity Informing Policies and Product Development Informing Policy Informing Decision Making Decision MakingInforming Health and health sector benefits Health and Health Sector Benefits Health Benefits Broader Economic Benefits Economic Benefits Broad Economic and Social Benefits

40. Biomedical Research Health Industry Economic and Social Prosperity Determinants of Health Public Information, Groups Knowledge Pool Improvements in Health and Well-being Healthcare Appropriateness, Access, etc. Prevention and Treatment Health Status, Function, Well- being, Economic Conditions Initiation and Diffusion of Health Research Impacts Global Research Research Capacity Impacts feed back into inputs for future research Government Research Agenda Interactions/Feedback Research Results Other Industries Advancing Knowledge Capacity Building Informing Decision Making Health Benefits Economic Benefits Canadian Health Research Biomedical Clinical Health Services Population and Public Health Cross-pillar Research 40

41. Clinical Research Health Industry Economic and Social Prosperity Determinants of Health Public Information, Groups Knowledge Pool Improvements in Health and Well-being Healthcare Appropriateness, Access, etc. Prevention and Treatment Health Status, Function, Well-being, Economic Conditions Initiation and Diffusion of Health Research Impacts Global Research Research Capacity Impacts feed back into inputs for future research Government Research Agenda Interactions/Feedback Research Results Other Industries Advancing Knowledge Capacity Building Informing Decision Making Health Benefits Economic Benefits Canadian Health Research Biomedical Clinical Health Services Population and Public Health Cross-pillar Research 41

42. Health Services Research Health Industry Economic and Social Prosperity Determinants of Health Public Information, Groups Knowledge Pool Improvements in Health and Well-being Healthcare Appropriateness, Access, etc. Prevention and Treatment Health Status, Function, Well-being, Economic Conditions Initiation and Diffusion of Health Research Impacts Global Research Research Capacity Impacts feed back into inputs for future research Government Research Agenda Interactions/Feedback Research Results Other Industries Advancing Knowledge Capacity Building Informing Decision Making Health Benefits Economic Benefits Canadian Health Research Biomedical Clinical Health Services Population and Public Health Cross-pillar Research 42

43. Population and Public Health Research Health Industry Economic and Social Prosperity Determinants of Health Public Information, Groups Knowledge Pool Improvements in Health and Well-being Healthcare Appropriateness, Access, etc. Prevention and Treatment Health Status, Function, Well-being, Economic Conditions Initiation and Diffusion of Health Research Impacts Global Research Research Capacity Impacts feed back into inputs for future research Government Research Agenda Interactions/Feedback Research Results Other Industries Advancing Knowledge Capacity Building Informing Decision Making Health Benefits Economic Benefits Canadian Health Research Biomedical Clinical Health Services Population and Public Health Cross-pillar Research 43

44. Cross Pillar Research Economic and Social Prosperity Determinants of Health Public Information, Groups Knowledge Pool Improvements in Health and Well-being Healthcare Appropriateness, Access, etc. Prevention and Treatment Health Status, Function, Well-being, Economic Conditions Initiation and Diffusion of Health Research Impacts Global Research Research Capacity Impacts feed back into inputs for future research Government Research Agenda Interactions/Feedback Research Results Other Industries Health Industry Advancing Knowledge Capacity Building Informing Decision Making Health Benefits Economic Benefits Canadian Health Research Biomedical Clinical Health Services Population and Public Health Cross-pillar Research 44

45. scientific advice for a healthy Canada Appropriate Use of the Framework understanding the logic model and impact categories 45

46. scientific advice for a healthy Canada Indicators vs Metrics  Indicators ‘indicate’ impact; they do not attempt to quantify that impact  Metrics are ‘numeric indicators’; they allow putting some numbers on impact A combination of indicators and metrics are recommended A combination of indicators and metrics are recommended 46

47. scientific advice for a healthy Canada Table of CAHS Indicators and Metrics arranged by impact category with advice about how to use each one Shows subcategories, indicators within each subcategory, indicator description, level of recommended application, comments and pillars * To begin a library with references on the web 47 REFER to MENU of INDICATORS*

48. scientific advice for a healthy Canada Sets of indicators and metrics chosen should meet FABRIC criteria:  Focussed on the organization’s objectives that will use them  Appropriate for the stakeholders who are likely to use the information  Balanced to cover all significant areas of work performed by an organization  Robust enough to cope with organizational changes (such as staff changes)  Integrated into management processes  Cost-effective (balancing the benefits of the information against collection costs) 48

49. scientific advice for a healthy Canada Indicators proposed meet qualities of attractiveness and feasibility  Attractiveness: validity, relevance, behavioural impact, transparency, coverage, recency, methodological soundness, replicability, comparability  Feasibility: data availability, cost of data, compliance costs, timeliness, attribution, avoids gamesmanship, interpretation, well- defined 49

50. scientific advice for a healthy Canada ‘Steps’ for Users of the Framework and Indicators 1.Define and prioritize specific evaluation question(s). 2.Use the framework to determine where to look for impacts 3.Based on question(s) choose the impact categories (and subcategories) of interest: advancing knowledge, capacity building, informing decision making, health impacts, and broad economic and social impacts. a.Be as specific as possible about where impacts are expected to occur and at what level (individual, group, institution, provincial, federal, international). b.Choose (or develop) attractive and feasible indicators and metrics from the appropriate categories of interest that will address the evaluation questions at the right level. 4.Choose sets of indicators that are appropriate. Avoid inappropriate uses: attribution, Halo, counterfactual, double-counting 50

51. Three Case Examples To demonstrate potential uses at different levels of complexity 51 scientific advice for a healthy Canada

52. Example #1 National funder : Is our research getting commercialized and can we improve that?  Step 1 Q: Are our researchers commercializing their research effectively right now? Q: Are our researchers commercializing their research effectively right now? Q: What proportion of projects we fund lead to a commercialized product each year? Q: What proportion of projects we fund lead to a commercialized product each year?  Step 2 Use the framework to determine where to look for impacts. Use the framework to determine where to look for impacts. 52

53. Biomedical Research Health Industry Economic and Social Prosperity Determinants of Health Public Information, Groups Knowledge Pool Improvements in Health and Well-being Healthcare Appropriateness, Access, etc. Prevention and Treatment Health Status, Function, Well- being, Economic Conditions Initiation and Diffusion of Health Research Impacts Global Research Research Capacity Impacts feed back into inputs for future research Government Research Agenda Interactions/Feedback Research Results Other Industries Advancing Knowledge Capacity Building Informing Decision Making Health Benefits Economic Benefits Canadian Health Research Biomedical Clinical Health Services Population and Public Health Cross-pillar Research 53

54. scientific advice for a healthy Canada Example #1 National funder: Is our research getting commercialized and can we improve that?  Step 3 Choose Sets of Indicators for each category of interest. Choose Sets of Indicators for each category of interest. Informing decision making (health products industry vs other industries) Informing decision making (health products industry vs other industries) –# patents licensed (per year)/ #projects funded (or per program) –# funded researchers consulted by industry (per year)/# projects Economic Benefits Economic Benefits –Licensing returns ($) –Valuation of spin-out companies ($) –Product sales revenues ($)  Step 4 Review evaluation results to identify potential enablers and barriers to commercialization and improve the system Review evaluation results to identify potential enablers and barriers to commercialization and improve the system 54

55. scientific advice for a healthy Canada Example #2 Provincial Funder: Are we building Research Capacity in our province?  Step 1 Q1: Are we developing and retaining highly qualified research personnel in our province? Q1: Are we developing and retaining highly qualified research personnel in our province? Q2: Are researchers from more than one pillar of research being retained? Q2: Are researchers from more than one pillar of research being retained?  Step 2 Use the framework to identify where to look for impacts and clarify what you mean by “research capacity” Use the framework to identify where to look for impacts and clarify what you mean by “research capacity” e.g. You could use direct indicators (people/$) and/or indirect indicators (impacts on decision makers) e.g. You could use direct indicators (people/$) and/or indirect indicators (impacts on decision makers) 55

56. Biomedical Research Health Industry Economic and Social Prosperity Determinants of Health Public Information, Groups Knowledge Pool Improvements in Health and Well-being Healthcare Appropriateness, Access, etc. Prevention and Treatment Health Status, Function, Well- being, Economic Conditions Initiation and Diffusion of Health Research Impacts Global Research Research Capacity Impacts feed back into inputs for future research Government Research Agenda Interactions/Feedback Research Results Other Industries Advancing Knowledge Capacity Building Informing Decision Making Health Benefits Economic Benefits Canadian Health Research Biomedical Clinical Health Services Population and Public Health Cross-pillar Research 56

57. Health Services Research Health Industry Economic and Social Prosperity Determinants of Health Public Information, Groups Knowledge Pool Improvements in Health and Well-being Healthcare Appropriateness, Access, etc. Prevention and Treatment Health Status, Function, Well-being, Economic Conditions Initiation and Diffusion of Health Research Impacts Global Research Research Capacity Impacts feed back into inputs for future research Government Research Agenda Interactions/Feedback Research Results Other Industries Advancing Knowledge Capacity Building Informing Decision Making Health Benefits Economic Benefits Canadian Health Research Biomedical Clinical Health Services Population and Public Health Cross-pillar Research 57

58. Cross Pillar Research Economic and Social Prosperity Determinants of Health Public Information, Groups Knowledge Pool Improvements in Health and Well-being Healthcare Appropriateness, Access, etc. Prevention and Treatment Health Status, Function, Well-being, Economic Conditions Initiation and Diffusion of Health Research Impacts Global Research Research Capacity Impacts feed back into inputs for future research Government Research Agenda Interactions/Feedback Research Results Other Industries Health Industry Advancing Knowledge Capacity Building Informing Decision Making Health Benefits Economic Benefits Canadian Health Research Biomedical Clinical Health Services Population and Public Health Cross-pillar Research 58

59. scientific advice for a healthy Canada Example #2 Provincial Funder: Are we building Research Capacity in our province?  Step 3 Choose sets of indicators for categories of interest (Capacity Building) Choose sets of indicators for categories of interest (Capacity Building) Infrastructure being built ($/year) Infrastructure being built ($/year) –Infrastructure grant $ attracted ($/year) Funding attracted ($/year) Funding attracted ($/year) –Levels of ‘additional funding’ attracted ($/year) Personnel trained/attracted in the province Personnel trained/attracted in the province –Graduated students per year (MSc or PhD or MD-PhD) –# hospital staff with MSc or PhD or MD/PhD) –# provincial government staff with MSc or PhD 59

60. scientific advice for a healthy Canada Example #2 Provincial Funder: Are we building Research Capacity in our province?  Step 3 (cont’) Choose sets of indicators for categories of interest (Eg. Decision-Making and subcategories in domains of potential impact interest) Choose sets of indicators for categories of interest (Eg. Decision-Making and subcategories in domains of potential impact interest) Health related Health related –Use of research in provincial health care guidelines –Survey of health policy makers Research Research –Citation analysis of successful funding applications Health Products Industry Health Products Industry –Use of research in stage reports by provincial industries General Public General Public –Media citation analysis  Step 4 – Use results to improve capacity development 60

61. scientific advice for a healthy Canada Example #3 Federal Government: Are we achieving national health benefit from funding Canadian health research?  Step 1 Q: How much health benefit are we achieving per $ invested in (any area of) health research in Canada? Q: How much health benefit are we achieving per $ invested in (any area of) health research in Canada? e.g. Cardiovascular research e.g. Cardiovascular research  Step 2: Use the framework to define the potential outcomes and (if attribution to Canadian health research alone is to be determined) – try to trace the stream that leads to Canadian ‘health benefits’ Use the framework to define the potential outcomes and (if attribution to Canadian health research alone is to be determined) – try to trace the stream that leads to Canadian ‘health benefits’ 61

62. Cross Pillar Research Economic and Social Prosperity Determinants of Health Public Information, Groups Knowledge Pool Improvements in Health and Well-being Healthcare Appropriateness, Access, etc. Prevention and Treatment Health Status, Function, Well-being, Economic Conditions Initiation and Diffusion of Health Research Impacts Global Research Research Capacity Impacts feed back into inputs for future research Government Research Agenda Interactions/Feedback Research Results Other Industries Health Industry Advancing Knowledge Capacity Building Informing Decision Making Health Benefits Economic Benefits Canadian Health Research Biomedical Clinical Health Services Population and Public Health Cross-pillar Research 62 $$ all Cdn CV Research $$ The Gross Approach

63. scientific advice for a healthy Canada Example #3 Federal Government: Are we achieving national health benefit from funding Canadian health research?  Step 3: Choose sets of indicators for categories of interest (Economic and Social Benefits) Indicators of Health Status Indicators of Health Status Mortality (potential years of life lost) Mortality (potential years of life lost) Quality adjusted mortality (quality adjusted life years) Quality adjusted mortality (quality adjusted life years) Indicators of “Determinants of Health” Indicators of “Determinants of Health” Prevalence of modifiable risk factors (eg. hypertension) Prevalence of modifiable risk factors (eg. hypertension) Indicators of Health Benefit Indicators of Health Benefit QALY gain per health care dollar spent in CV research and/or CV research $ plus CV treatment $ QALY gain per health care dollar spent in CV research and/or CV research $ plus CV treatment $  Step 4: Improve by defining Canadian attribution 63

64. Cross Pillar Research Economic and Social Prosperity Determinants of Health Public Information, Groups Knowledge Pool Improvements in Health and Well-being Healthcare Appropriateness, Access, etc. Prevention and Treatment Health Status, Function, Well-being, Economic Conditions Initiation and Diffusion of Health Research Impacts Global Research Research Capacity Impacts feed back into inputs for future research Government Research Agenda Interactions/Feedback Research Results Other Industries Health Industry Advancing Knowledge Capacity Building Informing Decision Making Health Benefits Economic Benefits Canadian Health Research Biomedical Clinical Health Services Population and Public Health Cross-pillar Research 64 $$ Cdn CV Research $$ AttributionAttribution The Attribution Approach

65. scientific advice for a healthy Canada What this framework can do  With strategic selection of appropriate sets of indicators the framework can be used to trace impacts within any of the four “pillars of health research” or domains that cut across these pillars.  It can also be used to describe impacts at various levels: individual, institutional, provincial, national, or international and define the “returns on investment” of funders by (eventually) quantifying the values of impacts as a function of dollars put in (ROI). 65

66. scientific advice for a healthy Canada What this framework can do The combination of breadth, depth, and flexibility suggests that the new framework fulfils our sponsor-requested criteria to be useful to a full range of funders/research types, compatible with what is already in place in Canada, transferable to international comparisons, and able to identify the full spectrum of potential impacts. The combination of breadth, depth, and flexibility suggests that the new framework fulfils our sponsor-requested criteria to be useful to a full range of funders/research types, compatible with what is already in place in Canada, transferable to international comparisons, and able to identify the full spectrum of potential impacts. 66

67. scientific advice for a healthy Canada What this framework cannot do  The framework can help guide evaluations but it does NOT provide the questions (or refine them).  The framework cannot resolve all complexities easily or immediately If applied prospectively, it will take time and depending on the scope, it could become costly to apply it If applied prospectively, it will take time and depending on the scope, it could become costly to apply it  It is only as good as the selection of domains, indicator sets and data within indicators to progressively resolve attribution, to achieve a balanced perspective, to avoid the Halo effect to progressively resolve attribution, to achieve a balanced perspective, to avoid the Halo effect The menu of 66 indicators provided is only a start – more are required. The menu of 66 indicators provided is only a start – more are required.  Balanced perspectives may not provide ‘black and white’ answers 67

68. Recommendations 68 scientific advice for a healthy Canada

69. Recommendation #1  The framework and indicators identified by this assessment should be used by all funders of health research in Canada for evaluation of their health research impacts. 69

70. scientific advice for a healthy Canada Recommendation #2  Sets of indicators and metrics chosen from our menu should be used by all funders of health research in Canada for evaluation of their health research impacts. 70

71. scientific advice for a healthy Canada Recommendation #3  Canadian health research funders should begin collaborations immediately to advance the practical (methodological) prerequisites for measuring returns on investment in Canada. To standardize and refine methods, and to routinely collect high quality and appropriate data (eg. common CV; end-of- grant reports, etc). To standardize and refine methods, and to routinely collect high quality and appropriate data (eg. common CV; end-of- grant reports, etc). A library of impact indicators and metrics should be created, beginning with the starting menu developed here. A library of impact indicators and metrics should be created, beginning with the starting menu developed here. A core set of key health research impact questions – based on what is practical and feasible – should be developed. A core set of key health research impact questions – based on what is practical and feasible – should be developed. Strategic and ethically sound selection of indicator sets is required in order to avoid biasing future health research Strategic and ethically sound selection of indicator sets is required in order to avoid biasing future health research Evaluation questions and the choice of indicators and metrics to be used reflect political and social choices; the motives underpinning such choices should be made transparent. Evaluation questions and the choice of indicators and metrics to be used reflect political and social choices; the motives underpinning such choices should be made transparent. 71

72. scientific advice for a healthy Canada Recommendation #4  Canada should immediately initiate a national collaborative effort to begin to measure the impacts of Canadian health research Leaders from national organizations, industry & government should organize a comprehensive evaluation effort that engages research communities, other stakeholders and members of the public Leaders from national organizations, industry & government should organize a comprehensive evaluation effort that engages research communities, other stakeholders and members of the public One option = funders, led by one national organization, form a national council to lead planning and execution with a formal secretariat & commissioned data collectors to begin this work One option = funders, led by one national organization, form a national council to lead planning and execution with a formal secretariat & commissioned data collectors to begin this work 72

73. scientific advice for a healthy Canada Recommendation #5  Canadian health research funders should collaborate internationally to advance the “basic science of health research impacts.” an international funding stream – Collaborative international funding would be beneficial in advancing the elements identified in Recommendation #3. an international funding stream – Collaborative international funding would be beneficial in advancing the elements identified in Recommendation #3. a research plan to identify “contribution indicators” and close “attribution gaps” – The distal indicators of the impacts of health research are very broad, and many factors other than funded health research can and do affect these outcomes. It is fundamental that methodologies be developed to separate the contribution of health research from other causal factors. a research plan to identify “contribution indicators” and close “attribution gaps” – The distal indicators of the impacts of health research are very broad, and many factors other than funded health research can and do affect these outcomes. It is fundamental that methodologies be developed to separate the contribution of health research from other causal factors. a research plan to use the recommended framework for learning (impact improvement) purposes a research plan to use the recommended framework for learning (impact improvement) purposes 73

74. scientific advice for a healthy Canada How recommendations address 6 evaluation issues  ‘Attribution’ (things other than health research impacting health) – solution R5: research on attribution  ‘Counterfactual’ (what would have happened without the research) solution R5: controls are required  Using isolated indicators (distorts impressions) – solution R2: indicator sets must be used  Costs of Evaluation (hard to justify) – solution R3-R5: collaboration nationally and internationally  Double-Counting (two disciplines taking credit for same advance) – solution R5: contributions defined  ‘Halo Effect’ (seeking only positive impacts) – solution R5: consider negative impacts within indicator sets 74

75. scientific advice for a healthy Canada Many Lessons from Process  There are a number of ROI in health research projects in progress internationally right now we had many of the global experts involved in our process & established excellent relationships we had many of the global experts involved in our process & established excellent relationships  There are common international needs & opportunities many potential synergies (*future potential) many potential synergies (*future potential) accurate content and a consistent approach are required to make informed decisions accurate content and a consistent approach are required to make informed decisions  The ‘principles’ of measuring ROI go far beyond health research Involving ‘others’ (e.g. Energy, etc) can add value Involving ‘others’ (e.g. Energy, etc) can add value 75

76. Making an Impact A Preferred Framework and Indicators to Measure Returns on Investment in Health Research Full Report available at http://www.cahs-acss.ca/e/assessments/completedprojects.php http://www.cahs-acss.ca/e/assessments/completedprojects.php Canadian Academy of Health Sciences Académie canadienne des sciences de la santé scientific advice for a healthy Canada