Evaluating the impact of health research: Revisiting the Canadian Institutes of Health Research Impact Assessment Framework Nicola Lauzon, Marc Turcotte & Laura McAuley Impact Assessments Unit Canadian Institutes of Health Research Ottawa, Ontario, Canada

Presentation Overview CIHR impact assessment framework history Study purpose & methods Findings Uses of CIHR framework Focus group feedback from framework users Application of Lessons learned Changes to the framework categories Development of a simplified conceptual model Next steps Questions 2

CIHR Impact Assessment Framework 2006 Category Description Advancing Knowledge discoveries, breakthroughs, and contributions to the scientific literature Building Capacity development and enhancement of research skills in individuals and teams Informing Decision-Making impacts in the areas of science, public, clinical, and managerial decision-making, practice and policy Health & Health Systems Impacts advances in prevention, diagnosis, treatment and palliation, functioning of the health system Economic Impacts commercialization of research discoveries and human capital gains 3

Study Purpose & Methods Purpose To document lessons learned and inform framework improvement Methods Document review of impact reports & protocols How has it been used, results of use, lessons learned Focus groups Feedback and input from users and potential users of framework Six groups; total 35 participants 4

Results: Document Review Uses of CIHR framework Topic-based assessment Focus on specific disease condition Investigate impacts across multiple categories Institute-based assessment Focus on research mandate areas of CIHR Institutes Investigate impacts across multiple categories Category-based assessment Focus exclusively on one impact category 5

Topic-based impact assessments Topic Cardiovascular disease Obesity SARS Main Lessons learned Validation of CIHR impact categories Impacts continue after 20+ years Attribution difficult Topic dictates types of impacts seen 6

Institute-based impact assessments CIHR International Review (2010 – 2011) Impacts of broad mandate areas of all 13 CIHR Institutes between 2000 – 2010 Lessons learned Limited availability of indicators in latter impact categories Need to revisit the categorization and utility of certain indicators Potential to improve bibliometric methodology 7

Results: Focus Groups Main theme of discussions: Attribution versus contribution Direct attribution desired Struggle to find value of latter categories and associated indicators Lessons Learned Need to clarify to stakeholders: Our conceptualization of the research impact process How CIHR fits into the research impact process How the CIHR framework fits into the research impact process 8

Simplified conceptual model of research impact 9 Research EnterpriseResearch Users Society Knowledge Decision-Making Health Knowledge Translation Research Findings EffectUse Funding Impacts on… Capacity Health System Economy

Research EnterpriseResearch Users Society Knowledge Decision-Making Health Knowledge Translation Research Findings EffectUse Funding Impacts on… Capacity Health System Economy Not-so-simplified version 10

Research Enterprise Research Users Society Knowledge Decision-Making Health Knowledge Translation Research Findings EffectUse Funding Impacts on… Capacity Health System Economy Not-so-simplified version 11 Government priorities Public priorities Funder priorities Government budget Partnerships & leveraged funds Peer review Researcher priorities Research Institution priorities Infrastructure Research team composition Publication bias Peer review Researcher characteristics Time lag to publication Continued relevance User characteristics Government priorities User acceptance Accessibility System capacity Economic outlook Personal characteristics Demand Political climate System capacity

Simplified conceptual model of research impact 12 Research EnterpriseResearch Users Society Knowledge Decision-Making Health Knowledge Translation Research Findings EffectUse Funding Impacts on… Capacity Health System Economy

Simplified conceptual model of research impact 13 Research EnterpriseResearch Users Society Knowledge Decision-Making Health Knowledge Translation Research Findings EffectUse Funding Impacts on… Capacity Health System Economy

Simplified conceptual model of research impact 14 Research EnterpriseResearch Users Society Knowledge Decision-Making Health Knowledge Translation Research Findings EffectUse Funding Impacts on… Capacity Health System Economy

Simplified conceptual model of research impact 15 Impacts on… Knowledge Capacity Decision- Making Health Health System Economy Research EnterpriseResearch Users Society Knowledge Translation Research Findings EffectUse Funding

16 CIHR Impact Assessment Framework Category Lower order indicators Higher order indicators Knowledge # Publications from grants in specific research area # Canadian publications in specific research area Capacity # CIHR-supported PhD trainee awards # Canadian PhD grads in health Decision- Making Citation in research Citation in health policy Health Outcome measures of condition X for clinical trial patients Prevalence of condition X in Canada Health Systems Health efficiencies from specific grants Wait times Economy License or royalty agreement counts # jobs created from spin-off companies

Next steps & Conclusions Continue testing model, framework categories and indicators Feedback & validation from stakeholders Alignment of model and framework with literature 17

Questions??? 18