1. Building a Complex Health Research Logic Model: Making Pathways to Impacts Clear American Evaluation Association Conference November 5, 2011 Gretchen B. Jordan Sandia National Laboratories gbjorda@sandia.gov Parts of work presented here was completed for the U.S. DOE by Sandia National Laboratories, Albuquerque, New Mexico. Opinions expressed are solely those of the author.

2. This talk will demonstrate A complex, one page logic model for the return on investment for health sciences research With several features to frame assessments that are similar enough to feed evaluation synthesis Built on agreed upon theory and definitions to specify aspects of both the healthcare system and the environment that affects health and individual behaviors; Categorized institutions and actors that are the pathways through which advances get to impact; and Individual studies look at on pathways to impact, noting other parts of the logic as context. 2 G. Jordan AEA 2011

3. Background and Charge This logic model was part of a Major Assessment Panel led by the Canadian Academy of Health Sciences (CAHS) to develop a “Preferred Framework and Indicators to Measure Returns on Investment in Health Research.” Twenty-three different organizations sponsored Charge to the Expert Panel: Develop a framework or model at the outset of the assessment and demonstrate that the model fulfills the required criteria: – be useful to a range of funders/research types, – be compatible with what is already in place in Canada, – be transferable to international comparisons, and – be able to identify the full spectrum of potential impacts. 3 G. Jordan AEA 2011

4. Model Design For Canada’s four “pillars of health research” and cross cuts Impacts at various levels — individual, institutional, provincial, national or international Track health-research impacts in five main categories: – advancing knowledge – building capacity, – informing decision-making, – health impacts, and – broad socio-economic impacts. Subcategories identify and partition evaluation methods A menu of indicators in each sub category Intermediate outcomes and (eventually) quantify the value(s) of impacts to end-users as a function of dollars invested 4 G. Jordan AEA 2011

5. Model relies on evidence AND is potentially hypothesis generating The model built on academic and practical studies of – the determinants of health: World Health Organization paper (2006); Evans and Stoddart (1990) and Dahlgren and Whitehead (1991) – diffusion of innovation theories: Everett Rogers (1995), Reed and Jordan (2007) for the five groups influenced by R&D Synthesis across matched studies can test hypotheses and build theory – ease of synthesis enhanced if studies use similar terminology, good research design, and make clear the full context – a single study cannot be large enough in scope; differences in context are informative; points to features of an intervention that matter most – synthesis also shows where there are gaps in knowledge 5 G. Jordan AEA 2011

6. Topic Identification Selection Inputs Process Dissemination Secondary Outputs Adoption Final Outcomes PAYBACK FRAMEWORK Primary Outputs Economic and Social Prosperity Improvements in Health and Well- being Initiation and Diffusion of Health Research Impacts Impacts feed back into inputs for future research Health Industry Public Information, Groups Other Industries Government Research Agenda Determinants of Health Health care Appropriateness, Access, etc. Prevention and Treatment Canadian Health Research Biomedical Clinical Health Services Population and Public Health Cross-Pillar Research Knowledge Pool Health Status, Function, Well-being, Economic Conditions Global Research Research Capacity Interactions/Feedback Research Results 6G. Jordan AEA 2011 CAHS Logic for Return on Investment to Health Research

7. Health Industry -Products/drugs -Services, databases -Practitioners’ behaviour -Clinical/manager’s guidelines -Institutional policies -Social care practices Government (multiple levels) -Resource allocation -Regulation -Policy -Intervention programs -Taxes and subsidies Determinants of health -Personal behaviour -Social/cultural determinants -Environmental determinants -Living and working conditions Public Information, Groups - Advocacy groups - Media coverage - General knowledge - Confidence in data Health care -Appropriateness -Acceptability -Accessibility -Competence -Continuity -Effectiveness -Safety Occur through prevention and treatment For disease, illness, injury, or progressive condition -Prevention -Diagnosis/prognosis -Treatment/palliation -Post-treatment Research Capacity -Increased understanding -Methodological advances -Larger, more comprehensive data sets -Human capital (absorptive capacity) -Student and faculty career paths -Reputation -Research revenues -Cross-fertilization of ideas/research -Education curriculum Research Decision Making -R&D agendas/investment (industry/gov’t/foundations) -Identify issues, gaps -Evidence problems are being addressed -Tackle harder problems Other Industries -Products/services -Built infrastructure -Work environment Economic and social prosperity Improvements in health and well- being (disease prevalence and burden) External Influences: Interests, Traditions Technical limitations, Political dynamics Canadian Health Research Biomedical Clinical Health services Population and public health Cross-pillar research Knowledge Pool Health status and function, well-being, economic conditions Global Research Consultation/ Collaborations Research Results That influence decision making in… That affect healthcare, health risk factors, and other health determinants Research activity That produces results That contribute to changing health, well - being and economic and social prosperity CAHS LOGIC in more detail G. Jordan AEA 20117

8. Applying the model to different research pillars By cross-mapping impact categories for each pillar, we can create a specialized framework to show where the majority of impacts are likely to fall within that pillar. Intent is to show where most impacts are likely to be found and thus where indicators or metrics for those impacts should be concentrated. However, the areas of impact for any one research pillar are not exclusive of impacts in other domains. 8 G. Jordan AEA 2011

9. Biomedical Research Canadian Health Research Biomedical Clinical Health Services Population and Public Health Cross-pillar Research Health Industry Economic and Social Prosperity Determinants of Health Public Information, Groups Knowledge Pool Improvements in Health and Well- being Healthcare Appropriateness, Access, etc. Prevention and Treatment Health Status, Function, Well-being, Economic Conditions Initiation and Diffusion of Health Research Impacts Global Research Research Capacity Impacts feed back into inputs for future research Government Research Agenda Interactions/Feedback Research Results Other Industries Advancing Knowledge Capacity Building Informing Decision Making Health Benefits Economic Benefits A 9G. Jordan AEA 2011

10. Population and Public Health Research Canadian Health Research Biomedical Clinical Health Services Population and Public Health Cross-pillar Research Health Industry Economic and Social Prosperity Determinants of Health Public Information, Groups Knowledge Pool Improvements in Health and Well- being Healthcare Appropriateness, Access, etc. Prevention and Treatment Health Status, Function, Well-being, Economic Conditions Initiation and Diffusion of Health Research Impacts Global Research Research Capacity Impacts feed back into inputs for future research Government Research Agenda Interactions/Feedback Research Results Other Industries Advancing Knowledge Capacity Building Informing Decision Making Health Benefits Economic Benefits D

11. Summary and Conclusion The logic model was a definite asset for categorizing indicators and ensuring that important indicators were not overlooked. The framework has been used for impact assessments by at least three organizations, – Alberta Innovates Health Solutions – Catalan Agency of Health Information, Assessment and Quality in Spain – Canadian Cancer Society Research Institute (CCS RI) Some organizations continue to use the Payback Model It remains to be seen if it is possible to copy this logic and menu of indicators in other areas of research – The groups to influence are similar – But the common language and indicators for the intermediate outcomes (“magic in the middle”) are less well defined outside of health area. 11 G. Jordan AEA 2011

12. Thank you for your attention. For a full copy of the assessment report see www.afmc.ca/pdf/ROI_FullReport.pdf www.afmc.ca/pdf/ROI_FullReport.pdf My contact information after December 15, 2011 gretchen.jordan@comcast.net 12 G. Jordan AEA 2011