1. Approaches for Supporting Evidence- and Values-Informed Policymaking
Knowledge, Sharing, Doing: 1st National KT Conference in Rehabilitation
4 May 2016
Montréal, Québec
Michael G. Wilson, PhD
Assistant Director, McMaster Health Forum
Assistant Professor, McMaster University

2. Evidence- and Values-Informed Policy (1)
Systematically and transparently using the best available data and research evidence, as well as citizens’ values and preferences, in each of:
Prioritizing problems and understanding their causes (agenda setting)
Deciding which option to pursue (policy development)
Ensuring the chosen option makes an optimal impact at acceptable cost (policy implementation)
… alongside the institutional constraints, interest-group pressure, values and other types of information (like jurisdictional reviews, consultations, expert review groups, and opinion polls) that influence the policy process 2

3. Evidence- and Values-Informed Policy (2)
Source: Hoffman SJ, Rottingen J-A, Bennett S, Lavis JN, Edge JS, Frenk J. Building Health Systems Research as a Field of Scientific Endeavour: Wading through Definitional Confusion, Conceptual Challenges and Opportunities for the Future, Manuscript under view. 3

4. Options Available to Support the Use of Research Evidence
EXTRA
6/3/2018
Options Available to Support the Use of Research Evidence
Two factors that consistently increased the prospects for research use in management / policy (based on a systematic review of 124 studies)
Citation: Lavis JN, Catallo C, editors (2013). Bridging the worlds of research and policy in European health systems. Copenhagen, WHO Regional Office for Europe.
Interactions between researchers and decision-makers
Timing/ timeliness of research evidence 4
(c) CHSRF/FCRSS

5. Examples of Approaches
Timeliness
Timeliness & interactions
Interactions
One-stop-shops
Rapid-response units
Citizen briefs & panels
Evidence briefs & stakeholder dialogues
Training/capacity building 5

6. One-Stop-Shops (1)
One-stop shops (and evidence services) are a promising ‘self-serve’ KT innovation for several reasons
Supports timely access (everything in one place)
Facilitates assessments of relevance
Organized by priority topics (e.g., health systems)
Provide decision-relevant information (e.g., quality, countries in which included studies are conducted)
Enhances communication
Presentation of evidence in several formats and in ways that are user friendly (e.g. links to free full text or user-friendly summaries) 6

7. ‘My health’ ‘Our health’ ‘Our system’ One-Stop-Shops (2)
McMaster PLUS (
Pre-appraised studies and reviews that address the full range of questions about clinical programs and services and about drugs
‘My health’
Health Evidence (
Pre-appraised reviews that address questions about the effectiveness of public health programs
‘Our health’
Health Systems Evidence (
pre-appraised reviews and economic evaluations that address how to strengthen health systems and get cost-effective programs, services and drugs to those who need them
‘Our system’ 7

8. One-Stop-Shops (3) 8

9. Evaluation & Impact One-stop-shops (Health Systems Evidence)
11,596 registered users (4403 signed up to receive a monthly customized evidence service)
Researchers = 3874; Policymakers = 2460; Healthcare professionals = 2630; Managers = 1261; Plus more than 3000 students and 2464 ‘other’
Endorsed by WHO’s Health Systems Research Synthesis Group as the one-stop shop for research syntheses about health systems
Incorporated into other resources (e.g., EVIPNet Virtual Health Library, McMaster Optimal Aging Portal)
Increasingly used to inform high-profile scientific articles / studies
Cited as the key source in the New England Journal of Medicine (Mills) and in Health Policy (Rockers et al.) 9

10. Rapid-Response Units (1)
EXTRA
6/3/2018
Rapid-Response Units (1)
Often need to address pressing health system issues in days or weeks
May need support with finding and synthesizing research evidence given competing demands (but not enough time to prepare an evidence brief and convene a dialogue)
Rapid-response units fills a gap between
‘Self-serve’ approaches (e.g., one-stop shops) and
‘Full-serve’ approaches (e.g., stakeholder dialogues informed by evidence briefs) 10
(c) CHSRF/FCRSS

11. Synthesize relevant evidence
Rapid-Response Units (2)
There are several existing programs in Canada (e.g., CADTH, INESSS, OHTN), but not specifically focused on health-system questions
11 stakeholder dialogue participants agreed that there is a clear need for such a program
Key recommendations:
build a model
decide what can done in what timelines
define success and measure it
Refine question
Conduct searches
Review search results
Synthesize relevant evidence
Requestor 11

12. Evaluation & Impact Rapid response
Evaluation approach is in development
Examples of impacts
A synthesis about suicide-prevention interventions directly informed the 2014 Toronto Public Health Suicide Prevention Strategy
Three syntheses for the College of Physicians and Surgeons of Ontario directly informed College policies
unsafe medical practices
preventing sexual abuse
educational approaches to improve clinical performance 12

13. Citizen Panels (1) Key challenges Complex health problems
different understandings of the problem
Uncertainties
e.g., about the most effective policy options to address these problems and their implementation considerations (e.g., equity, costs, unintended effects, acceptability and feasibility)
Lack of agreement
among all stakeholders about how to move forward 13

14. Citizen Panels (2) The role of citizens’ values and preferences
Citizens can help us to develop a shared understanding about a problem (challenge 1)
Citizens have experiential knowledge (challenge 2)
valid and legitimate evidence that can help to find innovative and local solutions to complex problems
Citizens can facilitate or trigger action (challenge 3)
offering guidance on how to move forward
identifying what options are socially, politically, and ethically sound
advocating for policy options 14

15. Citizen Panels & Stakeholder Dialogue Process Overview
1. Consulting with key stakeholders
2. Preparing a citizen and evidence brief
3. Convening a citizen panel and preparing a summary
4. Convening a stakeholder dialogue and preparing a summary
5. Supporting action 15

16. 1. Consulting with Stakeholders
Steering committee
Develop criteria to select diverse panel participants
Iteratively refine understanding of the problem, possible options to address it and implementation considerations
Identify key informants
Key informant interviews
15-20 interviews with representatives of citizen/patient groups, policymakers, stakeholders and researchers who are involved in or affected by the issue 16

17. 2. Preparing a Citizen & Evidence Brief
Presents (in plain language) relevant research evidence about a problem, options for addressing it, and key implementation considerations
Based on syntheses and local evidence (identified using systematic and transparent approach)
No recommendations
Subjected to merit review
Citizen brief poses questions for citizens to consider 17

18. 3a. Convening a Citizen Panel
Brings together a diverse group of 10 to 16 citizens for a one-day, off-the-record, dialogue that provides them with the opportunity to:
bring their own views and experiences to bear on an issue;
learn from the evidence and from others’ views and experiences; and
share their newly informed views about the issue and how to address it. 18

19. 3b. Preparing a Summary Thematic analysis of deliberations Describes:
views about and experiences related to the issue
values and preferences for addressing it
Identify areas of shared understandings, as well as divergent opinions 19

20. 4a. Convening a Stakeholder Dialogue
Brings together policymakers, stakeholders and researchers for ‘off-the-record’ deliberations about:
the problem
options to address it
implementation considerations
next steps
Participants chosen because of their ability to:
bring unique views and experiences
champion actions to address the challenge creatively 20

21. 4b. Preparing a Summary Thematic analysis of deliberations
Identifies areas of common ground
Divergent opinions
Respects Chatham House Rules 21

22. 5. Supporting Action Two examples of ways to support action
Take an integrated approach to supporting evidence- and values-informed policymaking by convening a citizen panel followed by a stakeholder dialogue on the same topic
informed by an evidence brief that includes findings from the citizen panel
‘off-the-record’ deliberations with those who are in a position to champion change
Personalized briefings to those who can take action 22

23. Evaluation & Impact (1) Citizen brief (n=14 briefs; n=211 respondents)
Mean overall assessment = 6.1 (SD = 1.2)
Range of ratings of 14 features of citizen briefs =
Citizen panels (n=20 panels n=215 respondents)
Mean overall assessment = 6.7 (SD = 1.2)
Range of ratings of 11 specific design features =
Self-rated knowledge of the topic increased from pre- to post-panel (mean of 5.1 to 5.6)
Example of impact
Directly informed the Ontario Medical Association’s end-of-life care strategy and the Canadian Medical Association’s national dialogue about end-of-life care 23

24. Evaluation & Impact (2) Evidence briefs (n=41, n= 610 respondents)
Overall rating of brief = 6.2, all but five features rated ≥ 6.0, and only three features had much variation (SD>1.1)
Dialogues (n=41, n=613 respondents)
Overall rating of dialogue = 6.2, all but one feature rated ≥ 6.1, and no features had much variation (SD>1.1)
Examples of impact
Directly informed an Ontario provincial cabinet submission about creating community-based specialty clinics
Spurred formation of committee reporting to Ontario deputy minister to optimize clinical practice based on data, evidence and guidelines 24

25. Evaluation & Impact (3) Dialogues (continued) Ratings
Strong behavioural intention to act and positive attitudes, but lower ratings of (and greater variability in) subjective norms and behavioural control
Behavioural intentions
I expect to use
6.2 [0.8]
I want to use
I intend to use
6.1 [0.8]
Positive attitudes
6.3 [0.9]
Subjective norms
5.7 [1.3]
Perceived behavioural control
5.5 [1.5] 25

26. Training/Capacity Building
Policymakers, stakeholders and researchers require support to develop skills in finding and using research evidence to inform the stages of policy development (clarifying problems, framing options, identifying implementation considerations)
E.g., - Health Systems Learning
Finding and Using Research Evidence to Inform Decision-making in Health Systems and Organizations
Three course objectives:
To develop knowledge about tools and resources available to help health system decision-makers in order to support their use of research evidence
To examine the attitudes that are supportive of using research evidence in health system decision-making
To enhance skills in acquiring, assessing, adapting and applying research evidence 26

27. Evaluation & Impact Training/capacity building
1,055 trainees (639 in-person, 190 in-person+online and 226 online)
531 policymakers, 42 managers, 47 students, 15 professionals, 16 researchers, 13 ‘other’ and 180 in multiple roles completed evaluation (RR=80%)
Strongly positive feedback from evaluations (measured on seven-point Likert scale)
Overall rating = 6.0 (range = 4 - 7)
Highest rated feature: Material relevant to my professional development = 6.6 (range = 3-7)
Lowest ratings: The workshop enhanced local applicability assessment skills = 5.8 (range = 2-7)
16 of 18 design features had average ratings of 6 or more 27

28. Facebook McMaster Heath Forum
Stay current with updates about Forum-led initiatives, events and training opportunities, via and
Facebook McMaster Heath Forum 28