1. Sarah Boesveld (PhD cand.) Francois-Pierre Gauvin, PhD
The Campaign for McMaster University
The Campaign for McMaster University
Strengthening Public and Patient Engagement in Health Technology Assessment: A Framework for Involving Patients and the Public in Health Quality Ontario's Evidence-Review Process
Julia Abelson, PhD
Frank Wagner, MA
Deirdre DeJean, PhD
Stephen Petersen, MSc
Sarah Boesveld (PhD cand.)
Francois-Pierre Gauvin, PhD
Mita Giacomini, PhD
John Lavis, PhD
CADTH Conference 2015
Saskatoon, SK
April 14th, 2015

2. Acknowledgements Funders Collaborators
The Campaign for McMaster University
The Campaign for McMaster University
Acknowledgements
Funders
Canadian Institutes of Health Research
Evidence-Informed Healthcare Renewal Policy Analysis grant
Government of Ontario
Ministry of Health and Long-Term Care Health System Research Fund grant “Harnessing Evidence and Values for Health System Excellence”
Collaborators
Health Quality Ontario
Public Engagement Subcommittee of the Ontario Health Technology Advisory Committee (OHTAC)
McMaster Health Forum

3. Background & Objectives
The Campaign for McMaster University
The Campaign for McMaster University
Background & Objectives
Increasing attention to involving patients and members of the public in HTA agencies worldwide
Efforts to produce conceptually robust, evidence-informed frameworks to guide public and patient engagement (PPE) in HTA have been rare
In particular, lack of attention to matching PPE goals and approaches to specific HTA stages

4. Background & Objectives (2)
The Campaign for McMaster University
The Campaign for McMaster University
Background & Objectives (2)
Context:
Ontario’s evidence-review process for health technologies (Health Quality Ontario (HQO), Ontario Health Technology Advisory Committee (OHTAC)) – non-drug HTA
Traditional stakeholder consultation & passive communication and consultation with patients and members of the public (interest in doing more)
Collaboration between researchers and HQO-OHTAC Public Engagement Subcommittee (Jan 2013 – July 2014)
Parallel activities (review of Decision Determinants framework; strategic emphasis on PPE by HQO leadership)
Description and results of framework development process

5. Key Elements of the Framework
Articulate the underlying principles, values and goals for public and patient engagement in HTA
Establish a common language to support public and patient engagement efforts
Describe a flexible menu of approaches that can be used depending on the goal and phase of the evidence-review process
Identify and develop measures of engagement and its outcomes that can inform adjustments over time

6. Final Report (released April 8, 2015)

7. Subcommittee membership
OHTAC members
External members
HQO/EDS
Frank Wagner (Chair)
Julia Abelson, Deirdre DeJean & Sarah Boesveld (McMaster/CIHR research grant)
Shamara Baidoobonso (EDS)
Pat Campbell
Renata Axler (University of Toronto)
Gaylene Pron (EDS)
Tony Easty
Sally Bean (Sunnybrook Health Sciences)
Stephen Petersen (EDS)
Murray Krahn
Tara Gomes (Ontario Drug Policy Research Network)
Heather Thomson (Strategic Partnerships)
Dorothy Pringle
Sine Mackinnon (Patient & Public Engagement)
Shirlee Sharkey
Charles Wright

8. Methods Committee deliberations informed by:
The Campaign for McMaster University
The Campaign for McMaster University
Methods
Committee deliberations informed by:
website review of 53 HTA organizations across 34 countries
stated goals, approaches and experiences with PPE in HTA
review of published literature (PPE in HTA)
reviews, empirical & conceptual studies ( )
expert consultations (international & local)
stakeholder dialogue (May 2014)

9. Conceptual Foundation for the Framework

10. Goals for Public & Patient Engagement
Democratic accountability & legitimacy
To achieve more informed, transparent, accountable and legitimate decisions about health technologies
Scientific
To promote a more robust and comprehensive approach to HTA that incorporates social and ethical values, as well as patients’ problems, lived experiences, outcomes and preferences
Instrumental
To enhance the quality of outputs across all stages of the HTA process
Developmental
To increase public understanding of health technologies and HTA, and strengthening the public’s and patients’ competence and capacity to contribute to health technology policy issues
Adapted from Abelson et al Bringing the public into health technology assessment and coverage policy decisions. Health Policy; 82(1):37-50.

11. Who to engage with? Patients, families, caregivers/providers
Individuals with experiential knowledge about living with an illness or condition who can provide valuable perspectives about the intended or unintended consequences of current or future health technologies
Broader publics
Individuals who can contribute broad social values regarding the efficiency or fairness of a technology but who may not have specific experience with a particular technology, disease or condition
Stakeholder groups
Groups with organized interests in the technology, program or service, including its funding and delivery arrangements (e.g., advocacy groups, industry, provider organizations)
It’s important to be clear about language. We are defining the relevant publics as follows…
Patients, families, caregivers…
Broader publics…
Stakeholder groups…
Informed by stakeholder engagement categories listed in 2007 report
- Patients and Families/ Caregivers
- Advocacy Groups/Patient and Caregiver Organizations
- General Public/Ontario Taxpayers
Facey et al Patients’ perspectives in health technology assessment: A route to robust evidence and fair deliberation. IJTAHC; 26(3):

12. How to engage? Three categories of PE activity: Inform Disseminate
COMMUNICATION
Disseminate
Involve
CONSULTATION
Gather Information
Interact
PARTICIPATION
Collaborate
It’s also important to be clear about what we mean by engagement and the purpose for which we are engaging. We are focusing on 3 categories of PE activity in our work, each of which have different goals
Communication, this is about communicating information to relevant publics which may be a first step as part of a more comprehensive public involvement process. [contributes to transparency & legitimacy goals]
Consultation refers to as set of activities where your organizations are seeking to gather information from relevant publics through a variety of means which could include surveys, public meetings or social media. There may be a public communication phase that is carried out in conjunction with this but the key objective is to consult with relevant publics to gather their perspectives. [contributes to accountability,legitimacy and improved decision making]
Participation involves a two-way exchange of information between the sponsor and the relevant publics. This type of engagement typically involves more intensive deliberative methods around ‘meatier’ issues that are heavily value-laden, where the evidence is contested and for which there may be low levels of trust among the public [contributes to accountability, legitimacy, improved decision making and increased trust]
Adapted from: Rowe & Frewer (2005). A Typology of public engagement mechanisms. Science, Technology & Human Values; 30(2):

13. HQO-OHTAC evidence-review process (at the beginning of the committee’s work)
1 SCOPING
2 EVIDENCE-BASED ANALYSIS
3 DRAFT OHTAC RECOMMENDATIONS
4 PROFESSIONAL
AND PUBLIC
CONSULTATION
5 ASSESSMENT OF COMMENTS
6 POST REVIEW
AND
RECOMMENDATION
7 FIELD
EVALUATION
HQO drafts brief overview of intervention
OHTAC determines which interventions proceed to a full review
HQO reviews evidence in consultation with:
- Clinical experts
and expert panels
- Scientific partners
- Industry
- Government
OHTAC drafts recommendations based on evidence-based review
Draft review and recommendations are posted on the HQO/OHTAC websites for public and professional comment
HQO reviews public and professional comment feedback
OHTAC modifies recommendations as required
HQO evidence-based review and OHTAC recommendations published on website and announced in e-bulletin
OHTAC may request a field evaluation to assess effectiveness and cost-effectiveness of an intervention in the Ontario context
This slide shows the evidence review process when the committee was reconvened in The committee was originally struck in 2006 and made numerous recommendations that were reflected in this current process through a series of consultation and communication stages. Most notabley, once OHTAC has reviewed and approved an evidence review and associated OHTAC recommendations, a 21-day public and professional consultation period is initiated (stage 4). This is best described as a passive communication/consultation process with material posted for those who are looking for it.
The reports and recommendations, along with a plain language summary, are posted on HQO’s website, under the “Evidence” tab, and under the “Get Involved” subsection. Along with the posting, the communications team at HQO sends out a stakeholder advisory, notifying any potentially interested parties that the materials have been posted. This usually includes HQO’s strategic partners (e.g., the Ontario Medical Association, OMA) and related advocacy groups (e.g., the Ontario Association of Medical Laboratories, OAML, may receive notice of public commenting for an OHTAC recommendation on COPD).
If/when comments arrive in the inbox, they are forwarded to the appropriate and responsible individuals for review. This usually includes the report authors and the VP of Evidence Development and Standards. If necessary, and on rare occasions, the public/professional commentator is invited to provide additional feedback.
Once the 21-day period has concluded, the results (if any) of the public/professional consultation are presented to OHTAC for feedback. At this time, OHTAC may request changes to either the report or the recommendation. After this presentation, and with any requested changes made, the report is posted as final to the HQO website.
JA NOTES FOR DISCUSSION: Can we get a list of the stakeholder organizations who are involved at either early or later stages in the process?
Stakeholder
consultation
Passive
consultation
Passive
communication

14. Preliminary Framework for Strengthening Public & Patient Engagement
1 SCOPING
2 EVIDENCE-BASED ANALYSIS
3 DRAFT OHTAC RECOMMENDATIONS
4 PROFESSIONAL
AND PUBLIC
CONSULTATION
5 ASSESSMENT OF COMMENTS
6 POST REVIEW
AND
RECOMMENDATION
7 FIELD
EVALUATION
HQO drafts brief overview of intervention
OHTAC determines which interventions proceed to a full review
HQO reviews evidence in consultation with:
- Clinical experts
and expert panels
- Scientific partners
- Industry
- Government
OHTAC drafts recommendations based on evidence-based review
Draft review and recommendations are posted on the HQO/OHTAC websites for public and professional comment
HQO reviews public and professional comment feedback
OHTAC modifies recommendations as required
HQO evidence-based review and OHTAC recommendations published on website and announced in e-bulletin
OHTAC may request a field evaluation to assess effectiveness and cost-effectiveness of an intervention in the Ontario context
Rationale for PE: Whom to engage with: How to engage:
- improved knowledge & awareness patients, families, providers - communication
- higher quality decisions broader publics - consultation
- increased transparency, legitimacy stakeholder groups - participation
This slide shows how we set out to move systematically through each stage of the OHTAC process and for each stage we will:
establish the rationale for PE (or if one exists),
consider the relevant groups to engage with depending on the rationale
determine how best to engage these groups informed by the published evidence & international practice
Sources: published research, international practice, expert consultations

15. Evidence Review (Key messages)
Literature review and evidence summary undertaken as part of CIHR grant “Strengthening governance and accountability in Ontario HTA” (acknowledge SB & DD)

16. Key messages from website scan and literature synthesis
General
one-third to two-thirds of organizations reporting some PPE activity (increasing)
very small number have comprehensive approach to PPE (all HTA stages)
Goals for PPE
improving the quality of assessments and achieving accountable, legitimate processes are regularly cited goals
no explicit links drawn between the goals for PPE and specific HTA stage
PPE (Who, When & How)
increasing efforts to engage the public and patients in early stages (e.g., topic selection, prioritization and scoping stages);
common PPE approaches: communication (web & some social media), consultation (through solicitation of input, feedback on draft documents, focus groups) and direct participation (through membership on committees)
So given paucity of rigorous studies in this area, what are some of the key messages we’ve gleaned from mostly descriptive studies and our scan of HTA websites…

17. Key messages from review of stage-specific PPE in HTA (2)
PPE (Who, When & How)
emphasis on incorporating patient values/perspectives through in-depth consultation, qualitative research & patient representation on advisory committees
broader publics tend to be involved at board level or when broader societal considerations are explicitly sought
Evaluation
minimal evaluation of the effectiveness or impacts of specific PE mechanisms
primarily descriptive and case study driven; small number of rich ethnographic studies
Feasibility/Implementation considerations
numerous cultural and organizational challenges
major enabler is organizational commitment and resources (e.g., time and expertise)
common challenges: how to seek ‘representative’ perspectives; how to balance broad public interest with narrower interests of organization reps
When it comes to who to engage…
Cultural challenges
-tensions between traditional focus on clinical and economic evidence and pressures to incorporate patient/social values input
-perceptions that engaging patients and publics will politicize what should be an evidence-informed process
-perceptions about the robustness of social/patient values as a source of evidence (what are we measuring? where’s the evidence that this works?)
-perceptions that patients and citizens are unable to contribute meaningfully
Organizational challenges
-time, resource and expertise required to support high-quality PPE
-HTA agency constraints re lack of PPE expertise, organizational commitment & capacity
Patient/public recruitment challenges
-obtaining ‘representative’input (what does this mean?)
-concerns public/patient involvement will allow narrow interests to trump fairness considerations

18. McMaster Health Forum Evidence Brief
- prepared for McMaster Health Forum stakeholder dialogue held on May 8
- included sources identified on previous slide + targeted search of Health Systems Evidence
a continuously updated database containing more than 3,800 systematic reviews and more than 1,900 economic evaluations of delivery, financial and governance arrangements within health systems
focus on: i) identifying promising public- and patient-engagement models that could be adapted to Ontario’s process; ii) building capacity within HTA organizations to engage the public and patients; and
iii) building capacity among the public and patients to engage in the HTA process.

19. FRAMEWORK & RECOMMENDATIONS

20. Key Elements of the Framework
Articulate the underlying principles, values and goals for public and patient engagement in HTA
Establish a common language to support public and patient engagement efforts
Describe a flexible menu of approaches that can be used depending on the goal and phase of the evidence-review process
Identify and develop measures of engagement and its outcomes that can inform adjustments over time
Refresh of what was presented earlier with focus on bullet 3

21. Building a Common Language for Public & Patient Engagement

22. Menu of approaches & tools
Refer them to relevant pages of the report for orientation…

23. Refresher of OHTAC process

24. Newly configured process to include new stage for PPE
Major discussion point in the committee was the lack of transparency in the topic selection process and the lack of societal input into the decisions about which technologies are reviewed

25. Key slide: PPE goals tailored to each stage…

26. Summary of Approaches
In this slide we move from the goals for each stage to a detailed menu of the different constituencies that might be the focus of engagement activity within each stage and the relevant approaches that might be used – each stage maps the different groups that might be involved, the broad engagement mechanisms that would be used and the specific tools within each category
*Note that asterisk indicates link to recommendations that follow
*Also note that some of these activities would be triggered by the tool that societal and ethical values working group is developing (e.g., identification of priority populations, primary qualitative research/synthesis, citizen panels for high priority topics)

27. Topic Selection: Who and How
0 TOPIC SELECTION & prioritization
Groups
Mechanism
Tools
General/interested public
Communication
Website description of topic selection process
General public
Patients
Patient organizations
Consultation
Polling/surveys
Horizon scanning (media analysis, focus groups, surveys)
Stakeholder meetings/Delphi process (e.g., 2-3 years)
Participation
Advisory committee representation
Focus on each stage of the process and the range of mechanisms and tools that could be used
Note that red asterisk indicates link to recommendations that will follow…

28. 1 SCOPING (may include expert panel input)
Scoping: Who and How
1 SCOPING (may include expert panel input)
Groups
Mechanism
Tools
Patient organizations
Consultation
Invited submissions (targeted and web)
Patients
Social media analysis
Interviews/focus groups
Participation
Committee representation (e.g., expert panel)

29. Evidence-Based Analysis: Who and How
2 EVIDENCE-BASED ANALYSIS (may include expert panel input & qualitative meta synthesis)
Groups
Mechanism
Tools
Patient organizations
Consultation
Invited submissions
Patients
Surveys
Social media analysis
Qualitative research/synthesis
Participation
Committee representation (e.g., expert panel)

30. Draft OHTAC Recommendations: Who and How
3 DRAFT OHTAC RECOMMENDATIONS (process begins in Evidence Review stage)
Groups
Mechanism
Tools
Patient organizations/patients
Consultation
Expert panel consultation with priority populations
Broader publics/patients
Participation
Explicit discussion of societal/patient values (OHTAC meeting)

31. Professional and Public Consultation: Who and How
Groups
Mechanism
Tools
Unsolicited public
Patients/patient organizations
Communication
HQO website posting
Social media profile (selected reports)
Lay review of summary
Patient organizations
Consultation
Targeted mailings
Solicited reviews of draft report/face-to-face meetings with priority groups
Patients
Broader publics
Participation
Committee representation (e.g., expert panel)
Citizen panels (for high priority topics triggered by DD framework)

32. Assessment of Comments: Who and How
Groups
Mechanism
Tools
Authors of public comment submissions
Communication
Web posting of comments and actions taken
Consultation
Face-to-face meetings to discuss and address concerns (selected)

33. Post Review and Recommendation: Who and How
Groups
Mechanism
Tools
Broader publics
Patients
Communication
Multimedia dissemination of OHTAC report
Lay review of plain language summary
Patient organizations
Targeted dissemination to high priority groups

34. Report Recommendations
1. Increase the transparency of the evidence review process to facilitate a fuller understanding among interested patients, broader publics, and stakeholder groups of how topics are selected and referred to OHTAC
2. Increase the relevance and responsiveness of HQO-OHTAC’s work by undertaking a range of horizon-scanning and consultation activities to prospectively identify potential review topics and emerging issues of concern to patients and priority populations

35. Report Recommendations
3. Embed public and patient engagement in the scoping of all review topics with the following exceptions:
the topic under review concerns technologies with no direct patient interface
The focus of the review is exclusively on the technical aspects of the technology
4. Draw on a range of sources for incorporating societal and patient values into the evidence-based analysis stage. e.g.,
patient members on expert panels,
consultations with individual patients and/or patient groups
synthesis of primary qualitative research studies

36. Report Recommendations
5. Identify dedicated agenda time at monthly meetings for the explicit consideration of societal and patient values relevant to all evidence reviews
6. Enhance public and patient consultation process at the post-appraisal stage to develop increased awareness of its work and to encourage broader input on its draft recommendations

37. Measurement & Evaluation
Weak evidence base for PPE in HTA (and related fields)
Framework provides opportunity to embed evaluation into the initiation of new PPE activities informed by clearly articulated goals, populations and methods
Formative evaluation metrics (examples)
are we meeting our goals for transparency and increased awareness of HQO and OHTAC’s work?
are patient and public members contributing meaningfully and at relevant stages of the HTA process?
(how) are the outputs from PPE activities informing OHTAC & HQO decisions?

38. Implementation Early stages
Report adopted by HQO Board
Presentations to HQO leadership and staff
Alignment with HQO’s strategic focus on patient, caregiver and public engagement
Need to prioritize and support areas of focus
Stage by stage approach or early vs. later?
Building awareness of Ontario’s evidence review process among interested constituencies
Ensure that training and supports for staff, OHTAC members, patient and citizens are in place from the outset
Ensure that evaluation metrics are developed early on

39. Future Work
How and when should PPE be triggered within the HQO-OHTAC process?
Which PPE approaches for which types of technologies, conditions or populations?

40. The Campaign for McMaster University
Link to Final Report