Presentation on theme: "Stakeholder –defined research"— Presentation transcript:
1 Stakeholder –defined research Designing Studies that reflect the perspectives of Patients, clinicians and PayersEPG Grand Rounds, University of MarylandFebruary 6, 2014Patricia Deverka, MD, MS
2 Center for Medical Technology Policy The Center for Medical Technology Policy (CMTP) is an independent ,non-profit 501(c) (3) organization that seeks to advance health care innovation and effectiveness by improving the quality, relevance, and efficiency of health care research. CMTP works on methods, infrastructure and policy to support the conduct of comparative effectiveness research that generates information to assist patients, clinicians, and payers in making informed clinical and health policy decisions.
3 Rationale for engaging stakeholders Definitions and approach OverviewRationale for engaging stakeholdersDefinitions and approachCase examplesGenerative discussion
4 How to reconcile: ~18,000 RCTs are published each year* The Evidence ParadoxHow to reconcile:~18,000 RCTs are published each year*A growing number of non-experimental studiesMany systematic reviews, health technology assessments, clinical guidelines conclude that the available evidence is limited or studies are poor qualityUp to 60% of clinical recommendations made by ACC or AHA based on expert opinion and/or low quality studiesSystematic review of off-label uses of 19 FDA-approved oncology drugs (428 pages, several thousand trials) “Because of the paucity of high quality evidence, the data available – though voluminous – may have little meaning or value for informing clinical practice”* Chalkidou, Tunis, Whicher, et al. The role for pragmatic, randomized controlled trials (pRCTs) in comparative effectiveness research.Clin icalTrials .Published online before print July 2, 2012, doi: /
5 Reasons evidence produced by current clinical research enterprise not translated into practice Differences between settings where research is conducted and where medicine is practicedPatient populationInterventions, including usual careProviders, referral patterns access to careFailure to (be able to) report how treatment effects vary in individual patients and subgroupsUnderrepresentation of children, women, elderly, ethnic & racial minorities, patients with comorbiditiesResearch priorities, study questions, endpoints, etc. defined by researchers and funders, not end users
6 The CER HypothesisGaps in evidence will be reduced with increased guidance from payers, patients and clinicians in study designA functional definition of CER would be research designed in light of meaningful engagement of these decision makers
7 Definitions of cer and pcor Definition of CERDefinition of PCORThe generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care. The purpose of CER is to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care at both the individual and population levels.Source: Institute of MedicinePatient-Centered Outcomes Research (PCOR) helps people and their caregivers communicate and make informed health care decisions, allowing their voices to be heard in assessing the value of health care options.PCOR has the following characteristics:• Actively engages patients and key stakeholders throughout the research process.• Compares important clinical management options.• Evaluates the outcomes that are most important to patients.• Addresses implementation of the research finings in clinical care environments.Source: PCORI
8 What is unique about CER? Many CER studies will require an understanding of the trade-offs between internal validity and increased generalizability, relevance, feasibility and timelinessThe right balance is not solely a scientific issue, it’s also a social judgment about an acceptable level of uncertainty, involving multiple stakeholdersProcess to achieve this with stakeholder input is evolving
9 Application of Evidence Dissemination Avenues THEORY OF STAKEHOLDER ENGAGEMENTEvidence suggests that engaging stakeholders in research increases:Relevance and ContentKnowledge ExchangeApplication of EvidenceDissemination AvenuesRelevance and Content
10 Barriers to involving stakeholders in cer Confusing terminology, lack of standard definitionsTiming; restrictions on availability of stakeholdersTraining needs for all stakeholders to maximize participationConcerns that process will add time and costs to project plansLack of shared conceptualization of what it means to “successfully” or “effectively” involve stakeholders in researchLimited data regarding impact; systematic evaluation rareSources: Guise, O'Haire, McPheeters, et al. A practice-based tool for engaging stakeholders in future research: a synthesis of current practices. J Clin Epidemiol Jun;66(6): doi: /j.jclinepi Epub 2013 Mar 13. and CMTP experience
12 Addressing the Barriers Literature reviewBiomedicalSocial sciencePractical experience based on projects involving stakeholdersDrafted definitions and conceptual modelReview and revision by an expert panelPatient and Consumer Advisory CouncilNICE Patient and Public Involvement Program and Citizen Council in the UKApplied it to a complex multi-stakeholder project
13 Public Participation Activities Diverse Roots ofPublic Participation ActivitiesCBPR*Reducing health disparitiesSocial change & actionHealth careHealth technology assessmentFDAHealth research (UK)Public policyEnvironmental planningNuclear powerBiotechnology*Community-based participatory research
14 Typology of stakeholder engagement Stakeholder engagement “light”Stakeholder engagementSource: Nass, Levine, and Yancy. Methods for Involving Patients in Topic Generation for Patient-Centered Comparative Effectiveness Research –An International Perspective
15 StakeholdersIndividuals, organizations, or communities that have a direct interest in the process and outcomes of a project, organization, or policy.Deverka, Lavallee, Desai, et al. Stakeholder participation in comparative effectiveness research: defining a framework foreffective engagement. J Compar Effect Res 2012; 2:
17 Stakeholder Engagement A process of actively soliciting the knowledge, experience, judgment and values of individuals selected to represent a broad range of direct interests in a particular issue, for the dual purposes of:1) Creating a shared understanding;2) Making relevant, transparent, and effective decisions.
18 Conceptual Model for Stakeholder Engagement in CER Analytic-Deliberative ModelTypes of evidenceInputsValuesResearchProfessional ExperiencePatient and consumer knowledge and experienceMethods of combining evidenceMethodsQuantitativeQuestionnairesDelphi methodMulti-Criteria MappingValue of Information modelingQualitativeFacilitatedworkshops/meetingsStakeholder decision analysisDecisionsOutputsTopic generationResearch prioritiesStudy designsEvidentiary thresholds for clinical and health policy decisionmakingImplementation strategiesOutcomesProcessMeta-criteria, Trust, Respect, Accountability, Legitimacy, Fairness, CompetenceChange in Knowledge/attitudesChange in CER project decisions (e.g. choice of interventions, study design, funding priorities)CERMore useful evidence for clinical and health policy decision makingMore efficient use of healthcare resourcesImproved health outcomes.
19 Deciding which stakeholder groups to involve in a project What topic(s) does the research address?What health care decision is the research meant to inform?Who are the decision makers responsible for these decisions?Who are the individuals and groups that are affected by these decisions?Concannon TW, Meissner P, Grunbaum JA, et al. A new taxonomy for stakeholder engagement in patient-centered outcomesresearch. JGIM 2012;27(8):
20 When to involve patients and other stakeholders in research? Topic identification and refinementPriority-settingWriting proposals (including deciding research methods)Reviewing research conductInterpretation of findingsDissemination of informationImplementationEvaluationCurtis, Slaughter-Mason, Thielke, et al. PCORI Expert Interviews Project: Final Report. Portland, OR: Center for Evidence-based Policy Oregon Health & Sciences University.
21 STAKEHOLDER INVOLVEMENT IN GOVERNANCE Infra-structure*Assess needsRecommend data elementsIdentify applicationsMonitorResearchGenerate & prioritize research questionsDesign studiesData collection, processing &analysisResults interpretationEducationTranslationDisseminationEvaluationContinuous learning healthcare systemDecision-makingData accessData securityData privacyConflicts of interestConsentCommunicationCoordination with other committeesAdapted from: Rein A, Holve E, Hamilton Lopez M, and Winkler J. A framework for patient and consumer engagement in evidence generation,” EDM Forum,Academy-Health, September 2012.*Refers to the creation/modification of the Patient Engagement Platform needed to support both clinical management and the conduct of PCOR
22 PATient-centered Involvement in Evaluating the effectiveNess of TreatmentS PATIENTS
23 Aims of PATIENTSFoster sustainable partnerships with local, regional, and national communities of diverse patients and healthcare systemsConduct and expand PCOR in partnership with patients and healthcare delivery systemsAdvance dissemination and implementation strategies for PCOR findings
24 Vision The PATIENTS Program vision is that its projects will: Further the process of UM institutional transformation for “MPowering the State” in the area of healthEliminate health disparities within Baltimore, throughout Maryland, and across the nationAlign with the spirit of the NIH roadmap for transformative and interdisciplinary research
25 University of Maryland has broad participation UMB professional schoolsPharmacyMedicineNursingSocial WorkDentistryLawUM College Park
27 Innovation in the PATIENTS Program Conducting PCOR with continuous patient and stakeholder engagementTranslating research into practiceContinuous development through formative and impact evaluationBidirectional learningSustainability
28 The Ten-Step Process for Conducting CER Public AnnouncementsPatient ForumsDelphi ProcessTopic SolicitationPrioritizationFraming the QuestionBased on: Mullins CD, Abdulhalim AM, Lavallee DC.Continuous Patient Engagement in Comparative Effectiveness Research.JAMA 2012; 307(15):
29 The Ten-Step Process for Conducting CER Selection of Comparators and OutcomesCreation of Conceptual FrameworkAnalysis PlanData CollectionIn-person MeetingsFocus Group Inter-viewsTeleconferencesElectronic Social MediaTelephone CallsBased on: Mullins CD, Abdulhalim AM, Lavallee DC.Continuous Patient Engagement in Comparative Effectiveness Research.JAMA 2012; 307(15):
30 The Ten-Step Process for Conducting CER Teach-Back MethodCritique Documents(e.g. Patient Guides)MediaReviewing & Interpreting ResultsTranslationDisseminationBased on: Mullins CD, Abdulhalim AM, Lavallee DC.Continuous Patient Engagement in Comparative Effectiveness Research.JAMA 2012; 307(15):
31 Education and Training: Engaging Partners Bidirectional LearningStakeholder EngagementCultural CompetenceUsing Online PlatformsCommunity PartnersResearch MethodsManuscript WritingGrant WritingUM Faculty and StaffExamples:PatientsLikeMe: Social MediaBSBHS/Riverside Heath System: Implementation
32 Examples:Study Designs Reflecting Views of Decision Makers
33 Genomic Testing in Cancer (CANCERGEN) Evidence Guidance Documents Two ExamplesGenomic Testing in Cancer (CANCERGEN)Evidence Guidance DocumentsMolecular Dx in Cancer
34 The Promise: Transform Cancer Care Molecular diagnostic (MDx) tests have the potential to transform oncology practice by helping physicians classify and manage various cancersDiagnose and stage cancersHelp guide therapy selection and dosingAssess treatment responseAid in detection of residual or recurrent disease
36 Thariani R, Wong W, Carlson JJ, et al Thariani R, Wong W, Carlson JJ, et al. Prioritization in Comparative Effectiveness Research: The CANCERGEN Experience in Cancer Genomics. Medical Care 2012; 50(5):Figure 1: CANCERGEN project milestones including landscape analysis, stakeholder assessment and final selection.
37 4. BRAF mutation testing in Colorectal Cancer Ranking of TestsNo Order1. ERCC1 Expression Testing for Platinum-Based Adjuvant Therapy in NSCLC2. EGFR Mutation Testing for Erlotinib Maintenance Therapy in Advanced NSCLC3. EGFR gene copy number (FISH) testing and first-line cetuximab therapy in stage IV or recurrent NSCLC4. BRAF mutation testing in Colorectal Cancer5. Genetic Expression Profile (GEP) in Multiple Myeloma (MM) to Identify Patients with Poor Prognosis6. Breast CA Tumor Markers for Detection of Recurrence After Primary Breast Cancer TherapyFinal Order1. ERCC12. BCTM3. EGFR mutationCenter for Comparative Effectiveness Research in Cancer Genomics (CANCERGEN)
38 Multi-modal approaches are necessary Lessons LearnedFull participation of all stakeholders on highly technical topics is possible with adequate preparationMulti-modal approaches are necessaryEngagement method should be matched to particular study questionStakeholders were open to novel methods (e.g., VOI), but more work needs to be done to ensure full benefits of the approach are fully realizedPossible to engage most stakeholders for a multi-year projectFederal officials are the most difficult
40 A Stakeholder-Driven Approach to Improving the Evidence Base Molecular Diagnostics (MDx) In Oncology
41 Decision Makers’ Key Questions for Cancer MDx Tests Does the MDx test provide correct information? (analytic validity)How well does the test result correlate with clinical outcome? (clinical validity)Does use of the MDx test lead to improved patient outcomes as compared with the alternative? (clinical utility)Does use of the MDx test lead to greater value as compared with the alternative? (cost-effectiveness)
42 Clinical utility largely unknown for most MDx tests ProblemAnalytic validity and clinical validity now available for an increasing number of MDx testsClinical utility largely unknown for most MDx testsUncertain clinical utility has consequences for patients and health care systemDecreases quality through inconsistent or unnecessary use of testsWastes health care resources
43 Structured data about MDx test use Stakeholder-driven process What’s neededStructured data about MDx test useStakeholder-driven processClear evidentiary standards for clinical utilityWillingness to consider range of methods, outcome measures that are relevant to real-world clinical decisions
44 THE Response: Effectiveness Guidance Documents Provide specific recommendations on the design of studies intended to inform decisions by patients, clinicians and payersDeveloped for specific clinical conditions and categories of technologiesBased on a structured, transparent, multi-stakeholder process led by CMTPAim to balance internal validity, relevance, timeliness and feasibilityprovide decision-makers with a reasonable level of confidence that the intervention improves net health outcomesAnalogous and complementary to FDA guidanceTargeted to researchers working in industry or academic settings
45 Technical Working Group Academic researchers 2Industry 3Payers 2Research funders 1Policy makers 1Patient advocate 1
46 Recommendations: 10 covering clinical validity and clinical utility MDx test development follows phases similar to the phases of drug developmentRecommendations have been organized around these phasesBiomarker discovery (Phase 0) and the assessment of population impacts (Phase 5) go beyond the scope of this EGD
47 By test developers and researchers in designing studies How EGDs Might be UsedBy test developers and researchers in designing studiesBy payers in evaluating evidence submitted for coverage and reimbursementBy guidelines developers in judging quality of evidence and strength of recommendationsBy research funding organizations in evaluating grant proposalsBy patient advocacy and other groups generating guidance for patients
48 Evaluation is critical for measuring impact and process improvement SummaryStakeholder engagement is essential for fulfilling the objectives of CERResearch funding requests and support need to account for resources required to meaningfully implement engagement activitiesThe terminology and methods are being developed and tested worked for CERMethods need to be tailored to the particular phase of research, but stakeholders should be involved throughout the processCareful attention to communication at multiple levels is critical to ensuring true collaboration and a respectful, accountable processThere is a growing body of examples of rigorous methods of SE being applied to CEREvaluation is critical for measuring impact and process improvement
50 CMTP: Principles for Involving Patients in Comparative Effectiveness Research Each CER-related project includes patient representatives.** Examples of such projects include identifying research topics, setting priorities, developing questions to be studied, designing study protocols and establishing methodological standards.Project leaders recruit a diverse group of patients for whom the project topic is relevant.Project leaders, patients and other stakeholders make their mutual expectations for patient involvement known to each other.The project team, other stakeholders and patients disclose potential conflicts of interest.Project budget includes appropriate remuneration for patients and support for their participation, including training, stipends, travel and lodging, and other resources critical for their full involvement in the project.**Includes patients, family caregivers and consumers representatives of all types
51 CMTP: Principles for Involving Patients in Comparative Effectiveness Research……..cont’d The project team and other stakeholders recognize and respect the different skills, knowledge and experience of patients. Patients recognize and respect those of the other participants.The project team communicates regularly with patients throughout the life of the project to ensure the quality and sustainability of the involvement process.The project team obtains periodic assessments from patients and other stakeholders to evaluate the effectiveness of the engagement process and inform the design of future research involving patients.Project reports and publications describe in the methods sections how patients were involved in research.The project team and other stakeholders work with patients to present study findings in a way that can be easily understood by patients.