Development of PRO Content Steven Blum Director, Patient Reported Outcomes Duke Industry Statistics Symposium September 7, 2017

Disclaimer Steven Blum is an employee and shareholder of GlaxoSmithKline (GSK). The views and opinions expressed in the following slides are mine and should not be attributed to GSK or any other organization

Patient-Reported Outcomes A Patient-Reported Outcome (PRO) measures the concepts that are relevant and important to a patient’s condition and its treatment directly from the patient without interpretation by a clinician. PROs are a type of Clinical Outcome Assessment (COA). Other types of COAs include Clinician-Reported Outcomes (ClinROs), Observer-Reported Outcomes (ObsROs) and Performance Outcome (PerfOs) Measures The principals of the PRO Guidance apply to both PROs and other COAs

Selecting Type of COA Measure Observable* concepts (e.g., signs, events, behaviors) Unobservable concepts (e.g., feelings, sensations) No clinical judgment needed Clinical judgment needed Self-report feasible and appropriate? No Yes ObsRO PRO ClinRO PRO * Observable concepts: must be able to be detected by a sense or senses -- vision, hearing, smell, or touch Source: Critical Path Institute - PRO Consortium

Patient-Reported Outcomes In clinical trials, PROs provide evidence of treatment benefit with respect symptoms, impact and function known only to the patient. PROs can also be used to capture the patient’s perspective on observable signs and functional impact In addition to being used in Clinical Trials, PROs can be used in real-world studies (e.g. surveys, registries, observational studies) as well as used in clinical practice and in quality measures The content of a PRO is developed through qualitative research to elicit key concepts of disease experience and impact from patients representing the target population. The US Food and Drug Administration and the European Medicines Agency emphasize the need to establish content validity as the initial step in developing a PRO.

Evaluation of PROs by Regulatory Authorities The evaluation of a PRO instrument to support claims in medical product labeling includes the following considerations: The population enrolled in the clinical trial The clinical trial objectives and design The PRO instrument’s conceptual framework The PRO instrument’s measurement properties Reliability Content Validity Construct Validity Ability to detect change Context of Use Source: FDA PRO Guidance (2009)

Content Validity Content validity is the extent to which the instrument measures the concept of interest. Content validity is supported by evidence from qualitative studies that the items and domains of an instrument are appropriate and comprehensive relative to its intended measurement concept, population, and use. Content validity is specific to the population, condition, and treatment to be studied (context of use). For PRO instruments, items, domains, and general scores reflect what is important to patients and comprehensive with respect to patient concerns relevant to the concept being assessed. Documentation of patient input in item generation as well as evaluation of patient understanding through cognitive interviewing can contribute to evidence of content validity. Evidence of other types of validity (e.g., construct validity) or reliability (e.g., consistent scores) will not overcome problems with content validity because we evaluate instrument adequacy to measure the concept represented by the labeling claim. It is important to establish content validity before other measurement properties are evaluated. Guidance for Industry - Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims (2009): https://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM193282.pdf

“Traditional” Development Process Preliminary Research Concept Elicitation Item Generation Cognitive Debriefing Psychometric Testing

FDA Roadmap to Patient-Focus Outcome Measurement https://www.fda.gov/Drugs/DevelopmentApprovalProcess/DrugDevelopmentToolsQualificationProgram/ucm284077.htm

Preliminary Research Prior to “development” of a new PRO, researchers should conduct preliminary research, including: Literature review of prior qualitative studies Identify relevant signs, symptoms and impacts associated with disease Informs on the development of protocol and interview guides Literature review of existing PRO/COA instruments Identify potential measures, concepts measured, potential items Conduct “gap analysis” to assess development history and adequacy of existing measures Seek input from experts (clinicians, patients, caregivers, methodologists) Conduct preliminary qualitative focus groups or interviews to develop disease understanding Identify, develop or revise conceptual disease model All of these preliminary steps help us to understand the disease or condition and informs on the design on qualitative studies including development of protocol, interview guide and related study documents.

Qualitative Interviews Concept Elicitation Cognitive Debriefing Used to identify relevant concepts (meaningful health aspects) and to understand the specific language used to describe the concept Can be conducted as focus groups or interviews Typically uses a semi-structured interview guide to allow for both spontaneous and probed responses Recorded for transcription and analysis Interviews are analyzed using Qualitative Data Analysis Software Development of coding framework, code book and summaries Assess Saturation of Concept Basis for Item Generation Used to assess relevance of items/ concepts selected and comprehension and understanding of the item stem, response scale, and recall period Used to revise/refine measure Typically uses “think aloud” method Problematic items can be revised or dropped All changes (revisions, item reduction, additions) are documented/detailed using an Item Tracking Matrix Original Item, Examples of Issues from Interviews, Revised Item, Rationale Cognitive Interviews can also be used when assessing equivalence between different modes of administration (P2E migration) or in translation and cultural adaptation

Analyzing Qualitative Data Caution: Sample Sizes are Often Small! Concept Elicitation: Often ~25-30 subjects (stop when reach saturation) Cognitive Interviews : Often ~15 subjects (3 rounds of 5 interviews) May need larger sizes if lots of heterogeneity Rare diseases and difficult to reach populations can be even smaller Recruitment focuses on diversity (e.g. age, gender, education level, disease status) Inclusion/exclusion criteria tends to be less restrictive than clinical trials May want to recruit a broader population than what will be recruited in clinical trials May incorporate some recruitment goals, but may not use strict recruitment quotas Demographics and clinical characteristics may not match prior clinical trials or real- world studies The analysis of qualitative data is not a quantitative process; there are no significance levels, effect sizes, or other quantitative metric. Qualitative analyses use respondent words and phrases as data, analyzing and classifying these data by concept and subconcept

Analyzing Qualitative Data Qualitative Interviews are deep and rich with data Unstructured data Code and analyze data using Qualitative Data Analysis software Try not to “over-quantify” data Look at frequency of mentions – by transcript and by mentions Be mindful of probing/follow-up Look at spontaneous vs. probed reporting Can also include rating exercises (severity, bother) Assess attributes like frequency, severity, duration, variability to help with framing of questions Analyzing coding process – inter-rater agreement (>90%) Assess Saturation (overall, sub-groups) Identify preferred patient language/terms for item generation Forms the basis for item generation process or justification for use or modification of existing measures

Coding Process The primary goal of transcript coding is to organize and catalog a patient’s descriptions of their experiences within the context of use. Coding is an iterative process with opportunities for data to be re-examined and reanalyzed until no new codes or code groupings are identified and all passages from the transcripts have been assigned one or more codes. Patrick et al. Value in Health (14):967-977, figure adapted by Mona Martin

Documenting Saturation Saturation: the point when no new relevant or important information emerges and collecting additional data will not add to the understanding of how patients perceive the concept of interest and the items in a questionnaire. FDA PRO Guidance (2009); Patrick et al. Value in Health (14):967-977

Mixed Methods Development Concept Elicitation (Qual) Item Generation Cognitive Debriefing (Qual) Pilot Survey 1 (Quant) Revision Pilot Survey 2 (Quant) Formal Testing (Quant) Combines both Qual-itative and Quant-itative data in the development of the measure Often done in an iterative process Example: Conduct traditional development Conduct Initial Pilot Quant Study to test measure (can use CTT, IRT, RMT) Identify potential issues (scaling issues, fit, redundancy, measurement gaps) Make additional revisions to measure Evaluate changes in additional cognitive interviews Conduct a second Wave Pilot Quant study to test revisions After revisions are complete, move to formal psychometric testing

Example: Pain Assessment for Lower Back Symptoms Original 0-10 NRS Response Scale Revised 4-level categorical VRS Item Threshold Map Person-Item Map Blum et al., ISOQOL, 2016

Conceptual Framework The conceptual framework explicitly defines the concepts measured by the instrument in a diagram that presents a description of the relationships between items, domain (subconcepts), and concepts measured and the scores produced by a PRO instrument. The conceptual framework of a PRO instrument will evolve and be confirmed over the course of instrument development as a sponsor gathers empiric evidence to support item grouping and scores (e.g. psychometric evaluation, factor analysis, finalization of scoring algorithm) FDA PRO Guidance (2009); Patrick et al. Value in Health (14):967-977

Summary PROs can be used to capture the patient’s perspective on the signs, symptoms and impacts associated with a disease or condition. PRO measures can be used to assess treatment effect in clinical trials In addition to being used in clinical trials, PROs can be used in real-world studies (e.g. surveys, registries, observational studies) as well in clinical practice and in quality measures Qualitative interviews ensure the selection of relevant concepts and patient understanding. Need to ensure the appropriate context of use It is important to establish content validity before other measurement properties are evaluated. Qualitative (unstructured) data are analyzed using different methods from structured data (including PRO data collected in studies) Well established methods for analyzing qualitative data Mixed Methods approaches can combine both qualitative and quantitative data as part of instrument development

Questions? steven.i.blum@gsk.com