1. Development of PRO Content
Steven Blum
Director, Patient Reported Outcomes
Duke Industry Statistics Symposium
September 7, 2017

2. Disclaimer
Steven Blum is an employee and shareholder of GlaxoSmithKline (GSK).
The views and opinions expressed in the following slides are mine and should not be attributed to GSK or any other organization

3. Patient-Reported Outcomes
A Patient-Reported Outcome (PRO) measures the concepts that are relevant and important to a patient’s condition and its treatment directly from the patient without interpretation by a clinician.
PROs are a type of Clinical Outcome Assessment (COA).
Other types of COAs include Clinician-Reported Outcomes (ClinROs), Observer-Reported Outcomes (ObsROs) and Performance Outcome (PerfOs) Measures
The principals of the PRO Guidance apply to both PROs and other COAs

4. Selecting Type of COA Measure
Observable* concepts (e.g., signs, events, behaviors)
Unobservable concepts (e.g., feelings, sensations)
No clinical judgment needed
Clinical judgment needed
Self-report feasible and appropriate? No
Yes
ObsRO
PRO
ClinRO
PRO
* Observable concepts: must be able to be detected by a sense or senses -- vision, hearing, smell, or touch
Source: Critical Path Institute - PRO Consortium

5. Patient-Reported Outcomes
In clinical trials, PROs provide evidence of treatment benefit with respect symptoms, impact and function known only to the patient.
PROs can also be used to capture the patient’s perspective on observable signs and functional impact
In addition to being used in Clinical Trials, PROs can be used in real-world studies (e.g. surveys, registries, observational studies) as well as used in clinical practice and in quality measures
The content of a PRO is developed through qualitative research to elicit key concepts of disease experience and impact from patients representing the target population.
The US Food and Drug Administration and the European Medicines Agency emphasize the need to establish content validity as the initial step in developing a PRO.

6. Evaluation of PROs by Regulatory Authorities
The evaluation of a PRO instrument to support claims in medical product labeling includes the following considerations:
The population enrolled in the clinical trial
The clinical trial objectives and design
The PRO instrument’s conceptual framework
The PRO instrument’s measurement properties
Reliability
Content Validity
Construct Validity
Ability to detect change
Context of Use
Source: FDA PRO Guidance (2009)

7. Content Validity
Content validity is the extent to which the instrument measures the concept of interest.
Content validity is supported by evidence from qualitative studies that the items and domains of an instrument are appropriate and comprehensive relative to its intended measurement concept, population, and use.
Content validity is specific to the population, condition, and treatment to be studied (context of use).
For PRO instruments, items, domains, and general scores reflect what is important to patients and comprehensive with respect to patient concerns relevant to the concept being assessed.
Documentation of patient input in item generation as well as evaluation of patient understanding through cognitive interviewing can contribute to evidence of content validity.
Evidence of other types of validity (e.g., construct validity) or reliability (e.g., consistent scores) will not overcome problems with content validity because we evaluate instrument adequacy to measure the concept represented by the labeling claim.
It is important to establish content validity before other measurement properties are evaluated.
Guidance for Industry - Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims (2009):

8. “Traditional” Development Process
Preliminary Research
Concept Elicitation
Item Generation
Cognitive Debriefing
Psychometric Testing

9. FDA Roadmap to Patient-Focus Outcome Measurement

10. Preliminary Research
Prior to “development” of a new PRO, researchers should conduct preliminary research, including:
Literature review of prior qualitative studies
Identify relevant signs, symptoms and impacts associated with disease
Informs on the development of protocol and interview guides
Literature review of existing PRO/COA instruments
Identify potential measures, concepts measured, potential items
Conduct “gap analysis” to assess development history and adequacy of existing measures
Seek input from experts (clinicians, patients, caregivers, methodologists)
Conduct preliminary qualitative focus groups or interviews to develop disease understanding
Identify, develop or revise conceptual disease model
All of these preliminary steps help us to understand the disease or condition and informs on the design on qualitative studies including development of protocol, interview guide and related study documents.

11. Qualitative Interviews
Concept Elicitation
Cognitive Debriefing
Used to identify relevant concepts (meaningful health aspects) and to understand the specific language used to describe the concept
Can be conducted as focus groups or interviews
Typically uses a semi-structured interview guide to allow for both spontaneous and probed responses
Recorded for transcription and analysis
Interviews are analyzed using Qualitative Data Analysis Software
Development of coding framework, code book and summaries
Assess Saturation of Concept
Basis for Item Generation
Used to assess relevance of items/ concepts selected and comprehension and understanding of the item stem, response scale, and recall period
Used to revise/refine measure
Typically uses “think aloud” method
Problematic items can be revised or dropped
All changes (revisions, item reduction, additions) are documented/detailed using an Item Tracking Matrix
Original Item, Examples of Issues from Interviews, Revised Item, Rationale
Cognitive Interviews can also be used when assessing equivalence between different modes of administration (P2E migration) or in translation and cultural adaptation

12. Analyzing Qualitative Data
Caution: Sample Sizes are Often Small!
Concept Elicitation: Often ~25-30 subjects (stop when reach saturation)
Cognitive Interviews : Often ~15 subjects (3 rounds of 5 interviews)
May need larger sizes if lots of heterogeneity
Rare diseases and difficult to reach populations can be even smaller
Recruitment focuses on diversity (e.g. age, gender, education level, disease status)
Inclusion/exclusion criteria tends to be less restrictive than clinical trials
May want to recruit a broader population than what will be recruited in clinical trials
May incorporate some recruitment goals, but may not use strict recruitment quotas
Demographics and clinical characteristics may not match prior clinical trials or real- world studies
The analysis of qualitative data is not a quantitative process; there are no significance levels, effect sizes, or other quantitative metric. Qualitative analyses use respondent words and phrases as data, analyzing and classifying these data by concept and subconcept

13. Analyzing Qualitative Data
Qualitative Interviews are deep and rich with data
Unstructured data
Code and analyze data using Qualitative Data Analysis software
Try not to “over-quantify” data
Look at frequency of mentions – by transcript and by mentions
Be mindful of probing/follow-up
Look at spontaneous vs. probed reporting
Can also include rating exercises (severity, bother)
Assess attributes like frequency, severity, duration, variability to help with framing of questions
Analyzing coding process – inter-rater agreement (>90%)
Assess Saturation (overall, sub-groups)
Identify preferred patient language/terms for item generation
Forms the basis for item generation process or justification for use or modification of existing measures

14. Coding Process
The primary goal of transcript coding is to organize and catalog a patient’s descriptions of their experiences within the context of use.
Coding is an iterative process with opportunities for data to be re-examined and reanalyzed until no new codes or code groupings are identified and all passages from the transcripts have been assigned one or more codes.
Patrick et al. Value in Health (14): , figure adapted by Mona Martin

15. Documenting Saturation
Saturation: the point when no new relevant or important information emerges and collecting additional data will not add to the understanding of how patients perceive the concept of interest and the items in a questionnaire.
FDA PRO Guidance (2009); Patrick et al. Value in Health (14):

16. Mixed Methods Development
Concept Elicitation (Qual)
Item Generation
Cognitive Debriefing (Qual)
Pilot Survey 1 (Quant)
Revision
Pilot Survey 2 (Quant)
Formal Testing (Quant)
Combines both Qual-itative and Quant-itative data in the development of the measure
Often done in an iterative process
Example:
Conduct traditional development
Conduct Initial Pilot Quant Study to test measure (can use CTT, IRT, RMT)
Identify potential issues (scaling issues, fit, redundancy, measurement gaps)
Make additional revisions to measure
Evaluate changes in additional cognitive interviews
Conduct a second Wave Pilot Quant study to test revisions
After revisions are complete, move to formal psychometric testing

17. Example: Pain Assessment for Lower Back Symptoms
Original 0-10 NRS Response Scale
Revised 4-level categorical VRS
Item Threshold Map
Person-Item Map
Blum et al., ISOQOL, 2016

18. Conceptual Framework
The conceptual framework explicitly defines the concepts measured by the instrument in a diagram that presents a description of the relationships between items, domain (subconcepts), and concepts measured and the scores produced by a PRO instrument.
The conceptual framework of a PRO instrument will evolve and be confirmed over the course of instrument development as a sponsor gathers empiric evidence to support item grouping and scores (e.g. psychometric evaluation, factor analysis, finalization of scoring algorithm)
FDA PRO Guidance (2009); Patrick et al. Value in Health (14):

19. Summary
PROs can be used to capture the patient’s perspective on the signs, symptoms and impacts associated with a disease or condition.
PRO measures can be used to assess treatment effect in clinical trials
In addition to being used in clinical trials, PROs can be used in real-world studies (e.g. surveys, registries, observational studies) as well in clinical practice and in quality measures
Qualitative interviews ensure the selection of relevant concepts and patient understanding.
Need to ensure the appropriate context of use
It is important to establish content validity before other measurement properties are evaluated.
Qualitative (unstructured) data are analyzed using different methods from structured data (including PRO data collected in studies)
Well established methods for analyzing qualitative data
Mixed Methods approaches can combine both qualitative and quantitative data as part of instrument development

20. Questions?