1. 17 March 2017 | DIA Statistics Community Webinar
Developing PRO Instruments in Clinical Trials: Issues, Considerations, and Solutions
17 March 2017 | DIA Statistics Community Webinar

2. Developing PRO Instruments in Clinical Trials: Issues, Considerations, and Solutions
Chairs: Laura Lee Johnson (FDA)
Cheryl Coon (Outcometrix)
Speakers: Wen-Hung Chen (FDA)
Dennis Revicki (Evidera)
Lisa Kammerman (AstraZeneca)

3. The views expressed in this presentation are those of the speakers, and do not necessarily represent an official FDA position.

4. Let’s Begin with a Poll...
Please raise your hand if you have heard of the term Clinical Outcome Assessments (COAs) before today?
If you have, do you know what COAs refer to?
Please raise your hand if you have no idea what I am talking about?
By the way, this is not uncommon.

5. A New Era of Patient Empowerment
Dr. Janet Woodcock (2015):
“It turns out that what is really bothering the patient and what is really bothering the doctor can be radically different things….patients are true experts in their disease.
It's clear you have to start with an understanding of the impact of the disease on the people who have it, and what they value most in terms of alleviation before you set up a measurement and go forward with truly patient-focused drug development.”
(PDUFA V Clinical Outcome Assessment Public Workshop, April 2015)

6. What are COAs? Clinical Outcome
An outcome that describes or reflects how an individual feels, functions, or survives
Clinical Outcome Assessment (COA)
Assessment of a clinical outcome can be made through report by a clinician, a patient, a non-clinical observer or though a performance based assessment. There are four types of COAs:
Clinician-reported outcomes (ClinROs)
Observer-reported outcomes (ObsROs)
Patient-reported outcomes (PROs)
Performance outcomes (PerfOs)
BEST (Biomarkers, EndpointS, and other Tools)

7. Patient-Reported Outcomes (PROs)
A measurement based on a report that comes directly from the patient (i.e., study subject) about the status of a patient’s health condition without amendment or interpretation of the patient’s response by a clinician or anyone else.
A PRO can be measured by self-report or by interview provided that the interviewer records only the patient’s response.
Symptoms or other unobservable concepts known only to the patient can only be measured by PRO measures. PROs can also assess the patient perspective on functioning or activities that may also be observable by others.

8. PRO Questions: Good/Bad/Ugly
REAL example. Asked…
We want to know the severity of bowel movement issue
“What is the severity of your bowel movement? Very severe, moderately severe, mild, none”
What does this mean? Unfortunately, not a whole lot
Usually not this bad
Source: Johnson 2016

9. Goals of this webinar
This webinar will address when and how to evaluate the psychometric properties of a patient-reported outcome (PRO) instrument or other clinical outcome assessment (COA) used to construct a primary endpoint in a clinical trial
What level of evidence is needed in terms of psychometric properties?
What is the ideal setting for evaluating psychometric properties?
What are the risks and ramifications for deviating from the ideal setting?
What other things should be considered when planning for a psychometric evaluation using clinical trial data (e.g., power, blinding, timing)?

10. What are psychometric properties?
Francis Galton defined psychometrics as “The art of imposing measurement and number upon operations of the mind”1 (1879!)
Psychometric properties are statistical information intended to demonstrate that an instrument measures what it is supposed to measure (aka validity) and measures it in a precise manner (aka reliability)
While not a psychometric property, score interpretation is a critical component to psychometrics – once you know that your instrument is reliable and valid, what does a score, or a score change, mean?
Psychometrics provide confidence that scores from an instrument are truly representative of the concept in a clinical trial endpoint
1. Galton, F. (1879). Psychometric experiments. Brain: A Journal of Neurology, 11,

11. How are psychometric properties evaluated?
Psychometric methods may be similar to or different from biostatistical methods
E.g., Construct validity can be assessed by comparing scores from groups known to be different using ANOVA
E.g., Reliability can be computed across various ranges of the scale using item response theory
While hypothesis testing is relevant, psychometric analyses are generally more exploratory in nature than traditional biostatistical analyses
Methods to assess many psychometric properties can rely on cross- sectional data (e.g., reliability, validity, dimensionality), while others require longitudinal data from an interventional study (e.g., responsiveness to change, score interpretation)

12. Part One: Setting the Stage
Wen-Hung Chen (WHC)
US Food and Drug Administration
Clinical Outcome Assessment Staff, OND/CDER
Lisa Kammerman (LK)
AstraZeneca
Regulatory Specialist, Oncology, Biometrics and Information Sciences
Dennis Revicki (DR)
Evidera
Senior Vice President, Outcomes Research

13. Wen-Hung Chen
Pieces and level of evidence required in terms of demonstrating an instrument’s psychometric properties
Ideal setting(s) to evaluate a COA’s psychometric properties

14. Evaluation of Psychometric Properties of Clinical Outcome Assessment for use in Clinical Trial: A Regulatory Perspective
Wen-Hung Chen, PhD
Clinical Outcome Assessment Staff
Office of New Drugs (OND)
Center for Drug Evaluation and Research (CDER)
Food and Drug Administration (FDA)

15. Disclaimer
The views expressed in this presentation are those of the speaker, and do not necessarily represent an official FDA position.

16. Evidence of Treatment Benefit
Direct evidence of treatment benefit is derived from studies with endpoints that measure survival, or how patients feel and function in daily life
Indirect evidence of treatment benefit is derived from studies with endpoints that measure other things (e.g., biomarkers*, distal measures) that are related to how patients survive, feel, or function
*Biomarker: a physiologic, pathologic, or anatomic characteristic that is objectively measured and evaluated as an indicator of some normal or abnormal biologic function, process or response to a therapeutic intervention

17. Types of Clinical Trial Study Endpoints as Seen by FDA/CDER
Performance Outcomes (PerfOs)
Clinician-reported Outcomes (ClinROs; includes overall survival)
Observer-reported Outcomes (ObsROs)
Patient-reported Outcomes (PROs)
Clinical Outcome Assessments (COAs)
Often a biomarker* that is intended as a substitute for how a patient feels, functions, or survives
Surrogates

18. When is a Clinical Outcome Assessment Adequate for use?
Regulatory standard: measures are well-defined and reliable
Empiric evidence demonstrates that the score quantifies the concept of interest in the targeted context of use
What does this mean?
This means measuring the right thing (concept of interest), in the right way in a defined population (targeted context of use), and the score that quantifies that ‘thing’ does so accurately and reliably, so that the effects seen in the outcome assessment can be interpreted as a clear treatment benefit.

19. FDA PRO Guidance
Defines good measurement principles to consider for “well-defined and reliable” (21 CFR ) PRO measures intended to provide evidence of clinical benefit
All COAs can benefit from the good measurement principles described within the guidance
Provides optimal approach to PRO development; flexibility and judgment needed to meet practical demands

20. Well-defined and Reliable
The tool adequately measures the concept of interest in the context or clinical setting of interest
To assess this, we review the tool’s measurement properties:
Content validity
Construct validity
Reliability
Ability to detect change
Information to support interpretation of change

21. Development Process for a Clinical Outcome Assessment

22. Evaluation of Measurement Properties
Focus on basic analysis and build up the evidence methodically using a systematic approach
The quantitative/psychometric evidence described in the next few slides should be gathered in a sample of patients with characteristics consistent with the targeted patient population expected in trials and targeted context of use

23. Evaluation of Measurement Properties (continued)
What quantitative evidence that we would like to see 
Item descriptive statistics including frequency distributions of both individual items and overall score, floor and ceiling effects, and percentage of missing responses
Inter-item correlation and dimensionality analysis (e.g., factor analysis or principal components analysis, evaluation of conceptual framework).
Item inclusion and reduction decisions, identification of subscales (if any), and modification to conceptual framework

24. Evaluation of Measurement Properties (continued)
What quantitative evidence and summaries that we would like to see
Preliminary scoring algorithm including how missing data will be handled
Reliability (e.g., Test-retest reliability, internal consistency, inter-rater reliability where applicable)
Construct Validity
Convergent and discriminant validity (e.g., correlation with other instruments assessing similar concepts)
Known groups validity methods (e.g., difference in scores between subgroups of subjects with known status)
Final instrument, conceptual framework, and scoring algorithm including the handling of missing data

25. Last but not the Least: Thinking about Meaningful Change
How much change is meaningful?
Anchor-based method
Cumulative distribution function

26. Other Considerations COA ≠ Study Endpoint
Study endpoint should be constructed, from the COA, in the way that can be used to clearly describe the treatment benefit
Endpoint hierarchy

27. Conclusion A COA is evaluated in its totality in the context of use
Content validity
Other measurement properties
The interpretation of the score and what is clinically meaningful change
Early in the drug development and prior to confirmatory studies
Plan ahead the study endpoint and the endpoint hierarchy

28. Lisa Kammerman Risks/ramifications of deviating from ideal setting
Do risks/ramifications depend on position in endpoint hierarchy

29. Risks and ramifications when deviating from the ideal setting
Lisa A. Kammerman, Ph.D.

30. What is the ideal setting? (Recap from Dr. Chen’s talk)
Goal: A COA that is adequate for use in a clinical trial
How?
Evaluate psychometric properties of COA
When?
Early in medical product development program
Prior to confirmatory studies
What type of setting?
Non-interventional observational stand-alone validation study
Phase 2 Study
Post hoc secondary analysis from earlier study

31. What is a less than ideal setting?
Sometimes, plans don’t go as expected – bad luck
You started early in your development program, but
Results of psychometric evaluations weren’t great
More time needed to revise instrument and retest
Multi-cultural translation work doesn’t go well
Result: May need to do more work during the confirmatory study

32. What is a less than ideal setting?
Sometimes, you’re working with a rare disease
You may not be able to start early; often there’s only 1 study
May need to create COAs based on expert and patient input only
May not have the time or capability to do the recommended evaluations
Result: May need to do more work during the confirmatory study

33. What is a less than ideal setting?
Sometimes, you started too late
You didn’t allow enough time
May have some preliminary results
But results are inconclusive or not comprehensive in order to conclude the COA is appropriate for use
Result: May need to do more work during the confirmatory study

34. What work can you do during a confirmatory study?
Create a substudy to evaluate psychometric properties
Use a subset of data from a confirmatory study
Perhaps an interim analysis from an ongoing study
Perhaps a subset from a completed study
Evaluate psychometric properties from subset
Use evaluation to update instrument for use at the final analysis
Questions
What is the size of the substudy
Who is included in the substudy
Does the final analysis include all subjects

35. Risks of deviation from ideal setting:
Psychometric evaluation during the confirmatory study
Type of endpoint
Primary
Secondary, intended for labeling
Exploratory
Reason
Rare disease
Started early, but unforeseen problems
Started late
If using an interim analysis,
Will results be highly data dependent, and can’t generalize beyond study
Risks of unblinding study

36. Dennis Revicki
Who takes on risk in each case? Sponsors? The FDA? The general public?

37. WHO TAKES ON RISK IN EACH CASE? SPONSORS? FDA? GENERAL PUBLIC?
Dennis Revicki, PhD
Outcomes Research
Evidera, Bethesda, MD, USA

38. Two main possible approaches
Risks and Implications for Deviating from Ideal Approach to Evaluating Psychometric Properties of COAs
Sometimes clinical development programs require psychometric evaluation of clinical outcome assessment (COAs) in Phase III clinical trials
Attributable to timing of COA development and schedule for conducting Phase III clinical trials
Tension between systematic development of new COAs and interest in advancing clinical development program (time is money)
Two main possible approaches
Design and conduct parallel stand-alone psychometric evaluation study
Planned psychometric sub-study and blinded analysis of clinical trial data

39. Psychometric study may demonstrate inadequate psychometric properties
Risks Associated with Parallel Stand-alone Psychometric Evaluation Study: Sponsor Risks
Recruiting patients for psychometric study may compete with recruiting for clinical trial
Psychometric study may demonstrate inadequate psychometric properties
Unless sufficient changes in clinical status are observed, may not be able to identify thresholds for interpreting change (i.e., responder definitions)
Psychometric study may not be completed by clinical trial data lock

40. Risks Associated with Psychometric Sub-study: Sponsor Risks
Administrative issues and difficulty in maintaining fidelity of psychometric sub-study
Resistance from clinical investigators and research staff
Risk that psychometric sub-study sample differs in terms of demographic or clinical characteristics from Phase III clinical trial sample
Challenges in maintaining mask to treatment group assignment for the analysts conducting the psychometric analyses
Psychometric sub-study may demonstrate inadequate psychometric properties

41. Risks Associated with Phase III Clinical Trial: Sponsor Risks
Conducting and maintaining fidelity of psychometric sub-study may impact conduct of the clinical trial
Reduction in clinical trial sample size (assuming sub-study patients not included in efficacy analyses)
May be minimized by increasing overall sample size to maintain statistical power for efficacy analyses
Increase in clinical trial expenditures
Can psychometric sub-study data be included in the clinical trial efficacy analyses?
Regulatory bodies recommend not including the sub-study data
May be possible to include these data in a sensitivity analysis

42. Risks Associated with Phase III Clinical Trial: COA is Primary or Secondary Endpoint
Increased risk associated with taking the psychometric sub-study approach
Risk that psychometric analyses demonstrate that the COA does not have adequate measurement properties (reliability, validity, responsiveness)
May find that estimated responder definition criteria is not demonstrated and/or require larger sample sizes
May be unknown whether lack of responsiveness is attributable to treatment or the COA
Basically proceeding with Phase III clinical trial with a COA endpoint with unknown psychometric characteristics
COA designated as primary endpoint more risky than COA designated as secondary endpoint
Not advisable to have COA with unknown psychometric qualities as primary endpoint

43. Risk to pharmaceutical or biotech sponsor
Overall Risk of These Approaches to Psychometric Evaluation During Phase III
Risk to pharmaceutical or biotech sponsor
Risk to regulatory agencies (Food and Drug Administration, European Medicines Agency, others)
Risk to study participants and the general public

44. Risk to Pharmaceutical or Biotech Sponsor
Some risk to sponsors in conducting stand-alone psychometric evaluation studies for new COAs
Substantial risk to sponsors associated with psychometric sub-study in Phase III clinical trials
Risk is reduced if there is some evidence (even from small studies) supporting psychometric properties of COA (dimensionality, reliability, validity) and content validity
Depending on findings of the psychometric analyses, potential for delay in clinical development program

45. Risk to Regulatory Agencies
Little overall risk to regulatory agencies
Regulatory agencies will not have sufficient evidence on COA to make a confident decision about efficacy of treatment
May come under criticism for delaying clinical development programs by recommending more ideal COA development and psychometric evaluation

46. Risk to Study Participants and General Public
Study participants may be exposed to adverse effects of treatment unnecessarily in clinical trial with inadequate COA endpoints
For general public and health care system, may delay in gaining access to potentially effective treatments

47. Part Two: Practical Implications
Panelists will provide 2 minute introduction to each of four topics, followed by audience questions and panel discussion
Ideal clinical trial scenario (WHC)
Using single clinical study to both develop instrument and demonstrate efficacy of medical product (LK)
Issues with using an interim analysis to evaluate measurement properties (LK)
Masking psychometricians to treatment group (DR)

48. Practical implications: Ideal clinical trial scenario
Wen-Hung Chen, PhD
Clinical Outcome Assessment Staff
Office of New Drugs (OND)
Center for Drug Evaluation and Research (CDER)
Food and Drug Administration (FDA)

49. The Setting to Evaluate the Measurement Properties of a COA
An important component of patient-focused drug development is describing the patient’s perspective of treatment benefit in labeling based on data from patient-focused COAs. Therefore, early in product development, we encourage sponsors to consider incorporating well- defined and reliable patient-focused COAs as efficacy endpoints in clinical trials, when appropriate, and to discuss those measures with the FDA in advance of confirmatory trials.
Non-interventional observational stand-alone validation study
Phase 2 Study
Post-hoc secondary analysis from earlier study

50. Advantages and Disadvantages of Conducting Psychometric Analyses Using Clinical Trial Data*
Ideally using phase II rather than phase III clinical trial data
Ideal situation is when some subjects are improving, some subjects remain the same, and some subjects are worsening in clinical status
Problem situation #1:
Active treatment is highly effective and there is only a small placebo group
May inflate estimates of responder definition and responders
Problem situation #2:
Active treatment is not very effective and small sample size
May attenuate estimates of responder definition and responders
* Slide courtesy of Dr. Dennis Revicki

51. Practical implications: Issues with using an interim analysis to evaluate measurement properties
Lisa A. Kammerman, Ph.D.

52. Using a single clinical study to develop instrument and assess efficacy in that clinical study
Instrument will be highly dependent on data collected in the study
Certain characteristics will be unique to the study, for example:
Enrollment criteria
Study sites and investigators
Item selection and deletion dependent on population
How can we be assured that we are measuring what we believe we are measuring without verification from an independent set of data.
If we conclude there’s a difference between treatment groups, how do we interpret the difference when we have only the single study.
If we want to use the instrument in a new study, can we use the same instrument?
Trial characteristics may be different
Study population may be different

53. Using an interim analysis to evaluate measurement properties
Need processes to preserve integrity of study
Need to consider timing of analysis
A study of long duration that is slowly enrolling (e.g., a rare disease)
A study of long duration that enrolls rapidly
Etc
Need sufficient number of subjects with endpoint of interest
If you conclude the instrument needs modification …
How will you do that
How many subjects remain to be enrolled
How many subjects currently enrolled
Scoring algorithm likely to be data dependent on interim analysis data set
Implications for overall efficacy analysis
Implications for generalizing to other studies that use instrument
If endpoint is not primary, then wait until end of the study

54. Risks of deviation from ideal setting:
Psychometric evaluation during the confirmatory study
Type of endpoint
Primary
Secondary, intended for labeling
Exploratory
Reason
Rare disease
Started early, but unforeseen problems
Started late
If using an interim analysis,
Will results be highly data dependent, and can’t generalize beyond study
Risks of unblinding study

55. MASKING PSYCHOMETRICIANS TO TREATMENT GROUPS FOR PSYCHOMETRIC ANALYSES
Dennis A. Revicki, PhD
Outcomes Research, Evidera, Bethesda, MD, USA

56. Masking Psychometricians to Treatment for Psychometric Analyses of Clinical Trial Data
Essential to mask psychometricians to treatment group for psychometric analyses of phase II or III clinical trial data
Decisions about item retention and deletion needs to be made without reference to treatment group membership
Usual approach is to provide psychometrician data files without any reference to treatment group status
Contract research organization vendors
In-house biostaticians (although often without psychometric expertise)
No adverse event data is provided, which could potentially be used to identify treatment group
Provide psychometrician with only those data files necessary for conducting the planned psychometric analyses

57. Masked to treatment group membership
Masking Psychometricians: Independent Psychometric Monitoring Committee
Set up like an independent data monitoring committee for a clinical trial
Includes psychometricians, clinicians and biostatisticians not directly involved with clinical trial
Masked to treatment group membership
Reviews psychometric analyses and makes independent decisions about item retention and deletion, and responder definitions

58. Part Three: Q&A Comments 2017 DIA Annual Meeting Session
Please type and submit your questions
Include the presenter’s name
2017 DIA Annual Meeting Session
Developing and Evaluating PRO Instruments in Clinical Trials
Overview and Importance of Key Psychometric Characteristics
Statistical Issues for Applications using Registration Trials
Risks and Advantages for Applications using Registration Trials