Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Patient-Reported Outcome Measurement Information System (PROMIS)

Similar presentations

Presentation on theme: "The Patient-Reported Outcome Measurement Information System (PROMIS)"— Presentation transcript:

1 The Patient-Reported Outcome Measurement Information System (PROMIS)
Nan Rothrock, Ph.D. Northwestern University May 22, 2012

2 Agenda Problems in patient-reported outcome measures
PROMIS approach to PRO instrument development Available PROMIS instruments Reliability, validity PROMIS and the FDA

3 Challenges in PRO Measurement
Many measures of same health concept Widely varying quality Difficult to compare and combine data . . . across studies . . . across conditions Complex Long These challenges were appreciated by NIH many years ago. [TRANSITION]

4 What is wrong with today's static measures ?
3 2 2 Questionnaire with a high precision - but small range 1 1 Questionnaire with a wide range - but low precision Static questionnaire are either not precise enough or their measurement range is to narrow … ceiling and floor effects … - 1 - 2 - 3

5 “The clinical outcomes research enterprise would be enhanced greatly by the availability of a psychometrically validated, dynamic system to measure PROs efficiently in study participants with a wide range of chronic diseases and demographic characteristics.” National Institutes of Health, 2003

6 PROMIS Aims Attack the Patient- Reported Outcome (PRO) “Tower of Babel” Harness modern psychometric methods Improve quality and interpretability of PROs According to Genesis, the monolingual Babylonians wanted to make a name for themselves by building a mighty city and a tower “with its top in the heavens.” God disrupted the work by so confounding their speech. Workers spoke different languages and could no longer understand one another "Then they said, 'Come, let us build ourselves a city, and a tower with its top in the heavens, and let us make a name for ourselves; otherwise we shall be scattered abroad upon the face of the whole earth.'" (Genesis 11:4). Bruegel – Flemish painter. This work is currently in Vienna. Bruegel, 1563

7 Resources Nine-year commitment of NIH $80+ million investment
15 funded research sites

8 What is PROMIS? Methodology Measures (Instruments) Software

9 Glossary Item = question or statement a patient answers
Instrument = collection of items Legacy = existing instrument that is “gold standard” or a commonly used and widely accepted instrument

10 PROMIS Instruments Domain focused, not disease focused Item Banks
Domain = feeling, function or perception you want to measure (e.g., anxiety, physical function, general health perceptions) Item Banks A large collection of items measuring one domain Any and all items can be used to provide a score Can be administered as Computerized Adaptive Tests (CATs) or fixed-length short forms

11 The Life Story of a PROMIS Item
Focus groups Archival data analysis Binning and winnowing Domain Framework Literature review Expert review/ consensus Literacy level analysis Large-scale testing Cognitive interviews Expert item revision Translation review Statistical analysis Development of new and modified items (Beginning pool 8000 total items) 28 focus groups Binning and winnowing to 1064 items Cognitive interviews (784 items) Intellectual property Calibration decisions Short form CAT Validation studies

12 What instruments were created?

13 PROMIS Domain Framework
Symptoms Physical Health Function Affect Self-Reported Health Mental Health Behavior Cognition Relationships Social Health Function

14 PROMIS Current (2012) Physical Health Banks
Adult Pediatric/Parent Proxy Pain Behavior Pain Interference Pain Intensity Fatigue Pain Interference Upper Extremity Physical Health Fatigue Mobility Sleep Disturbance Asthma Impact Sleep-related Impairment Physical Function Sexual Function

15 PROMIS Current (2012) Mental Health Banks
Adult Pediatric/Parent Proxy Anxiety Anxiety Depression Depression Anger Mental Health Anger Psychosocial Illness Impact Applied Cognition Concerns Applied Cognition Abilities Alcohol Use Alcohol Consequences Alcohol Expectancies

16 PROMIS Current (2012) Social Health Banks
Adult Pediatric/Parent Proxy Ability to Participate in Roles & Activities Peer Relationships Satisfaction with Roles & Activities Social Health Companionship Emotional Support Informational Support Instrumental Support Social Isolation

17 Sample PROMIS Fatigue Short Form
Reprinted with permission of the PROMIS Health Organization and the PROMIS Cooperative Group © 2007.

18 Language Availability
Available Universal Spanish In Process German Portuguese Mandarin Chinese French Italian Norwegian Others – see

19 What is the metric? T Score Referenced to the US General Population
Mean = 50 Standard Deviation = 10 Referenced to the US General Population

20 Ongoing PROMIS Development
Adult GI Symptoms Self-efficacy for management of chronic disease Pediatric Pain Behavior, Quality, Intensity Physical Activity Experience of Stress Subjective Well-being Impact of Child Illness on Family Family Belongingness

21 How does PROMIS compare to other PRO instruments?

22 Reliability (Precision)
This leads to precise measurement that improves the power and efficiency of clinical research.

23 Physical Function Measurement Precision and Range
PROMIS Short Form 20 items PROMIS Short Form 10 items SF items SE = 3.3 rel = 0.90 SE = 2.3 rel = 0.95 Error CAT 10 items HAQ 20 items rheumatoid arthritis patients US general population

24 Validity This leads to precise measurement that improves the power and efficiency of clinical research.

25 Scores on PROMIS measures should correlate with accepted measures of the same domain. (Concurrent Validity)

26 Depression PROMIS Depression

27 When people experience clinical benefit or decline, their PROMIS scores should also change. (Responsiveness)

28 Effect Size: PROMIS Pain Interference vs
Effect Size: PROMIS Pain Interference vs. BPI Interference (Patients with pain intensity > 4 at baseline) Back pain with sciatica for at least 6 weeks Scheduled for an epidural steroid injection Baseline, 1 month, 3 month

29 Effect Size: PROMIS Pain Behavior vs
Effect Size: PROMIS Pain Behavior vs. Roland-Morris (Patients with pain intensity > 4 at baseline)

30 PROMIS and the FDA: Agreement
Importance of PRO development to include patient voices Importance of sound measurement Confusion in selecting an instrument because of huge array of choices Ongoing discussions via Interagency Clinical Outcomes Assessment Working Group to qualify PROMIS Fatigue measures, attendance and presentations at PRO Consortium

31 PROMIS and FDA: Differences Concerning Content Validity
FDA Approach  evaluate content validity in each clinical population in which the measure may be used PROMIS Approach  there is commonality in patients’ experiences of symptoms/outcomes and their impact on QOL Need to re-validate a well-developed & valid instrument in a target population is questionable Content Validity = extent to which a scale or questionnaire represents the most relevant and important aspects of a concept in the context of a given measurement application Magasi, S. et al (2011) Content validity of patient-reported outcome measures: Perspectives from a PROMIS meeting. Quality of Life Research

32 Generic fatigue = MS-specific fatigue (r=0
Generic fatigue = MS-specific fatigue (r=0.92) (Cook et al, QOLR, 2011) Content Validity = extent to which a scale or questionnaire represents the most relevant and important aspects of a concept in the context of a given measurement application When include most relevant/important aspects of fatigue for MS, see same result as general fatigue measure

33 PROMIS FatigueSFv1.0 and PROMIS FatigueMS Scores
by Disability Status, Fatigue Severity, and Vitality Scores N PROMIS FatigueSF v1.0 PROMIS FatigueMS Mean SD Expanded Disability Status Scale (EDSS) Mild (0-4) 83 52.2 8.2 52.5 9.2 Moderate ( ) 104 60.5 6.4 60.7 5.6 Severe ( ) 43 8.3 8.7 Fatigue Severity (0-10 NRS) None/Mild (0-1) 18 43.0 4.5 42.5 5.4 Moderate (2-4) 58 51.0 6.0 51.3 6.6 Severe (5-10) 154 61.7 5.8 61.9 5.5 Vitality (item from the MOS) None/A little 52 63.8 5.3 64.2 Some 88 59.9 6.3 60.1 Quite a lot 44 55.7 56.0 6.8 Very Much 45 47.5 7.3 47.0 7.9 Known groups validity equal across 2 PROMIS short forms Cook et al, QOLR, 2011

34 Use generic measures as the foundation for PRO content validity
Supplement with targeted measures Item banking allows flexible item choice without loss of a standard scoring base Alternative is a messy array of contenders that fail to communicate across themselves regarding severity or result interpretation

35 The “Promise” of PROMIS Instruments
Comparability Provide the ability to compare or combine results from multiple studies. Reliability and Validity Reduce response burden. Improve measurement precision. Simplify administration via computer-based administration, scoring, and reporting

36 Questions. PROMIS website www. nihpromis
Questions? PROMIS website Acknowledgements National Institutes of Health (Grants U54 AR , U05 AR , U01 AR ) PROMIS Pis: David Cella, Richard Gershon, San Keller, Joan Broderick, Arthur Stone, Heidi Crane, Paul Crane, Donald Patrick, Dagmar Amtmann, Karon Cook, Darren DeWalt, Chris Forrest, jim fries, stephen haley, david tulsky, dinesh khanna, brennan spiegel, paul pilkonis, carol moinpour, arnold potosky, esi morgan dewitt, lisa shulman, kevin weinfurt

37 Thank you!

38 Minimally Important Differences (MID)

39 MID Methods IRT-based MIDs on a T-Score scale
Multiple cross-sectional and longitudinal anchors (18) Summarized with nonparametric statistics (median, interquartile range) Due to the large number of anchor-based MID estimates calculated, we summarized the results using non parametric statistics, namely medians and interquartile ranges. All self-reported anchors (not clinical). Longitudinal anchors were prospective (e.g., performance status) or retrospective (e.g., global rating of change) The MID for a scale should be larger than its measurement error. To ensure this, the lower bound of the anchor-based MID range was not allowed to go below the SEM. The SEM for the IRT-based MIDs was the average standard error for the sample. The SEM for the raw score MIDs was based on the standard formula (SEM=SD*sqrt(1-r) In the cross-sectional analysis, anchors were used to categorize patients into multiple clinically-distinct groups. Many different anchors can be used for this purpose, provided individuals can be classified into distinct categories that are both clinically relevant but also minimally different. Score differences between adjacent, clinically distinct categories represent estimates of the MID. Effect sizes for these estimates were computed by dividing the adjacent category score difference by the overall SD for the sample. Both prospective and retrospective anchors were used. For prospective, we identified the amount of change in the anchor that represents minimally important change (e.g., 1-pt change in ECOG, 2-pt change on 0-10 pain scale, or change >=MID for a multi-item scale like FACIT-Fatigue). The MID is the mean score change within those categories of minimally important change (decline or improvement). Mean change scores on the PROMIS-Cancer scales corresponding to GRC item responses of +1 or +2 (“a little better,” “moderately better”) and -1 or -2 (“a little worse,” “moderately worse”) were considered estimates of the MID.

40 T-Score MID Effect Sizes* Raw Score MID Effect Sizes‡
Short Form Results Recommended IRT-based T-Score MIDs and Raw Score MIDs for PROMIS-Cancer Short Forms in Advanced Cancer Patients Instrument T-Score MID Points T-Score MID Effect Sizes* Raw Score MID Points Raw Score MID Effect Sizes‡ Fatigue 3 -5 2-3 Pain Interference 4 -6 4-7 Physical Function 3-6 Anxiety 3-5 3-4 Depression *Calculated as the T-Score MID divided by the Assessment 1 T-Score standard deviation ‡Calculated as the Raw Score MID divided by the Assessment 1 Raw Score standard deviation All lower bounds of the MID ranges were greater than the SEMs for both IRT and raw score MIDs. Although methods and results for IRT-based T-Scores were presented in detail in this presentation, the exact same methods were used to derive the raw score MIDs. That is, we did not simply take the IRT-based MIDs and transform them to a raw score scale. 7-item fatigue SF

41 T-Score MID Effect Sizes
CAT Results Recommended IRT-based T-Score MIDs for PROMIS-Cancer CATs CAT T-Score MID Points T-Score MID Effect Sizes Fatigue Pain Interference Physical Function Anxiety Depression All lower bounds of the MID ranges were greater than the SEMs for both IRT and raw score MIDs.

Download ppt "The Patient-Reported Outcome Measurement Information System (PROMIS)"

Similar presentations

Ads by Google