1. PROMIS DEVELOPMENT METHODS, ANALYSES AND APPLICATIONS Presented at the Patient-Reported Outcomes Measurement Information System (PROMIS): A Resource for Clinical & Health Services Research, Academy Health Annual Research Meeting, Orlando, Florida, June 3, 2007 Dennis A. Revicki, Ph.D. Center for Health Outcomes Research, United BioSource Corporation, Bethesda, Maryland, USA

2. OVERVIEW Development of PROMIS item banks Development of PROMIS item banks Psychometric analysis of item bank data Psychometric analysis of item bank data Clinical and health services research applications Clinical and health services research applications

3. GOAL FOR PROMIS Improve assessment of self- reported symptoms and domains of health-related quality of life for application across a wide range of chronic diseases Improve assessment of self- reported symptoms and domains of health-related quality of life for application across a wide range of chronic diseases Develop and test a large bank of items for measuring PROs Develop and test a large bank of items for measuring PROs Develop computer-adaptive testing (CAT) for efficient assessment of PROs Develop computer-adaptive testing (CAT) for efficient assessment of PROs Create a publicly available, flexible, and sustainable system allowing researchers to access to item banks and CAT tools Create a publicly available, flexible, and sustainable system allowing researchers to access to item banks and CAT tools

4. PROMIS DOMAIN HIERARCHY Negative Impacts of illness Anxiety Anger/Aggression Depression Substance Abuse Performance Satisfaction Physical Health Satisfaction Mental Health Satisfaction Social Health Satisfaction Self- reported Health Satisfaction Other Cognitive Function Emotional Distress Role Participation Social Support Self Concept Stress Response Spirituality/Meaning Social Impact Positive Impacts of Illness Subjective Well-Being (positive affect) Meaning and Coherence (spirituality) Mastery and Control (self-efficacy) Positive Psychological Functioning Pain Fatigue Sleep/Wake Function** Sexual Function Symptoms Upper Extremities: grip, buttons, etc (dexterity) Central: neck and back (twisting, bending, etc) Activities: IADL (e.g. errands) Lower Extremities: walking, arising, etc (mobility) Function/Disability

5. Item Respons e Theory (IRT) Item Bank (IRT-calibrated items reviewed for reliability, validity, and sensitivity) Short Form Instruments CAT Items from Instrument A Item Pool Items from Instrument B Items from Instrument C New Items  Questionnaire administered to large representative sample             Secondary Data Analysis Cognitive Testing Focus Groups Content Expert Review

6. ITEM BANKS no pain mild pain moderate pain severe pain extreme pain    Pain Item Bank Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Item n These items are reviewed by experts, patients, and methodologists to make sure: Item phrasing is clear and understandable for those with low literacy Item content is related to pain assessment and appropriate for target population Item adds precision for measuring different levels of pain An item bank comprises a large collection of items measuring a single domain, e.g., pain…

7. Skewness Unidimensionality Local Independence IRT Analysis Differential Item Function Item Fit Item Development Evaluation Item Parameter Stability STEPS FOR PROMIS ITEM BANKS Qualitative Review 1 2 3 4 5 6 7 8 9 Frequency Analysis CFA Residual Correlations Item Response Curves Regression Exclusion of Items Fit Tests Simulation Studies — Focus groups and cognitive interviews < 95% response in one category >.60 factor loading <.10 residual correlation monotonic R 2 <.03 DIF p>.05 Chi 2 test ? Criteria

8. ITEM RESPONSE THEORY MODELS IRT models enable reliable and precise measurement of PROs IRT models enable reliable and precise measurement of PROs –Fewer items needed for equal precision –Makes assessment briefer More precision gained by adding items More precision gained by adding items –Reducing error and sample size requirements Error is understood at the individual level Error is understood at the individual level –Allowing practical individual assessment

9. WHICH RANGE OF MEASUREMENT? sit on the edge of the bed climb up several stairs heavy work around the house strenuous activities usual physical activities 5 = Not at all 4 = Very little 3 = Somewhat 2 = Quite a lot 1 = Cannot do 5 = Without any difficulty 4 = With a little difficulty 3 = With some difficulty 2 = With much difficulty 1 = Unable to do Are you able to … Does your health now limit you in... DisabilityPhysical Function

10. People with more fatigue Items less likely to be endorsed Items more likely to be endorsed People with less fatigue PEOPLE AND ITEMS DISTRIBUTED ON THE SAME METRIC: FATIGUE

11. BANK PRECISION LEVEL ALONG THE PAIN CONTINUUM

12. THE ADVANTAGES OF SHORT-FORMS DEVELOPED FROM PROMIS ITEM BANKS Select a set of items that are matched to the severity level of the target population. Select a set of items that are matched to the severity level of the target population. All scales built from the same item bank are linked on a similar metric. All scales built from the same item bank are linked on a similar metric.

13. FATIGUE MEASURE AND STANDARD ERROR COMPARISON BY TEST LENGTH

14. THE ADVANTAGES OF CAT-BASED ASSESSMENT 1. Provide an accurate estimate of a person’s score with the minimal number of questions. Questions are selected to match the health status of the respondent.Questions are selected to match the health status of the respondent. 2. CAT minimizes floor and ceiling effects. People near the top or bottom of a scale will receive items that are designed to assess their health status.People near the top or bottom of a scale will receive items that are designed to assess their health status.

15. Item Bank (Validated & IRT-Calibrated Emotional Distress Items) -3-20123 Severe high moderatelowvery low Emotional Distress How often did you feel nervous? All of the time Most of the time Little of the time Some of the time None of the time

16. Item Bank (Validated & IRT-Calibrated Emotional Distress Items) -3-20123 Severe high moderatelowvery low Emotional Distress How often did you feel nervous? Some of the time

17. Item Bank (Validated & IRT-Calibrated Emotional Distress Items) -3-20123 Severe high moderatelowvery low Emotional Distress How often did you feel nervous? Some of the time

18. Item Bank (Validated & IRT-Calibrated Emotional Distress Items) -3-20123 Severe high moderatelowvery low Emotional Distress How often did you feel hopeless? All of the time Most of the time Little of the time Some of the time None of the time

19. Item Bank (Validated & IRT-Calibrated Emotional Distress Items) -3-20123 Severe high moderatelowvery low Emotional Distress How often did you feel hopeless? Some of the time

20. Item Bank (Validated & IRT-Calibrated Emotional Distress Items) -3-20123 Severe high moderatelowvery low Emotional Distress How often did you feel worthless? All of the time Most of the time Little of the time Some of the time None of the time

21. Item Bank (Validated & IRT-Calibrated Emotional Distress Items) -3-20123 Severe high moderatelowvery low Emotional Distress How often did you feel worthless? Little of the time

22. Item Bank (Validated & IRT-Calibrated Emotional Distress Items) -3-20123 Severe high moderatelowvery low How often did you feel worthless? Little of the time Target in on emotional distress score

23. CLINICAL AND HEALTH SERVICES RESEARCH APPLICATIONS Brief, psychometrically sound short-form or CAT instruments Brief, psychometrically sound short-form or CAT instruments –Pain, fatigue, physical function, emotional distress, social activities/function Efficient collection of health outcomes data in clinical trials Efficient collection of health outcomes data in clinical trials –Comparing health interventions and strategies –Comparing pharmaceutical treatments Monitoring the health outcomes of populations Monitoring the health outcomes of populations –Health plan members –Medicare beneficiaries –US general population (i.e., MEPS)

24. TREATMENT COMPARISONS AND EFFECT SIZE ESTIMATES FOR BASELINE TO ENDPOINT CHANGES FOR DEPRESSION SEVERITY SCALES FOR PAROXETINE AND PLACEBO GROUPS Score Least Square Mean Change F-ValueP-ValueEffect Size ParoxetinePlacebo HDRS Total -11.4407-8.3757.450.0070.43 MADRS Total -13.617-8.79311.930.0010.54 DS-1 T-Score -17.333-12.1718.730.0040.46 DS-2 T-Score -21.919-13.69014.570.00020.59 DS-3 T-Score -23.135-14.23416.090.00010.63 a. Sample size: Paroxetine N = 98; Placebo N = 99 b. Sample size: Paroxetine N = 82; Placebo N = 85

25. SUMMARY AND CONCLUSION PROMIS item banks, short-form measures and CAT will enable the efficient and psychometrically sound assessment of health outcomes PROMIS item banks, short-form measures and CAT will enable the efficient and psychometrically sound assessment of health outcomes PROMIS items banks, instruments and software will be in the public domain PROMIS items banks, instruments and software will be in the public domain –PROMIS Health Organization –Not-for-profit organization for management and dissemination of PROMIS products Development of PROMIS item banks and instruments is ongoing Development of PROMIS item banks and instruments is ongoing –Preliminary measurement systems available late 2007 Health outcome measures may assist patients, their families, clinicians, and other health care decision-makers in understanding the outcomes of health care interventions and treatment Health outcome measures may assist patients, their families, clinicians, and other health care decision-makers in understanding the outcomes of health care interventions and treatment