1 Evaluating Multi-Item Scales Using Item-Scale Correlations and Confirmatory Factor Analysis Ron D. Hays, Ph.D. RCMAR Methods Seminar June 18, 2012, 3-4pm Drew Cobb Room 131/Broxton 2 nd Floor Conference Room

2 Development and Evaluation of PROMIS GI Distress Scale (n = 574) Dinesh Khanna and Brennan Spiegel, PIs Irritable Bowel Syndrome, Irritable Bowel Disease, Scleroderma 61% female; median age = 48 (18-85 yr. range) Some college = median educational level Race/ethnicity – 13% Hispanic – 17% non-Hispanic black – 12% non-Hispanic Asian – 5% non-Hispanic other

3 8 Hypothesized Scales (101 items) 1. Gastroesphageal reflux (29 items) 2. Disrupted swallowing (7 items) 3. Diarrhea (7 items) 4. Bowel incontinence/soilage (4 items) 5. Nausea and vomiting (13 items) 6. Constipation (15 items) 7. Belly pain (8 items) 8. Gas/bloat/flatulence (18 items)

4 Gastroesphageal Reflux (29 items) In the past 7 days … How often did you have regurgitation—that is, food or liquid coming back up into your throat or mouth without vomiting? – Never – One day – 2-6 days – Once a day – More than once a day

5 Disrupted swallowing (7 items) In the past 7 days … How often did food get stuck in your throat when you were eating? – Never – Rarely – Sometimes – Often – Always

6 Diarrhea (7 items) In the past 7 days … How many days did you have loose or watery stools? – No days – 1 day – 2 days – 3-5 days – 6-7 days

7 Bowel incontinence (4 items) In the past 7 days … How often did you have bowel incontinence— that is, have an accident because you could not make it to the bathroom in time? – No days – 1 day – 2-3 days – 4-5 days – 6-7 days

8 Nausea and vomiting (13 items) In the past 7 days … How often did you have nausea—that is, a feeling like you could vomit? – Never – Rarely – Sometimes – Often – Always

9 Constipation (14 items) In the past 7 days … How often did you strain while trying to have bowel movements? – Never – Rarely – Sometimes – Often – Always

10 Belly pain (8 items) In the past 7 days … How often did you have belly pain? – Never – Rarely – Sometimes – Often – Always

11 Gas/bloat/flatulence (18 items) In the past 7 days … How often did you feel bloated? – Never – Rarely – Sometimes – Often – Always

12 Problematic Item* Constipation (Gisx62) In the last 7 days … How often did you have a bowel movement? – Never {3} – One day {2} – 2-6 days {1} – Once a day {1} – More than once a day {1}

13 Problematic Item #2 Gastroesphageal Reflux (Gisx22) In the last 7 days … How often did you burp? – Never {1} – One day {2} – 2-6 days {3} – Once a day {3} – More than once a day {4}

14 Problematic Item #3 Gas/Bloat/Flatulence (Gisx105) In the last 7 days … How often did you pass gas? – Never {1} – Rarely (only once or twice a day) {2} – About every 3-4 hours {3} – About every 2 hours {4} – About every hour {4}

15 Internal Consistency Reliability for 8 Scales (101 items) ScaleNumber of itemsAlpha Gastroesphageal Reflux Disrupted swallowing70.90 Diarrhea70.89 Bowel incontinence40.87 Nausea/vomiting Constipation Belly pain80.93 Gas/bloat Reliability Standards: 0.70 or above for group comparisons; 0.90 for individual assessment How often did you have a bowel movement? did not go with Constipation scale.

16 Item-scale correlation matrix

17 Item-scale correlation matrix

18

19

20 Summary of Item-Scale Correlations Gisx22 (burp) correlates only 0.30 with Gastroesphageal Reflux scale Gisx33 (pain in chest when swallowing food) correlates as highly with Gastroesphageal Reflux as with Disrupted swallowing scale Three items correlate as highly with Disrupted Swallowing as with Gastroesphageal Reflux scale: – Gisx28 (lump in throat frequency) – Gisx29 (lump in throat interfere) – Gisx30 (lump in throat bother)

21 Confirmatory Factor Analysis

22 Confirmatory Factor Analysis Eight-factor categorical model fit the data well (CFI=0.952 and RMSEA=0.049). Lower loadings on Gastroesphageal Reflux for: – Gisx22 (burp frequency): – Gisx20 (pain behind breastbone): Secondary loadings on Gas for: – GIsx48 (thought gas but liquid; BI): – GIsx72 (unfinished frequency; Con.): – GIsx73 (feeling unfinished bother; Con.):

23 Correlated Residuals r = 0.58 (Gas/bloat/flatulence) – GI108 (freq. gurgling in belly) with GI109 (freq. gurgling when not hungry) r = 0.53 (Constipation) – GI75 (freq. pain in anus) with GI76 (bother pain in anus) r = 0.50 (Gas/bloat/flatulence) – GI106 (gas interference) with GI107 (gas bother)

24 Item Response Theory (IRT) IRT models the relationship between a person’s response Y i to the question (i) and his or her level of the latent construct  being measured by positing b ik estimates how difficult it is for the item (i) to have a score of k or more and the discrimination parameter a i estimates the discriminatory power of the item.

25 Item Responses and Trait Levels Item 1 Item 2 Item 3 Person 1Person 2Person 3 Trait Continuum

26 Bowel Incontinence Items How often did you soil or dirty your underwear before getting to a bathroom? How often did you have bowel incontinence— that is, have an accident because you could not make it to the bathroom in time? How often did you leak stool or soil your underwear? How often did you think you were going to pass gas, but stool or liquid came out instead?

27 Graded Response Model Parameters (Incontinence) ItemSlope Threshold 1Threshold 2Threshold 3Threshold 4 Gisx Gisx Gisx Gisx How often did you soil or dirty your underwear before getting to a bathroom? 2. How often did you have bowel incontinence—that is, have an accident because you could not make it to the bathroom in time? 3. How often did you leak stool or soil your underwear? [No days; 1 day; 2-3 days; 4-5 days; 6-7 days] 4. How often did you think you were going to pass gas, but stool or liquid came out instead? [Never; Rarely; Sometimes; Often; Always]

28 Questions?

29 Coefficient Alpha Formula is Two-Way Fixed Model Below ModelIntraclass CorrelationReliability One- way Two- way fixed Two- way random BMS = Between Ratee Mean Square N = n of ratees WMS = Within Mean Square k = n of replicates JMS = Item or Rater Mean Square EMS = Ratee x Item (Rater) Mean Square

30 Confirmatory Factor Analysis Fit Indices Normed fit index: Non-normed fit index: Comparative fit index:  -  2 null model 2  2 null  2 model 2 - df nullmodel 2 null  df -1  - df 2 model  - 2 null df null 1 - RMSEA = SQRT (λ 2 – df)/SQRT (df (N – 1))