Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 MEASUREMENT Goal To develop reliable and valid measures using state-of-the-art measurement models To develop reliable and valid measures using state-of-the-art.

Similar presentations


Presentation on theme: "1 MEASUREMENT Goal To develop reliable and valid measures using state-of-the-art measurement models To develop reliable and valid measures using state-of-the-art."— Presentation transcript:

1 1 MEASUREMENT Goal To develop reliable and valid measures using state-of-the-art measurement models To develop reliable and valid measures using state-of-the-art measurement models –Members: Chang, Berdes, Gehlert, Gibbons, Schrauf, Weiss

2 2 Why Item Response Theory? Classical Test Theory (Traditional) Item Response Theory (Modern) Measures of precision fixed for all scores Precision measures vary across scores Longer scales increase reliability Shorter, targeted scales can be equally reliable (Short Form) Scale properties are sample dependent Item & scale properties are invariant within a linear transformation (DIF) Comparing person scores dependent on item set Person scores comparable across different item sets (CAT) Comparing respondents requires parallel scales Different scales can be placed on a common metric (Instrument Linking/Equating) Mixed item formats leads to unbalanced impact on total scale scores Easily handles mixed item formats Summed scores are on an ordinal scale Scores on interval scale Graphical tools for item and scale analysis

3 3 Item Response Theory (IRT) A family of mathematical descriptions of what happens when a person meets a test or survey question A family of mathematical descriptions of what happens when a person meets a test or survey question Relates characteristics of items (item parameters) and characteristics of persons (person latent traits) to the probability of a correct or rating/categorical response Relates characteristics of items (item parameters) and characteristics of persons (person latent traits) to the probability of a correct or rating/categorical response Models the test-taking behavior at the item level Models the test-taking behavior at the item level

4 4 Likely (easy) Unlikely (hard) PoorGood Person Latent Trait Item Location Q Q Q Item-Person Map Chang & Gehlert (2002).

5 5 Dichotomous Unidimensional IRT Models 1-PL (Rasch) 1-PL (Rasch) –Difficulty (b) 2-PL 2-PL –Difficulty (b) –Discriminating (a) 3-PL 3-PL –Difficulty (b) –Discriminating (a) –Guessing (c)

6 6 Polytomous IRT Models Polytomous Polytomous –1-PL (threshold) Partial Credit Rating Scale –2-PL (threshold & discriminating) Nominal Graded Response Generalized Partial Credit 123 * Vigorous activities, such as running, lifting heavy objects, participating in strenuous sports 1=Yes, Limited a lot 2=Yes, Limited a little 3=No, Not Limited at all

7 7 Potential Advantages of Using IRT in Geriatric Pain Assessment Refine existing instruments Refine existing instruments Evaluate item and scale characteristics Evaluate item and scale characteristics Evaluate different response formats Evaluate different response formats Detect differential item functioning Detect differential item functioning Evaluate person fit (clinical diagnosis) Evaluate person fit (clinical diagnosis) Equate/Link instruments Equate/Link instruments Establish item banks and brief forms Establish item banks and brief forms Develop computerized adaptive testing Develop computerized adaptive testing

8 8 Item Banking and CAT E E D D C C B B A A new F F Item Pool (Sets of Questions) IRT Q Q Q Brief Forms CAT Item Bank (Catalogued; Hierarchically Structured) Q

9 9 Principles of Adaptive Testing IRT pre-calibrated item bank IRT pre-calibrated item bank Initial item selection Initial item selection Test scoring method Test scoring method Item selection during test administration Item selection during test administration Stopping rules Stopping rules

10 10 Item Bank Set of carefully IRT-calibrated questions Set of carefully IRT-calibrated questions Items covers entire latent trait continuum Items covers entire latent trait continuum Items represent differing amounts of trait Items represent differing amounts of trait Items represent differing amounts of information Items represent differing amounts of information Basis for tailored/adaptive testing Basis for tailored/adaptive testing Items can be selected to maximize precision and retain clinical relevance Items can be selected to maximize precision and retain clinical relevance

11 11 Item Banking is Inter- disciplinary Psychometricians Psychometricians Information scientists Information scientists Clinicians/healthcare providers Clinicians/healthcare providers Outcomes researchers Outcomes researchers Content experts Content experts …

12 12 Approaches to Develop Item Banks Top-Down Approach Top-Down Approach Bottom-Up Approach Bottom-Up Approach Health Physical Functioning PainSymptom Mental DepressionAnxiety SocialSpiritual

13 13 Development and Maintenance of an Item Bank How to best calibrate existing items? How to best calibrate existing items? –Model selection –Whose item parameters to use? –Standardization? –Generic vs. disease-specific Item parameter drift Item parameter drift –Anchor or Re-calibrate? How to write and best test new items? How to write and best test new items?

14 14 Adaptive Test An adaptive test is a tailored, individualized measure which involves selecting a set of test items for each individual that best measures the psychological characteristics of that person (Weiss, 1985) An adaptive test is a tailored, individualized measure which involves selecting a set of test items for each individual that best measures the psychological characteristics of that person (Weiss, 1985) Weiss DJ. Adaptive testing by computer. J Consult Clin Psychol. Dec 1985;53(6):

15 15 Why Computerized Adaptive Testing? Adaptive testing selects questions based on previous responses Adaptive testing selects questions based on previous responses Tailored item and test difficulties Tailored item and test difficulties Eliminates floor and ceiling effects Eliminates floor and ceiling effects Require fewer questions to arrive at an accurate estimate Require fewer questions to arrive at an accurate estimate Automate question administration, data recording, scoring, and prompt reporting Automate question administration, data recording, scoring, and prompt reporting Allows for immediate feedback Allows for immediate feedback

16 16 CAT Algorithm Administer Item of Median Difficulty (or Screening Item) Score Item Estimate Latent Trait (Theta) Terminatio n Criterion Satisfied Stop Choose and Administer Next Item with Maximum Information Yes No

17 17 Increase of Accuracy of Ability or Latent Trait Estimation in CAT Ability ( ) Item 1 Item 1-4 Item 1-2 Item 1-3 Item 1-5 For each item added to the test, the width of the interval decreases.

18 18 Potential Problems with CAT in Pain and Health Outcomes Measurement Context effects Context effects Unbalanced content Unbalanced content Time frame Time frame Response categories Response categories Multidimensionality Multidimensionality

19 19 What kind of short form? Rarely or none of the time (less than 1 day) Some or a little of the time (1-2 days) Occasional ly or a moderate amount of time (3-4 days) All of the time (5-7 days) 1. I was bothered by things that usually don't bother me Question 1 0 I do not feel sad. 1 I feel sad 2 I am sad all the time and I cant snap out of it. 3 I am so sad or unhappy that I cant stand it. Are you basically satisfied with your life?True/False

20 20 MORE Research Still Needed for Effective CAT Implementation Item production Item production Item statistics Item statistics Item exposure Item exposure Maintaining a valid bank of items for test construction Maintaining a valid bank of items for test construction Fairness Fairness Delivery options Delivery options Effects of modes of administration Effects of modes of administration Cost-benefit considerations Cost-benefit considerations

21 21 Individual Researchers Pharm. Industries Non-profit Institutions Subscriber IRT Analyses Item Parameters Consortium Approval Customized Information Retrieval; CAT; (automated) Brief Form National Central Item Bank Infrastructure of a National Geriatric Pain Item Bank Government Agencies CollectorBuilderAnalyzerRetriever Public

22 22 An Integrated Solution for Pain and Outcomes Assessments Chang, C.-H., & Yang, D. (2003, April 15). Patient-Reported Outcomes Information Technology: The PROsIT TM System. ISPOR CONNECTIONS, 9(2), 5-6.


Download ppt "1 MEASUREMENT Goal To develop reliable and valid measures using state-of-the-art measurement models To develop reliable and valid measures using state-of-the-art."

Similar presentations


Ads by Google