Presentation on theme: "MEASUREMENT Goal To develop reliable and valid measures using state-of-the-art measurement models Members: Chang, Berdes, Gehlert, Gibbons, Schrauf, Weiss."— Presentation transcript:
1 MEASUREMENT GoalTo develop reliable and valid measures using state-of-the-art measurement modelsMembers: Chang, Berdes, Gehlert, Gibbons, Schrauf, Weiss
2 Why Item Response Theory? Classical Test Theory (Traditional)Item Response Theory (Modern)Measures of precision fixed for all scoresPrecision measures vary across scoresLonger scales increase reliabilityShorter, targeted scales can be equally reliable (Short Form)Scale properties are sample dependentItem & scale properties are invariant within a linear transformation (DIF)Comparing person scores dependent on item setPerson scores comparable across different item sets (CAT)Comparing respondents requires parallel scalesDifferent scales can be placed on a common metric (Instrument Linking/Equating)Mixed item formats leads to unbalanced impact on total scale scoresEasily handles mixed item formatsSummed scores are on an ordinal scaleScores on interval scaleGraphical tools for item and scale analysis
3 Item Response Theory (IRT) A family of mathematical descriptions of what happens when a person meets a test or survey questionRelates characteristics of items (item parameters) and characteristics of persons (person latent traits) to the probability of a correct or rating/categorical responseModels the test-taking behavior at the item levelItem response theory (IRT) is a statistical theory consisting of mathematical models expressing the probability of endorsing a particular response to a test or survey item as a function of the abilities or latent traits of the persons and of certain characteristics of the item
6 Polytomous IRT Models Polytomous 1-PL (threshold) Partial CreditRating Scale2-PL (threshold & discriminating)NominalGraded ResponseGeneralized Partial Credit2=Yes, Limited a little1=Yes, Limited a lot3=No, Not Limited at all123* Vigorous activities, such as running, lifting heavy objects, participating in strenuous sports
7 Potential Advantages of Using IRT in “Geriatric” Pain Assessment Refine existing instrumentsEvaluate item and scale characteristicsEvaluate different response formatsDetect differential item functioningEvaluate person fit (clinical diagnosis)Equate/Link instrumentsEstablish item banks and brief formsDevelop computerized adaptive testing
8 Item Banking and CAT A B C D E F Item Pool (Sets of Questions) IRT Q newItem Pool (Sets of Questions)IRTQQQQItem Bank (Catalogued; Hierarchically Structured)CATBrief Forms
9 Principles of Adaptive Testing IRT pre-calibrated item bankInitial item selectionTest scoring methodItem selection during test administrationStopping rulesA procedure for estimating a person’s trait or ability levelA procedure fro choosing, from an available item bank, the item that is maximally informative at a person’s current trait-level estimateA termination rule used to discontinue item administration
10 Item Bank Set of carefully IRT-calibrated questions Items covers entire latent trait continuumItems represent differing amounts of traitItems represent differing amounts of informationBasis for tailored/adaptive testingItems can be selected to maximize precision and retain clinical relevance
11 Item Banking is Inter-disciplinary PsychometriciansInformation scientistsClinicians/healthcare providersOutcomes researchersContent experts…
12 Approaches to Develop Item Banks Top-Down ApproachBottom-Up Approach
13 Development and Maintenance of an Item Bank How to best calibrate existing items?Model selectionWhose item parameters to use?Standardization?Generic vs. disease-specificItem parameter driftAnchor or Re-calibrate?How to write and best test new items?
14 Adaptive TestAn adaptive test is a tailored, individualized measure which involves selecting a set of test items for each individual that best measures the psychological characteristics of that person (Weiss, 1985)Weiss DJ. Adaptive testing by computer. J Consult Clin Psychol. Dec 1985;53(6):
15 Why Computerized Adaptive Testing? Adaptive testing selects questions based on previous responsesTailored item and test difficultiesEliminates floor and ceiling effectsRequire fewer questions to arrive at an accurate estimateAutomate question administration, data recording, scoring, and prompt reportingAllows for immediate feedbackAdaptive testing is a process of test administration in which items are selected on the basis of the examinee’s responses to previously administered itemsCAT is a special type of computerized testing that targets the “difficulty” of questions to the “ability” of examinees
16 CAT Algorithm Score Item Estimate Latent Trait (Theta) Administer Item of Median Difficulty (or Screening Item)Score ItemEstimate Latent Trait (Theta)Termination Criterion SatisfiedChoose and Administer Next Item with Maximum InformationNoYesStop
17 Increase of Accuracy of Ability or Latent Trait Estimation in CAT For each item added to the test, the width of the interval decreases.Item 1-5Item 1-4Item 1-3Item 1-2Item 1Ability ()
18 Potential Problems with CAT in Pain and Health Outcomes Measurement Context effectsUnbalanced contentTime frameResponse categoriesMultidimensionality
19 What kind of short form? Question 1 0 I do not feel sad. 1 I feel sad Rarely or none of the time (less than 1 day)Some or a little of the time (1-2 days)Occasionally or a moderate amount of time (3-4 days)All of the time (5-7 days)1. I was bothered by things that usually don't bother meQuestion 10 I do not feel sad.1 I feel sad2 I am sad all the time and I can’t snap out of it.3 I am so sad or unhappy that I can’t stand it.Are you basically satisfied with your life? True/False
20 MORE Research Still Needed for Effective CAT Implementation Item productionItem statisticsItem exposureMaintaining a valid bank of items for test constructionFairnessDelivery optionsEffects of modes of administrationCost-benefit considerations
21 Infrastructure of a National Geriatric Pain Item Bank SubscriberPublicIndividual ResearchersPharm. IndustriesNon-profit InstitutionsGovernment AgenciesNational “Central” Item BankCustomized Information Retrieval; CAT; (automated) Brief FormCollectorAnalyzerBuilderRetrieverConsortium ApprovalIRT AnalysesItem Parameters
22 An Integrated Solution for Pain and Outcomes Assessments Chang, C.-H., & Yang, D. (2003, April 15). Patient-Reported Outcomes Information Technology: The PROsITTM System. ISPOR CONNECTIONS, 9(2), 5-6.
Your consent to our cookies if you continue to use this website.