1. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 25 Critiquing Assessments Sherrilene Classen, Craig A. Velozo

2. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Learning Objectives After reading this chapter, you will be able to: 1.Discuss the application and importance of measurement theory relative to occupational therapy assessment 2.Describe traditional versus modern testing approaches as they relate to occupational therapy practice 3.Describe and apply a framework to evaluate assessments by type and structure 4.Recognize the uses of standardized versus nonstandardized tests

3. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Learning Objectives (Continued) 5.Define and apply the concepts of reliability to occupational therapy assessments 6.Identify the components of validity and apply that knowledge to assessments 7.Describe the basic components of item response theory (IRT) 8.Discuss factors important in critiquing IRT-based assessments

4. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Evaluation has two main purposes in the practice of OT –As part of the therapy process to aid the determination of occupational performance issues –To provide support for the evidence base of our profession OTs critique standardized assessments to determine their appropriateness for: –Individuals or groups to be evaluated –Purpose of the evaluation Introduction

5. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Traditional Approaches Measurement: rules for quantifying a classification of certain attributes or characteristics Measurement enables therapists to: –Quantify attributes of individuals in a standardized way; –Make comparisons about performance or capacity across individuals or groups of individuals; and –Document how an individual’s performance has changed over time and across performance contexts

6. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Nonstandardized versus Standardized Assessments Nonstandardized assessments –Do not follow a standard approach or protocol –May contain data collected from interviews, questionnaires and observation of performance Standardized assessments –Are developed using prescribed procedures –Are administered and scored in a consistent manner under the same conditions and test directions

7. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Types of Assessments Descriptive assessments Use items to describe individuals within groups and to characterize the differences between individuals on the attribute being measured Evaluative Assessments Use criteria or items to measure an individual’s trait or attribute over time The most appropriate characteristics included in an evaluative assessment are those that can be sensitive to change within an individual

8. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Predictive Assessments Use criteria to classify individuals to predict a certain trait in comparison to set criteria For example, a predictive tool can measure skills underlying driving performance to predict whether an older adult will be able to successfully return to driving Types of Assessments (Continued)

9. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Structure of Assessments Characteristics Format Cost Orientation Clinical utility

10. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Structure of Assessments (Continued) Construction Writing the items Scales of measurement –Nominal –Ordinal –Interval –Ratio

11. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Reliability Definition of Reliability Reliability is based on the correlation coefficient and referred to as a reliability coefficient The reliability coefficient can range from 0 to +1 –0 indicates no reliability –+1 indicates perfect reliability

12. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Theoretical Perspectives of Reliability Classical test theory suggests that: –A test score has two components—the true score and measurement error –While there are many sources of error, only one is estimated with any given study Generalizability theory –Recognizes different sources of error and attempts to quantify the error from those various sources

13. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Sources of Error Random error Is represented by inconsistencies that cannot be predicted, for example, fatigue or mechanical inaccuracy of the measurement assessment, causing the error Systematic error Predictable fluctuations occurring during measurement Usually occurs in one direction and consistently overestimate or underestimate the true score

14. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Types of Reliability Test-retest reliability: A measure of the consistency of an assessment, given to the same clients, over time Rater reliability –Intrarater reliability: The stability of data collected by one rater on two or more trials over time –Interrater reliability: Rater variability between two or more raters who measure the same clients

15. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Types of Reliability Internal consistency: Degree of agreement between the items in a test measuring an underlying trait –Alternative form reliability: Testing the same group of people on two separate occasions using two distinct but parallel forms of the test –Split-halves: The same group of people partakes in the test where the total set of items in the test is divided in half –Cronbach’s alpha or Kruder-Richardson: To estimate the reliability of scales or commonality of one item in a test with other items in a test

16. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins How High Should the Reliability Coefficient Be? The purpose of the test should be considered when deciding whether the reliability reported is acceptable A perfect correlation is indicated by a correlation coefficient expressed as r=1 0, p < 0 05 –Strong correlation is indicated if r > 0 75, p < 0 05 –Moderate correlation if r > 0 50 but < 0 75, p < 0 05 –Weaker correlation if r <0 50, p < 0 05

17. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Validity Definition of Validity Validity is defined as the extent to which any assessment measures what it is intended to measure Types of Validity Face validity: Is the measure testing what it is supposed to measure? Are the items plausible? Content validity: Does the measurement instrument reflect a specific domain of content?

18. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Types of validity (continued) Construct validity: Does the assessment measure a construct and the theoretical components underlying the construct? –Convergent validity: Is the level of agreement between two tests that are being used to measure the same construct acceptable? –Discriminant validity: Is the level of disagreement when two tests measure a trait, behavior, or characteristic acceptable?

19. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Types of validity (continued) Criterion validity: Can the outcome of one assessment be used as a substitute test to the established gold standard criterion test? –Concurrent validity: Do the results of a criterion measure and a target test, given at the same relative time point, concur with one another? –Predictive validity: Does the outcome of a target test predict a future criterion score or outcome? Ecological validity: The outcome of an assessment can “hold up” in the real-world circumstances

20. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Screening Tools A screening tool: A brief measure that tests the presence or absence of a disease, condition, or an outcome –Because screening tools are used by clinicians to determine if a client will require further treatment, it is very important to verify their validity

21. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Validity of Screening Tools Sensitivity: The predictor test’s ability to obtain a positive test when the condition really exists (a true positive) Specificity: Thee predictor test’s ability to obtain a negative result when the condition is really absent (a true negative) Positive predictive value: The probability of the client, given a certain cut point on the predictor test to fail in the actual situation (e.g., the on-road test) Negative predictive value: The probability of the client, given a cut point on the predictor test, to pass (in this case to pass the on-road test)

22. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Framework for Critiquing Assessments Purpose Structure –Format, Cost, Orientation, Clinical utility Level of measurement –Nominal, Ordinal, Ratio, Interval Psychometric properties –Reliability –Validity

23. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Modern Approaches to Critiquing Assessments – Item Response Theory Focus on the item/question not the test –Fit statistics – how well items fit the IRT model –Places the item and person on the same measurement continuum – detect “ceiling” or “floor” effects The measurement instrument is independent of the person or object being measured

24. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Proposed Scale of Motor Development

25. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Rasch Formula

26. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Comparison of Person Ability to item Difficulty

27. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Computer Adaptive Test Algorithm

28. Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Critiquing IRT-Based Assessments Unidimensionality –Determination of whether an assessment measures a single trait Local Independence –Indication of whether the items of an assessment independently contribute to the measurement of a particular trait Precision –Error and information Person-Item Match –Ceiling and floor effects