1. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 47 Critiquing Assessments

2. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Learning Objectives After reading this chapter, you will be able to: 1. Describe a format for critiquing assessments in occupational therapy and apply it in practice 2. Describe the process of construction of a standardized test 3. Describe five benefits of a standardized test over a nonstandardized test

3. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Learning Objectives (cont.) 4. Define reliability and critique evidence of reliability relevant to a specific assessment 5. Define validity and critique evidence for it relevant to a specific assessment 6. Describe issues related to fair testing practices and apply them in practice

4. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Evaluation has two main purposes in the practice of occupational therapy –As part of the therapy process to aid the determination of occupational performance issues –To provide support for the evidence base of our profession Occupational therapists must critique standardized assessments to determine their appropriateness for the: –Individuals or groups to be evaluated –Purpose of the evaluation

5. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Measurement Measurement: The process of assigning numbers to represent quantities of a trait, attribute, or characteristic, or to classify objects. Measurement enables therapists to: –Quantify attributes of individuals but not individuals themselves. –Operationally define behaviors in order to make comparisons between individuals or to compare the same person at two different times.

6. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Purposes of Evaluation in Occupational Therapy Three main purposes have been identified for evaluation: 1.To help occupational therapists discriminate between members of a group. 2.To predict either future function or function in a related area. 3. To evaluate outcomes of therapeutic intervention.

7. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Critiquing Assessments Theoretical Context of Assessment In selecting an assessment, consider whether the theoretical foundation of the assessment is congruent with that of the therapist. Clinical Utility Defined as ease, efficiency, and use of a test and the clinical relevance and meaning of the information that it provides.

8. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Test Construction The test manual should describe how the test was constructed –It should give sufficient information to determine whether it was developed in a logical, systematic, and stringent fashion. Standardized Tests Have undergone a rigorous development process and are administered and scored in a prescribed manner.

9. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins A Framework for Critiquing Assessments 1.The purpose of the assessment 2.Theoretical frame of reference 3.Clinical utility 4.Technical considerations 5.Fair testing issues 6.External review comments 7.Summary of strengths and weaknesses of the test

10. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Reliability Definition of Reliability Reliability is based on the correlation coefficient and referred to as a reliability coefficient. The reliability coefficient can range from 0 to +1 –0 indicates no consistency –+1 indicates perfect consistency

11. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Theoretical Perspectives of Reliability Classical test theory suggests that: –A test score has two components—the true score and measurement error –While there are many sources of error, only one is estimated with any given study Generalizability theory also recognizes different sources of error and attempts to quantify the error from those various sources.

12. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Sources of Error Variance There are six sources of variation that can contribute to measurement error: 1.Lasting general characteristics 2.Lasting specific characteristics 3.Temporary general characteristics 4.Temporary specific characteristics 5.Administration and appraisal of test performance sources 6.Chance variance

13. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Types of Reliability Test-retest reliability: A measure of the consistency of an assessment over time. Alternative form reliability: Testing the same group of people on two separate occasions. Interrater reliability: The consistency between scores assigned by different raters when using the same assessment at the same time or when observing the same behavior. Internal consistency: An estimation of the homogeneity of the structure of a test.

14. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Standard Error of Measurement (SEM) The SEM indicates how much variability in the test scores can be attributed to error. Once the SEM has been calculated, a confidence interval can be constructed around the obtained score. How High Should the Reliability Coefficient Be? The purpose of the test should be considered when deciding whether the reliability reported is acceptable. Generally, a reliability co-efficient of.90 is considered to be high,.80 moderate,.70 low, and.60 unacceptable for clinical use.

15. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Validity Definition of Validity Validity is the process of accumulating evidence to support the underlying construct that an evaluation is considered to measure. Evidence of Validity Derived from content Evidence from test structure Evidence from response processes Evidence from relationship to external variables

16. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Social Consequences There is debate as to whether the social consequences of inferences and actions based on a test score are relevant to the validity argument. Constructing the Validity Argument An initial argument to support the use and interpretation of test scores is constructed from the available evidence. However, subsequent studies could provide information that weakens the initial argument.

17. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Fair Testing Practices It is the responsibility of the test user to take measures to minimize bias in testing for particular groups. Occupational therapists must seriously consider whether an assessment is fair for clients and what extraneous variables might influence their scores, because: –The outcome of occupational therapy evaluation has implications for: The lives of our clients The distribution of scarce health care dollars

18. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Culturally Fair Evaluation In many countries, cultural diversity is increasing. Two main issues include the influence of: –A native language other than English on interpretation of and response to test items –Cultural expectations, behaviors, and values on test performance. Critiquing Assessments for Older Adults Personal issues may influence an older adult’s performance on a standardized test.

19. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Issues in the Evaluation of People with Disabilities Individuals with disabilities may have impairments that affect their ability to: –Perceive test instructions or materials –Process information related to formation of a response –Indicate a response through verbal or motor means When the outcome of the evaluation has the potential to influence education, employment, or living situation, the therapist must demonstrate the validity of the inferences made from the score.

20. Copyright © 2009 Wolters Kluwer Health | Lippincott Williams & Wilkins Sources of Information on Published Tests The Mental Measurements Yearbooks provide reviews of most commercially available tests. Test reviews are available online at http://burros.unl.edu/buros/jsp/search.jsp http://burros.unl.edu/buros/jsp/search.jsp Tests in Print provides descriptive information and bibliographies. A source of test information specific to occupational therapy is Measuring Occupational Performance: Supporting Best Practice in Occupational Therapy.