1. Home 1 Computer-Based Test Interpretation James P. Sampson, Jr., M. Page Purgar, and Jonathan D. Shy Florida State University Copyright 2002 by James P. Sampson, Jr., All Rights Reserved

2. Home 2 A Brief History of CBTI Over forty years old Most are computer versions of paper and pencil tests Unrestricted self-help tests do not require practitioner to score and interpret Well designed self-help tests do indicate when a practitioner is needed Restricted practitioner-assisted tests are designed for use in counseling settings Delivered via personal computers and the Internet

3. Home 3 Organization Responsibility for Meeting Testing Standards The Nature of CBTI CBTI in Career Testing Establishing the Validity of CBTI Communicating the Validity of CBTI Need for Research

4. Home 4 Meeting Testing Standards To what and to whom do standards apply? What is a test? What is an assessment? Who is responsible for test development?

5. Home 5 Meeting Testing Standards Test: –Standardized, scored numeric measure –Measures a specific construct or behavior –Meaning is related to scores Assessment: –Inclusive and inferential description of a person –Utilizes tests as well as other sources of data such as observations, interviews, rating scales, and projective techniques –Considered within an ecological context

6. Home 6 Meeting Testing Standards Tests vs. checklists Tests must meet standards Checklists are exempt If it looks like a duck, walks like a duck, then it’s a duck

7. Home 7 Meeting Testing Standards Who is a test developer? Standards apply irrespective of the test developer Tests must meet standards

8. Home 8 Organization Responsibility for Meeting Testing Standards The Nature of CBTI CBTI in Career Testing Establishing the Validity of CBTI Communicating the Validity of CBTI Need for Research

9. Home 9 The Nature of CBTI Definition of CBTI Consumers of CBTI Categorizations of CBTI Appropriate Use of CBTI Sequence for Using CBTI Qualifications for Competent Use of CBTI Potential Benefits of CBTI Potential Problems with CBTI

10. Home 10 Definition of CBTI Uses information technology to integrate theory, practitioner judgment, and empirical evidence One source of data Help practitioners and clients better understand the meaning of test scores For the purpose of gaining insight and making decisions

11. Home 11 Consumers of CBTI In practitioner-assisted assessment, the client and the practitioner are the consumers of CBTI –Some CBTI is intended to be used only by the practitioner (such as the MMPI) In self-assessment, the individual taking the assessment is the CBTI consumer –The CBTI may also be used in counseling

12. Home 12 Categorizations of CBTI Descriptive interpretations generate narrative statements based on test scores Clinician-modeled interpretations generate narrative statements and clinical hypotheses by simulating the judgment of a renowned clinician or a group of expert practitioners Clinical-actuarial interpretations generate narrative statements and clinical hypotheses based on empirical research

13. Home 13 Appropriate Use of CBTI Should be used in a consulting role Should not be used to replace the practitioner Should not be used as a stand-alone psychological report

14. Home 14 Sequence for Using CBTI Review the client’s career concerns Consider client data in light of theory Review the score profile Develop hypotheses the meaning of results based on test data and theory

15. Home 15 Sequence for Using CBTI Review the CBTI to confirm or disconfirm hypotheses Use hypotheses to help the client understand the meaning of the results Use these insights to develop a service delivery plan to help resolve problems Create and review any assessment report

16. Home 16 Qualifications for CBTI Use Completion of a degree or obtaining a credential is not sufficient for ethical use Competencies necessary –Understanding the construct or behavior being measured –Understanding of the test –Understanding of the standard interpretation –Understanding of the CBTI –Supervised experience in using the CBTI

17. Home 17 Potential Benefits of CBTI Capability to process a greater number of complex variables Access to an expanded knowledge base Capable of organizing and systematically accessing extensive and rapidly growing data bases

18. Home 18 Potential Benefits of CBTI Less affected by subjectivity Less subject to interpreter bias More thorough and better documented More rapid access to interpretive reports More rapid updating of interpretations

19. Home 19 Potential Problems of CBTI Developers may have no appropriate qualifications to develop CTBI Many Web sites do not list developer credentials Systematic interpretation bias Perception of infallible accuracy and validity

20. Home 20 Potential Problems of CBTI Overworked, or inadequately trained practitioners may rely too heavily on CBTI Differences among CBTI reports for the same personality test generated by different software programs Some reports contain contradictory statements

21. Home 21 Organization Responsibility for Meeting Testing Standards The Nature of CBTI CBTI in Career Testing Establishing the Validity of CBTI Communicating the Validity of CBTI Need for Research

22. Home 22 CBTI in Career Testing Most CBTI in career testing is is descriptive and actuarial in nature Descriptive and actuarial statements are typically derived from theoretical constructs or research data from the professional manual for the original paper-and-pencil instrument –Justifiable only if the the original measure and the computer version has been shown to be equivalent Few clinician-modeled CBTI are used in generating interpretive reports

23. Home 23 CBTI in Career Testing Current examples: –Descriptive statements based on theory and research are used to interpret scores on the Strong Interest Inventory –Descriptive statements from research on the Career Factors Inventory is used to make recommendations on the need for counseling as part of career exploration –Clinical-actuarial statements based on empirical data as the basis for interpretation of occupational scales on the Strong Interest Inventory

24. Home 24 CBTI in Career Testing Possible applications: –Clinician-modeled statements based on the judgment of expert practitioners on counseling strategies that are potentially appropriate for clients with specific Holland codes or levels of vocational identity on the interpretation of flat profiles on the Strong Interest Inventory

25. Home 25 Organization Responsibility for Meeting Testing Standards The Nature of CBTI CBTI in Career Testing Establishing the Validity of CBTI Communicating the Validity of CBTI Need for Research

26. Home 26 Establishing the Validity of CBTI Test validity concerns judging if theory and empirical evidence support test interpretations for a specific use of a test CBTI validity concerns judging if theory, practitioner judgment, and empirical evidence support test interpretations for a specific use Invalid interpretations are possible for valid tests Validity studies for CBTI are scarce

27. Home 27 Establishing the Validity of CBTI Establishing validity for all types of CBTI –Identify the developer, including his or her qualifications for creating the interpretation –Document the lack of systematic bias in interpretation –Describe the elimination of interpretations that are so general as to be meaningless –Document the efforts to eliminate obviously contradictory statements –Document the evidence that CBTI interpretations compare with credentialed and experienced practitioners

28. Home 28 Establishing the Validity of CBTI Establishing validity for descriptive CBTI –Assuming equivalency exists, document how theory and empirical data from the original paper-and-pencil test are used in the CBTI –Document that the links between test data and interpretive statements in the CBTI have been verified for accuracy

29. Home 29 Establishing the Validity of CBTI Establishing validity for clinician-modeled CBTI –Identify the credentials of any renowned practitioner that were used to generate the CBTI –Identify the credentials of any group of experienced practitioners used to generate the CBTI –Document the methods used to create a statistical model of multiple practitioner judgment for the CBTI

30. Home 30 Establishing the Validity of CBTI Establishing validity for clinical-actuarial CBTI –Document the empirical evidence for the decision rules that were used in developing actuarial predictions in the CBTI –Document how clinician-modeled interpretive statements are used to augment a clinical-actuarial CBTI to avoid leaving a large number of test takers unclassified

31. Home 31 Organization Responsibility for Meeting Testing Standards The Nature of CBTI CBTI in Career Testing Establishing the Validity of CBTI Communicating the Validity of CBTI Need for Research

32. Home 32 Communicating the Validity of CBTI Establishing evidence of validity is meaningless if practitioners cannot access and evaluate this information Print materials traditional method for delivering technical information Internet delivery is a more cost-effective method of delivering information

33. Home 33 Organization Responsibility for Meeting Testing Standards The Nature of CBTI CBTI in Career Testing Establishing the Validity of CBTI Communicating the Validity of CBTI Need for Research

34. Home 34 Need for Research CTBI at a formative stage of development Many advanced applications of CTBI are possible, yet unavailable at present Integration of clinician-modeled and clinical- actuarial CBTI appear promising What types of CTBI is available for what categories of test? How is CTBI being used in practice?

35. Home 35 Conclusion CTBI is a mixture of benefits and problems Despite lingering questions about design and use, CTBI is becoming increasingly popular Balance economic pressure to use CTBI to cut costs with the need to maintain quality CTBI design and use

36. Home For Additional Information www.career.fsu.edu/techcenter/ Thank You