2. Experts Respond to Your Questions 2006 Annual ConferenceAlexandria, Virginia Council on Licensure, Enforcement and Regulation Expect the Unexpected: Are We Clearly Prepared? Are You Ready?

3. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Moderator/Coordinator Chuck Friedman Assistant Vice President ACT

4. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Panelist David Foster Chief Executive Officer Kryterion, Inc.

5. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Panelist Ron Rodgers President Employment Research Institute Director of Measurement Services Continental Testing Services

6. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Panelist Barbara Showers Director, Education and Examinations Wisconsin Department of Regulation and Licensing

7. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Speakers You... the audience!!!

8. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Developing and Implementing ADA Accommodations and Preparing for ADA Litigation Credentialing & Exam Issues Sessions Early Bird Sessions Thursday, September 14 Psychometrics of Testing with Small Numbers Friday, September 15

9. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Demystifying Testing: Guide to Managing Exam Programs Leveraging the Internet in Exam Development and Delivery When the Unexpected Happens: Logistical and Technical Challenges Credentialing & Exam Issues Sessions

10. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Physical Ability Testing and Practical Examinations: They Fought the Law and the Law Won Reporting Examination Results to Stakeholders: Issues and Best Practices Test Compromise and Security: Be Prepared Credentialing & Exam Issues Sessions

11. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Responding to YOUR Testing Questions

12. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Test Blueprint Job Analysis Reliability Scoring Validity Passing Score Item Writing Standard Administration Item Analysis Item Review

13. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Test Blueprint Job Analysis Reliability Scoring Validity Passing Score Item Writing Standard Administration Item Analysis Item Review

14. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia How often should my program conduct a job analysis? Job analysis

15. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Job analysis required to support test Link content outline to test references Survey representative sample of profession Support test specifications with job analysis results Document each question with test references identified and available to all candidates

16. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Job analysis should be updated when: References supporting test have changed (new editions, out of print, etc.) Emerging technology changes practice Changes in practice redefine critical knowledge & skills needed to protect public & practitioners

17. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Should my Board or program sponsor allow candidates to review test results including test questions?

18. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Considerations: Credibility/defensibility of test process Content reviews/expert input Verification measures for scoring History of scoring errors

19. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Considerations: Test exposure/security issues Extended study time with test content Limit to questions wrong? Provide correct answers?

20. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Considerations: Current law or legislative climate Law on books? Transparency of government Fairness Remedies – is retake enough?

21. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia How do we increase security for both paper-and-pencil test delivery and computerized test delivery in the face of new technologies for cheating and test theft?

22. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Be informed; know the threats and risks

23. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Question Do you know what this is?

24. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Question Do you know what this is? Two- Way Radio

25. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Question What is the threat here?

26. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Question What is the threat here? Web Camera to capture items

27. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Question What is the threat here? Display to show answers

28. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Be informed; know the threats and risks Detecting and monitoring the threats will reduce the risks: Examples include Better training of site and program staff Data forensics Watch the Web Protect your items, tests and test results with better access controls, authentication and identification, encryption, etc. Take decisive action when a breach occurs

29. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia What is the relationship between the examination process and reciprocity between states? Between countries?

30. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Between states: Uniform national exams aid reciprocity State-unique exams can create barriers Authority to regulate professions is given to state government, but reciprocity is gaining importance

31. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Between countries: Global delivery of exams is increasingly possible Evaluations of foreign credentials have been aided by supplemental exams – ECFMG, ECFVG CGFNS – will they remain necessary?

32. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia When it is appropriate to translate or adopt and exam? See you soon

33. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Public Health & Safety Issues Are safety sources (MSDS, etc.) and product labels available to practitioners in the target language? Are comparable training materials and textbooks available in English & in target language? How are target language candidates trained? In what language? With comparable health/safety resources?

34. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia If translation/adaptation is appropriate: Are practice standards compromised in any way by training as delivered in target language? Follow standards & guidelines for adapting test into new language Review CLEAR Resource Brief on translating or adapting tests

35. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia What new technologies can we expect in the next five years that will affect the testing process?

36. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia New security technologies will enable us to prevent fraud and catch test thieves, cheaters and proxy test takers in the act The Internet combined with test administration technologies will make possible truly ubiquitous high-stakes testing where people work, learn and live Better measurement with innovations at the item and test levels (performance testing using simulations, for example) Faster, virtually immediate, test revision and worldwide distribution

37. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia How should my program deal with an error: for example an incorrect score report, setting of an incorrect passing score, incorrect scoring on an item?

38. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Guiding Principles Accept responsibility for the problem and the resolution Apologize promptly and sincerely Remediate with candidate and with internal or external systems

39. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Steps to Take First, investigate who is impacted and how it happened Most important in licensing is pass/fail status and licensing errors Identify remedies – Rescore? Free retest? Other? Request back the license?

40. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Thank you and Enjoy Lunch!!!!!!!!!