1. CLEAR 2008 Annual Conference Anchorage, Alaska “It’s not my fault!”– Effective Responses to the Disgruntled Examinee Amy Hilson, ASPPB Troy Elliott, ASWB Pat Hartman, NASBA David Capper, ACT

2. CLEAR 2008 Annual Conference Anchorage, Alaska Welcome Objective: Explore how candidate complaints are addressed –different examination programs –different points in the examination process Schedule: –8:30 – 9:30 Presentation –9:30 – 10:00 Questions, Comments, War Stories, etc.

3. CLEAR 2008 Annual Conference Anchorage, Alaska Who we are… Our panel includes –Amy Hilson Associate Executive Officer – Association of State & Provincial Psychology Boards –Troy Elliott Communications Director – Association of Social Work Boards –Pat Hartman Director, CPA Exam Services and Nat’l Candidate DB Admin – National Association of State Boards of Accountancy –David Capper Asst. VP Operations – ACT, Inc.

4. The Question… Can the disgruntled… ever be gruntled?

5. CLEAR 2008 Annual Conference Anchorage, Alaska “It’s not my fault!” Effective responses to the disgruntled examinee Amy Hilson Associate Executive Officer Association of State & Provincial Psychology Boards

6. CLEAR 2008 Annual Conference Anchorage, Alaska Association of State and Provincial Psychology Boards Association of licensing boards/colleges for psychologists in the U.S. and Canada Established 1961 64 member boards: 50 U.S. states, District of Columbia, 3 U.S. territories, 10 Canadian provinces Develops/owns Examination for Professional Practice in Psychology (EPPP)

7. CLEAR 2008 Annual Conference Anchorage, Alaska The Examination for Professional Practice in Psychology (EPPP) Licensure exam used by 50 states, District of Columbia, 3 U.S. territories, and 10 Canadian provinces Originally paper / pencil format - given 2 times per year Shifted to CBT format in April 2001 – available any day and time the test site is open

8. One Candidate’s Story Once upon a time... CLEAR 2008 Annual Conference Anchorage, Alaska

9. One Candidate’s Story CLEAR 2008 Annual Conference Anchorage, Alaska

10. One Candidate’s Story CLEAR 2008 Annual Conference Anchorage, Alaska

11. One Candidate’s Story CLEAR 2008 Annual Conference Anchorage, Alaska

12. One Candidate’s Story CLEAR 2008 Annual Conference Anchorage, Alaska

13. One Candidate’s Story CLEAR 2008 Annual Conference Anchorage, Alaska

14. One Candidate’s Story CLEAR 2008 Annual Conference Anchorage, Alaska

15. One Candidate’s Story CLEAR 2008 Annual Conference Anchorage, Alaska

16. One Candidate’s Story CLEAR 2008 Annual Conference Anchorage, Alaska

17. One Candidate’s Story CLEAR 2008 Annual Conference Anchorage, Alaska

18. One Candidate’s Story CLEAR 2008 Annual Conference Anchorage, Alaska

19. One Candidate’s Story CLEAR 2008 Annual Conference Anchorage, Alaska

20. One Candidate’s Story CLEAR 2008 Annual Conference Anchorage, Alaska

21. One Candidate’s Story CLEAR 2008 Annual Conference Anchorage, Alaska

22. CLEAR 2008 Annual Conference Anchorage, Alaska Quotes from One Complainant 1 March 23 “According to your results I did not pass. Moreover I did not pass by a significant margin. In fact, the margin is so significant, I seriously question the results. I realize that you are not aware of my credentials; however there is no way that the score reported to me is accurate.”

23. CLEAR 2008 Annual Conference Anchorage, Alaska Quotes from One Complainant 2 March 23 - continued “According to the results sent to me, I received a standard score of 427 with a passing score of 500, which roughly translates to 119 correct responses and 106 incorrect responses. This is simply not possible... Someone, somewhere has made an error.”

24. CLEAR 2008 Annual Conference Anchorage, Alaska Quotes from One Complainant 3 March 23 – continued “On the last five online tests with ( test prep company ) my average score was 78% -- far above the 59% you reported to me. “I would greatly appreciate an investigation into my examination results.”

25. CLEAR 2008 Annual Conference Anchorage, Alaska Quotes from One Complainant 4 April 13 – 12:18 p.m. “My EPPP examination results as reported were slightly higher than random responding – an unacceptable result. “It is important to note that I am the consumer and I have paid a significant amount of money for the EPPP examination. As such, I believe that I have certain consumer rights that are protected by Federal law.”

26. CLEAR 2008 Annual Conference Anchorage, Alaska Quotes from One Complainant 5 April 13 1:28 p.m. “I have two questions: 1.Can I take the EPPP with paper and pencil, that is, not using the computer? 2.Can I take the EPPP examination without the 25 pretest questions? I do not want to be a participant in this research.”

27. CLEAR 2008 Annual Conference Anchorage, Alaska Quotes from One Complainant 6 May 7 “I found your unwillingness to investigate my concerns with your “contractor” to be inadequate. Forwarding my concerns without my permission violated trust, engendered disdain for the process of licensure, and demonstrated ASPPB’s willingness to abdicate responsibility.”

28. CLEAR 2008 Annual Conference Anchorage, Alaska Quotes from One Complainant 7 May 7 – continued “Ms. Hilson, since you appear too eager to brush away my concerns, I request that you forward my concerns, in their entirety, to your immediate supervisor for his or her review and ultimate response. I have asked many questions and I have not been provided with any genuine answers – merely opinions disguised as facts.”

29. CLEAR 2008 Annual Conference Anchorage, Alaska Quotes from One Complainant 8 May 12 “After waiting for my feedback on my EPPP performance, I finally received a letter. Unfortunately the information contained is useless trivial crap. “I have already contacted the State District Attorney and I am filing a complaint. In addition, I’ve forwarded all emails and previous correspondence to my congressman.”

30. CLEAR 2008 Annual Conference Anchorage, Alaska Things We Never Knew About Our Exam – Lessons from Candidates 1.There’s a hard version called The Beast. It’s unfair if you get The Beast. 2.The passing rate is kept artificially low to control access to the profession. 3.The exam is filled with trivia questions.

31. Things We Never Knew About Our Exam – Lessons from Candidates 4.There are trick questions on the exam. 5.It’s unfair that test questions and practice exams are different from the prep course materials. CLEAR 2008 Annual Conference Anchorage, Alaska

32. CLEAR 2008 Annual Conference Anchorage, Alaska Top Ten Test Site Complaints 1.Testing room was too hot 2.Testing room was too cold 3.Keyboard noise from people taking essay exams was distracting 4.Screen flicker from CRT monitors gave me a headache

33. Top Ten Test Site Complaints 5.Conversations of staff could be heard 6.Staff kept coming in and out to check on us and that was distracting 7.There was noise outside of the test center that could be heard CLEAR 2008 Annual Conference Anchorage, Alaska

34. Top Ten Test Site Complaints 8.Biometric check-in didn’t work so start of my exam was delayed 9.Don’t like the erasable white boards – would prefer paper and pencil 10.Bodily noises or vocalizations from other test takers was distracting (!?!) CLEAR 2008 Annual Conference Anchorage, Alaska

35. CLEAR 2008 Annual Conference Anchorage, Alaska Why care what candidates think? 1.ASPPB is Trusted Partner throughout psychologist’s career: –Selection of doctoral program –Credentials banking and verification –Handbook of licensure requirements –EPPP preparatory information & practice exams –EPPP exam and score bank

36. CLEAR 2008 Annual Conference Anchorage, Alaska Why care what candidates think? 2.Prevent the idea that ‘licensure boards are out to get me’ 3.Ignored angry candidates produce more angry candidates 4.It’s the right thing to do

37. CLEAR 2008 Annual Conference Anchorage, Alaska What Does ASPPB Do to Help? Item pretesting has virtually eradicated complaints about exam content. Candidate education is the key to avoiding unfounded complaints.

38. CLEAR 2008 Annual Conference Anchorage, Alaska What Does ASPPB Do to Help? Information for Candidates brochure –contains info on all aspects of exam –printed copies are provided to boards for every exam applicant at no charge –available online for free download Practice exam at CBT site Practice exam online

39. What Does ASPPB Do to Help? Myths of the EPPP online and in brochure format Activities at American Psychological Association Convention –Exhibitor booth – many students come for information –Participate in a session for students: What Students Need to Know about Licensure CLEAR 2008 Annual Conference Anchorage, Alaska

40. What Does ASPPB Do to Help? Working on closer relationship with doctoral training and internship programs Items from Previous Exams Chart of passing score requirements by jurisdiction and license type available online CLEAR 2008 Annual Conference Anchorage, Alaska

41. CLEAR 2008 Annual Conference Anchorage, Alaska What Does ASPPB Do to Help? Sorry... No recommendations for dealing with bodily noises from other test takers!

42. CLEAR 2008 Annual Conference Anchorage, Alaska Speaker Contact Information Amy Hilson, Associate Executive Officer Association of State & Provincial Psychology Boards ahilson@asppb.org 334-832-4580 P.O. Box 241245 Montgomery Alabama 36124-1245 www.asppb.org

43. CLEAR 2008 Annual Conference Anchorage, Alaska “It’s not my fault!” Effective responses to the disgruntled examinee Troy Elliott, Communications Director Association of Social Work Boards

44. CLEAR 2008 Annual Conference Anchorage, Alaska The basics ASWB – social work regulatory boards 61 members – 49 states, 10 provinces, DC, US Virgin Islands Five categories of exams given – about 29,000 annually CBT, 170-item, 4 option m.c., administered through ACT

45. The presence of cameras, proctors, and two-way mirrors made me paranoid. I couldn’t focus on the test. The room was too hot/cold. The room was too noisy. Computer problems affected my performance. I was flustered by difficulties I experienced registering/finding the test site/signing in, etc. The behavior of test center personnel affected my performance. CLEAR 2008 Annual Conference Anchorage, Alaska Your hit parade of disgruntlement Group 1 – The Evaluators

46. Your hit parade of disgruntlement Group 2 – The Doubters The test questions did not resemble the questions provided to me by the private test-prep company. The test is biased against a particular group, (or me personally). The test does not reflect the profession. Almost every question had more than one right answer. The test was rooted in a particular theoretical perspective. CLEAR 2008 Annual Conference Anchorage, Alaska

47. CLEAR 2008 Annual Conference Anchorage, Alaska Your hit parade of disgruntlement Group 3 – The Dreamers While taking the test I slipped off my shoes, and lost one down an air vent that had no grill. The distraction prevented me from passing. I was unable to perform to my fullest potential because I was upset after hearing about the untimely death of [a particular celebrity]. My new glasses prescription was filled out incorrectly. My lenses were reversed. I will fax you a letter from the eyeglass hut. I was close to passing, I have never failed a test. My college gpa and/or professional experience should be factored in as bonus points on my score. The proctor’s shoes/ test room door/ mice in the walls squeaked too much.

48. The question we’re dying to ask Our exams are psychometrically sound, fairly administered, carefully scored… AND REQUIRED— so why do we have to bother with this stuff? CLEAR 2008 Annual Conference Anchorage, Alaska

49. Protecting validity includes paying attention to perceptions. Unresponsive entities become devalued. Devalued tests become a hurdle to be gotten over – or around. We do a good job. People should know. Disgruntled candidates may have some valid points. CLEAR 2008 Annual Conference Anchorage, Alaska The answers you already know

50. What it’s all about: fairness. And justice. And fair justice in fairness. And just fairness. "All of us in America want there to be fairness when it comes to justice.“ George W. Bush Two tasks, influenced by each other: Anticipating problems Reacting to concerns At three points along the way: Registration and appointments Test-day events Content and scoring CLEAR 2008 Annual Conference Anchorage, Alaska

51. At Registration Anticipating problems Candidate H’books Live people Confirmations Follow-up materials Documentation Everything bears repeating CLEAR 2008 Annual Conference Anchorage, Alaska Reacting to concerns Clear processes Limited outcomes Everything bears repeating

52. On test day Anticipating problems Candidate H’books Live people “Remedies” Acknowledgment Post-test (pre score) survey Everything bears repeating CLEAR 2008 Annual Conference Anchorage, Alaska Reacting to concerns Clear processes Limited outcomes Everything bears repeating

53. On test day – a signed agreement CLEAR 2008 Annual Conference Anchorage, Alaska Basically: The rules of the road What happens when things go wrong When a candidate can push the “eject” button

54. Is all this scrutiny necessary? Yes. For all the obvious reasons (test security, liability, etc.) Also, because rigidity can allow us to be fair. And hey, apparently George Washington did it. CLEAR 2008 Annual Conference Anchorage, Alaska "President Washington, President Lincoln, President Wilson, President Roosevelt have all authorized electronic surveillance on a far broader scale." Attorney General Alberto Gonzalez

55. Content and Scoring This is where things get more challenging for us. Balancing act – security vs. “transparency”; layman’s terms vs. psychometric precision; education vs. test-prep CLEAR 2008 Annual Conference Anchorage, Alaska

56. Content and Scoring How much information is too much? What really gets read? What’s really useful, what only seems useful, and who are we to decide? Equating, pretesting, scaled scores, raw scores, IRT, anchor exams, subcategory performance…. CLEAR 2008 Annual Conference Anchorage, Alaska As we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know.“ Donald Rumsfeld

57. Content and Scoring Mostly, not as complicated as all that What we used to do What we do now, or How We Learned to Stop Worrying and Love Faceless Bureaucracies (a teeny bit) CLEAR 2008 Annual Conference Anchorage, Alaska

58. Content and Scoring What we don’t do and why we don’t do it ASWB’s culture; culture of testing population Exam review, formalized item challenges, subscores…but for how long? New offerings and scoring systems. Are we helping? What about you? CLEAR 2008 Annual Conference Anchorage, Alaska

59. Wrapup "My belief is we will, in fact, be greeted as liberators.“ Dick Cheney Bottom line: Don’t be surprised by gaps between expectation and reality. Be ready to learn and adapt. CLEAR 2008 Annual Conference Anchorage, Alaska

60. Wrapup Sometimes, complaints make things better. What we’ve improved on What about you? CLEAR 2008 Annual Conference Anchorage, Alaska

61. Final thoughts The disgruntled candidate can point out real problems. Problems can be opportunities. Opportunities can provide inspiration. “You can't wait for inspiration. You have to go after it with a club.” Jack London CLEAR 2008 Annual Conference Anchorage, Alaska

62. CLEAR 2008 Annual Conference Anchorage, Alaska Speaker Contact Information For more information: Troy Elliott telliott@aswb.org

63. CLEAR 2008 Annual Conference Anchorage, Alaska “It’s not my fault!” Effective responses to the disgruntled examinee Patricia Hartman, Director of Examination Operations, National Association of State Boards of Accountancy

64. CLEAR 2008 Annual Conference Anchorage, Alaska The Uniform CPA Exam The Uniform CPA Examination is provided on behalf of the Boards of Accountancy who have the ultimate licensing authority. The Examination is offered jointly by three organizations: NASBA (The National Association of State Boards of Accountancy), the AICPA (American Institute of Certified Public Accountants), and Prometric. The AICPA is responsible for developing and scoring the Examination, NASBA for the National Candidate Database, and Prometric,, for examination delivery at authorized test centers.

65. The Uniform CPA Exam The Uniform CPA Examination consists of four sections Auditing and Attestation (AUD) Business Environment and Concepts (BEC) Financial Accounting and Reporting (FAR) Regulation (REG)

66. The Uniform CPA Exam The examination is composed of testlets (groups of 24 or 30 multiple-choice questions) and simulations (condensed case studies). Three testlets and two simulations comprise each of these sections: Auditing and Attestation (AUD), Financial Accounting and Reporting (FAR), and Regulation (REG). Business Environment and Concepts (BEC) consists of three multiple-choice testlets only.

67. Candidate Resources Candidate Information Bulletin News and Email Blasts Frequently Asked Questions Sample Tests and Tutorial NASBA Candidate Care Unit Post Exam Survey Score Review & Appeal CLEAR 2008 Annual Conference Anchorage, Alaska

68. Candidate Resources

77. NASBA’s Candidate Care Acts as candidate advocate Responds to all concerns Tracks and reports all concerns Reviews all Center Problem Reports Reviews all exit survey comments

78. Score Review Score review is a service to candidates offering additional assurance that operational quality controls in the scoring process were complete and the scores are accurate. The review is a verification of a candidate's Uniform CPA Examination score. It involves making certain that the approved answer key was used and that it was applied correctly.

79. Score Review It is an automated and validated process, and does not involve human graders. It is NOT an opportunity to “find additional points” or to have new responses considered.

80. Score Review All reported scores are subject to routine quality controls and are, in fact, scored twice before they are reported to state boards. As a result, the likelihood of a score change following score review is exceedingly small, or less than 1 percent of all requested score reviews since the inception of the computer based test.

81. Appeal Process The appeal option enables candidates to view the test questions or simulation problems that they answered incorrectly together with their responses, and to submit comments online. The confidentiality of the examination requires that such viewing sessions take place only in authorized locations, under highly secure conditions, and in the presence of a representative of the candidate's board of accountancy or its designee. The appeal process is not available in all jurisdictions.

82. Appeal Process Candidates will be allowed to review incorrect responses only. Candidates should requesting an appeal only if they believe that there is a question or simulation problem that they would like to challenge. Upon review of the questions or simulation problems that were answered incorrectly, candidates may decide to challenge the validity of one or more items.

83. Appeal Process Candidates must be prepared to present a cogent, vigorous, and compelling defense of incorrect responses. The appeal is not an opportunity to submit new responses. The appeal is an opportunity to challenge question(s) or simulation problem(s) and defend the response(s) provided at the examination.

84. Appeal Process The board, or its designee, will schedule the candidate’s appeal session. Online comments during this session are transmitted to the AICPA through NASBA. After the session, the AICPA reviews responses, considers the online comments submitted, verify the score and forwards results to NASBA. Candidates do not receive detailed information about the question(s) challenged because of the need to preserve the confidentiality of examination content.

85. The End Result Approximately 200,000 examinations administered yearly. Approximately 2,400 concerns reported to Candidate Care annually. Approximately 275 retests awarded annually. Volume continues to increase, concern continue to decline.

86. CLEAR 2008 Annual Conference Anchorage, Alaska Speaker Contact Information For more information: Patricia Hartman phartman@nasba.org

87. Test Vendor Perspective Test Sponsor/Vendor Alignment Crucial!! –Reasonable, calibrated expectations Principle-based Neither too lenient, nor too stringent –Clearly defined roles –Standard routing/resolution path/plan –Communicate, communicate, communicate CLEAR 2008 Annual Conference Anchorage, Alaska

88. CLEAR 2008 Annual Conference Anchorage, Alaska Final Thoughts… Complaints are gold; complainers your ally –Think “continuous improvement” Opportunity to work toward perfection –Think of future complaints being eliminated –If nothing else… Think of all the great stories you can tell

89. CLEAR 2008 Annual Conference Anchorage, Alaska Thank You! Additional questions and comments: –Amy Hilson, ahilson@asppb.orgahilson@asppb.org –Troy Elliott, telliott@aswb.orgtelliott@aswb.org –Pat Hartman, phartman@nasba.orgphartman@nasba.org –David Capper, david.capper@act.orgdavid.capper@act.org