Leveraging the Internet in Exam Development and Delivery Tony Zara, PhD Ho J. Mun Pearson VUE / Promissor Expect the Unexpected: Are We Clearly Prepared? Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Agenda Introductions Exam Development –Requirements for virtual meetings –Guidelines for success –Disadvantages/advantages of virtual meetings –Regulatory client experiences Examination Delivery Credentialing Process Discussion –The next big thing Conclusions

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Exam Development How to coordinate geographically diverse panels committees, and/or employees? –High cost of travel –Time involved in travel –Multiple schedules to coordinate –Jet lag Sounds like an opportunity for virtual meetings

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Requirements for Virtual Meetings Need to provide an experience more rich than phone/ Evaluate each meeting type for appropriateness (e.g., training, item review, standard setting, job analysis, item writing etc.) Must evaluate capabilities of different software/systems Bandwidth Sharing applications Messaging (public/private) Plug-ins Security Asynchronous viewing Firewall issues Host/guest connectivity

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Running Successful Virtual Meetings Allow extra time Decide in advance on the format for each part of the presentation (self-directed vs. host-controlled) Participants login ahead of time Have a wingman – multiple hosts allow you to deal with participants individually Distribute necessary print materials (e.g., content outlines, checklists) prior to meeting Review the different software options Ensure support is readily available If this is your first time, conduct a dry run internally

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Advantages of Virtual Meetings Lower costs Faster turnaround Shorter, more focused meetings Access to wider range of experts (allows for geographic diversity) May allow small clients to perform work that they otherwise couldn’t afford May allow side conversations (chat/IM) that are less disruptive to the whole group Easier to include large groups for training Supplement – not replacement for – face to face (FTF) meetings; also allows for FTF meetings to be more productive

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Disadvantages of Virtual Meetings Technology issues Lack of face time No body language/ nonverbal cues Participants “zoning-out”/multi-tasking Travel perk for participants

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Client Experiences Benefiting Industries –Insurance departments –Real Estate commissions –Appraisers –Nurse Aide boards –Contractor boards –Certification clients

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Examination Delivery Types of computer-based testing –Web-based testing Browser-based (connected) Internet-based (disconnected) –Center-based testing Third party sites Private sites Owned/operated sites –Independent testing

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Web-based Testing Browser-based –Live, connected testing across the Internet –Either center-based or independent –No standardization of testing stations –Internet performance issues Internet-based –Internet used to distribute testing information –Center-based, disconnected –Client-server testing center model –Standardization of testing stations possible

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Center-based Testing Third party sites –Can deliver web-based or internet-based testing –Standardized procedures –Proctoring Private sites –Corporate environments Owned/operated sites –Most standardized environment –Optimized for testing –Professional motivated staffing

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Independent Testing Generally browser-based Broad distribution of testing Standardization issues Security challenges –Unproctored delivery –Examinee identification –Testing intellectual property Inexpensive assessment events

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Appropriate Applications Independent testing –Connected, browser-based –Low-stakes assessments –Diagnostic assessments –Practice testing Center-based testing –Disconnected, client server –High-stakes examinations (licensure, certification, academic) –High-complexity examinations Use the tool appropriate for the assessment situation

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Credentialing Process Enhanced communications Eligibility process Data transfer Results processing State systems

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Discussion The Next Big Thing?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Conclusions As with all technological advances, the internet is here to stay in testing The Internet is a useful tool for many contexts Just because you have a hammer, not all problems are nails

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Contact Information Anthony R. Zara, Ph.D. Pearson VUE Ho J. Mun Promissor