CLEAR 2003 Annual Conference Toronto, Ontario Technology, Service, and Delivery Stephen Greenfield Product Services Manager

CLEAR 2003 Annual Conference Toronto, Ontario Using the Internet to Enhance Your Testing Program

Internet-based tools for licensure programs Scheduling Communication Data management Reporting

Scheduling

Reporting Access for sponsors to data about the program –Registration and scheduling –Testing locations –Testing volumes –Exam results

Data Management Enhanced options for program management –Information on candidates –Custom reports –Incident tracking –Managing candidate eligibility, special accommodations etc.

Internet provides tools for Scheduling Communication Reporting Program management

CLEAR 2003 Annual Conference Toronto, Ontario Innovative Uses of Technology in Testing

“... advances in Internet security...” Network-layer security (SSL, firewall, etc.) Message-level security (encryption, etc.) Application-level security (logins, etc.).

Test security Technological protection of test items is just one part of test security –Protection of materials during test development –Confirmed identity of test takers –Confirmed identity of testing staff –Proper procedure by proctors / invigilators –Prevention of sharing of test items

Technology support for test security –Controlled access to test materials –Testing center procedures –Testing center facilities –Biometric ID of candidates & staff

Test security Computer-based testing allows security enhancements –Alternate forms, randomized or adaptive presentation, to reduce item exposure –Analysis of test-taker performance (responses, latency) to detect possible cheating –Analysis of result patterns –Monitoring of proctor actions to detect errors and possible fraud

Innovative item types Benefits –Realistic stimuli (images, audio, motion) –Recall rather than recognition –More realistic tasks Performance-based Simulation –Face validity

Innovative item types Issues include –Time and cost to create items –Objective scoring –Statistical tools for analysis –File sizes

Automated scoring of narrative answers / essays Commercially-available programs show high correlations with human raters Methodologies include –Syntactic and rhetoric analysis –Semantic and content analysis –“Training” with scored sample essays

Automated scoring of narrative answers / essays Vast majority of essays still go to human raters Internet-based tools to support human raters –Rubrics –Analyze scoring patterns –Exchange cases

Internet-based technology and certification programs Scheduling Delivery of tests Collecting and reporting results Managing the program Security benefits as well as challenges Advanced item types