ECDL Test Development and Validation Process Paul Davis, Oxford University Computing Services Garry Cleere The European Computer Driving Licence Foundation Ltd.

2CAA Conference Test development process

3CAA Conference Syllabus Definition (Expert Opinion)

4CAA Conference Soft Test Characterisation Non-threatening test experience Predominantly performance based Straight forward language No trick questions Coherent outputs Sustains the metaphor of the driving licence Ample time

5CAA Conference Forum score

6CAA Conference Test characteristics – real life

7CAA Conference Forum duration of tests

8CAA Conference Mean duration

9CAA Conference Criticality (Core Item Identification) Exercises

10CAA Conference Model Tests Exercise SMEs create 4 streams of tests Spread core items across streams Provide a reasonable coverage in each test Provide syllabus coverage across streams Average number of items

11CAA Conference Model Tests Grids Scheme REF.TASK ITEMMT 1MT 2MT 3MT Open (and close) a word processing application. NE Open one, several documents Create a new document (based on default, other available template). 1111

12CAA Conference Item Count 36 Items Item Count dictated by coverage requirement Dictated by congruency and criticality in tests 100% Syllabus coverage throughout ECDL tests Balanced, discriminating test profiles for Syllabus V.4 Tests Syllabus Version 4.0 rigorously High reliability in tests and between tests Item Count standardisation in all Modules

13CAA Conference Result against completion time

14CAA Conference Cut Score / Pass Mark Systematic approach and Industry based cut score technique (modified Angoff Method) Underpinned by expert opinion Cut Score exercises carried out for each module Borderline Candidate, or Minimally Competent Candidate (MCC) Meaningful Cut Scores based on profile of ECDL tests derived from ECDL V4 Syllabus Item Count standardisation in all Modules An auditable, standard cut score process Common cut score across all ECDL core level modules, (M1 - 7)

15CAA Conference Cut Score Exercises

16CAA Conference Model Tests Collation and Balancing

17CAA Conference Characterisation Test Template Version 4.0

18CAA Conference Item / Test Development Dependant upon: Syllabus Soft Test Characterisation CTT ECDL-F Evaluations Outcome Probability Grids (expected pass rate from question)

19CAA Conference Item / Test Development

20CAA Conference Expert Review of Items / Tests Syllabus Congruence Syllabus Coverage Compliance with CTT Appropriateness of Item format Technical Accuracy Language & Clarity Text Content

21CAA Conference Expert Review : ATES Systems Test Presentation / Interface Features Timing & Information Appropriateness of ATES Test Content Test Content Marking Algorithm Candidate Information and Guidance Practice Tests Documentation and Training Materials ATES Technical Support Reporting Test Generation Security Features

22CAA Conference Perceptions of tests

23CAA Conference Choice of test engine (V3) Simulation In- Application Rich Simulation Web-based MCQ/HS MQTB First Second87555 Third Fourth Fifth110915

24CAA Conference Expert Review : Test Design / ATES Systems ATES System Overview ATES System Hardware Requirements ATES Operating System Requirements Other Software Requirements Minimum Screen Resolution Assistive Technologies Screen Design and Visual Impairment

25CAA Conference Assembly and Administration of Pilot Tests Tests are made available for Pilot Test activity Qualitative analysis Test taker and administrator observation sheets / questionnaires Test data requirements Remember: Across 4 streams for 7 tests across 100 countries and many languages.

26CAA Conference Pilot Tests Review Pilot Test results are reviewed by the ATES Evaluations Group. Any changes requested in the tests based on Pilot testing are then undertaken before the live certification tests are administered. To ensure statistical validity some pre-testing on the fly is used until results don’t change. Maintenance then takes over.

27CAA Conference Empirical Evidence Evaluation ECDL Tests are evaluated and monitored on an ongoing basis by the ATES Evaluations Group using recognized psychometric measurements. CTT Version Test behaviour statement

28CAA Conference Ongoing Maintenance of Tests ECDL tests are monitored on an ongoing basis by the ECDL Foundation ATES Evaluations Group. Tests are updated in line with Syllabus revisions and also where test results or empirical evidence indicates that items or tests are not functioning in line with test behaviour parameters.

29CAA Conference Process Steps and the Validity Argument Documented process - expert opinion Detailed test data reporting requirements Data fed back to the ECDL Foundation Industry Standard techniques Process Loop Valid ECDL Tests

30CAA Conference Contacts - Garry Cleere Programmes Manager ECDL Foundation 107 The Windmill Sir John Rogersons’ Quay Dublin 2 Ireland Tel Tel Dr Paul Davis Deputy Head, LTG, OUCS 13 Banbury Road Oxford OX2 6NN UK Tel: