1. Challenges of Piloting Test Items
Branka Petek
School of Foreign Languages
Slovenia

2. Content Challenges Slovenia had to face when piloting test items
What we learned from experience

3. Why pilot test items? To get a clear picture about candidates’
language skills.
To get a clear picture we need good test items.
Impossible to have good test items without pre-
testing.

4. Challenges SFL had to face
Appropriate population for piloting
Administration of the items
Test format
Statistical analyses

5. Population for piloting
Size
Similarity to the Slovenian testing population
Level of proficiency
Test fatigue

6. Lessons learned
SIZE: the population should be as big as posible, (but) anything is better than nothing;
SIMILARITY: the population should be similar to the testing population;
LEVEL OF PROFICIENCY: normal (or near normal) distribution otherwise the results will be unreliable.
TEST FATIGUE: Have the canidates piloted before? Are they tired of taking the tests, piloting?

7. Administration Administrators Time Courses
Collecting data on testakers

8. Lessons learned
ADMINISTRATORS: the most reliable results when we administer the tests;
TIME: depends on a course cycle;
COURSES: courses designed to prepare students for STANAG tests normally give the most reliable results;
QUESTIONNAIRES: help investigate face validity of tests, time allocated, clarity of rubrics, appropriacy of test methods, text topics (if well designed).

9. Test format Length Number of items Task types
Topics (cultural background, influence of the course)

10. Lessons learned LENGTH: Similar to the live test version;
NUMBER OF ITEMS: approximately the same number of items;
TASK TYPES: different countries use different methods – candidates might not be familiar with the task types we use;
FAMILIARITY WITH THE TOPICS: e.g. military topics (cultural background);

11. Statistical analyses CTT IRT ‘Manual check’
The influence of a particular population

12. Lessons learned Small population, CTT – the only option;
Sometimes less than 30 - manual checking:
odd answers and strange behaviour,
can help eliminate some problems and improve the items;
With small population the data is less reliable - always an element of risk.

13. Perfect & real-world of piloting
A perfect world piloting session would mean at least 300 test takers, IRT, revising test items, repilot, IRT, final version of the test and experts to determine cut-off scores.
In real world piloting is difficult to plan and carry out.
Absolutely essential part of a testing cycle.
Piloting internationally can produce more reliable results but also represents many pitfalls we have to be aware of.
Being aware of possible problems might help us plan.
The more we invest (in the sense of time, effort and money), the more we get.

14. Thank you
Questions,
suggestions?