1. Gilles Guillard, Markus Nussbaum
Implementing a New Multilingual Selection Model Across 27 European Countries
Gilles Guillard, Markus Nussbaum

2. The European Union Commission Economic & Social Committee Council
Parliament
Court of Justice
Court of Auditors
Commission
Economic & Social Committee
Committee of
the Regions
Ombudsman
Data Protection Supervisor

3. The EU Institutions Posts in 2012 European Parliament 6655
Council of the European Union
3153
European Commission
25478
Court of Justice
1952
European Court of Auditors
887
European Economic and Social Committee
724
Committee of the Regions
531
European Ombudsman 66
European Data Protection Supervisor 43
European External Action Service
1670
Total
41159

4. Entry Streams
Administrators (AD) e.g. policy officers, lawyers, auditors, scientific officers, translators, interpreters, communication & press officers …
Qualification: University degree (bachelor)
Assistants (AST) e.g. secretaries, HR assistants, financial assistants, conference technicians …
Qualification: Secondary education (at least) and relevant professional experience

5. linguistic skills test -
Selection in Cycles
AST
Registration +
Self-assessment
CBT
Assessment
Recruitment
Self-assessment
Registration +
CBT
Assessment Centre
Recruitment AD
December
March
June
September
Registration +
Self-assessment
CBT / professional
linguistic skills test -
CV sift
Recruitment
Assessment Centre
ADL

6. The AD Cycle 2012 Competition launched in March 2012
candidates for 296 posts
5 fields: Audit, EU Public Administration, Communication, Law, External Relations

7. Computer Based Tests
Verbal reasoning 20 items from a database, 35 min, 23 languages
Numerical reasoning 10 Items from a database, 20 min, 23 languages
Abstract reasoning 10 Items in fixed test forms, 10 min
Situational Judgement 20 Items in fixed test forms, 30 min, EN/DE/FR

8. Verbal Reasoning

9. Numerical Reasoning

10. Abstract Reasoning

11. Situational JudgementX X

12. Results EU Public Admin Audit Communication Law External Rel Sum AD5
validated application
12287
7892
2675
1389
3467
2023
4636
2130
6519
2338
45356
booked CBT
9557
6076
2182
1005
2525
1430
3615
1592
5016
1706
34704
participated CBT
8323
5336
1942
893
2132
1263
3113
1330
4412
1470
30214
reached CBT pass-marks
5502
3767
1427
603
1253
723
2076
830
2935
948
20064
succeeded CBT
190
102
108 34 77 51
144 43 64 21
834

13. Score Distribution - VR

14. Score Distribution - NR

15. Score Distribution - AR

16. Situational Judgement

17. Intercorrelations Verbal Reasoning Numerical Reasoning
Abstract Reasoning
SJT Analysis & Problem Solving
SJT Planning & Organising
SJT Quality & Results
SJT Resilience
SJT Working with others -
.41
.29
.16
.23
.15
.17
.18
.32
.20
.14
.12
.40
.33
.31
.35

18. How EPSO monitors the behaviour of items
Item Analysis
How EPSO monitors the behaviour of items

19. Item Analysis Based on the dicotomous / polytomous Rasch model
Conducted after every major competition
Calibrate items and monitor their behaviour
Making psychometrics as vivid as possible
Georg Rasch ( )

28. Example: Cumulative Analysis VR+NR
Test
Responses
Candidates
Items VR
5 071 NR
3 490
Total
8 561
Oct 2012

29. Item Difficulty Adjustment Across Languages (IDAAL)
How EPSO ensures fair opportunities for all candidates although testing in 23 different languages

30. Aligned Averages Algorithm (AAA)1 2 3 4 -4 -3 -2 -1 5 1 2 3 4 -4 -3 -2 -1 5 EN A EN A -4 -3 -2 -1 +1 +2 +3 +4

31. Aligned Averages Algorithm (AAA)
citem = common item in en and lang

32. Minimum Meansquare Method (MMM)1 2 3 4 -4 -3 -2 -1 5 1 2 3 4 -4 -3 -2 -1 5 EN C A B EN A B C

33. Minimum Meansquare Method (MMM)

34. Person Parameter Contrast Measure (PPCM)1 2 3 4 -4 -3 -2 -1 5
Cdts EN
Cdts A

35. Person Parameter Contrast Measure (PPCM)

36. Item Difficulty Adjustment Across Languages (IDAAL)
... is a combination of these three approaches:
Aligned Averages Algorithm (AAA)
Minimum Meansquare Method (MMM)
Person Parameter Contrast Measure (PPCM)

37. How to assemble test forms
EPSO Rasch Item Combination Algorithm (ERICA)

38. Framework Candidates have to be treated equally:
Same number of items
Same difficulty of test form
➾ Adaptive testing is not possible
EPSO has developed an algorithm involving two steps:
Set up a "master test form"
Create test forms corresponding to master test form

39. Target (quality criteria)
Actual test forms correspond to master test form
Limited overlap between test forms
No over-exposed items
Further improving gender balance

40. Defining the master test form
Master test form difficulty: 5
easy
difficult
Master test form difficulty: 8
Master test form difficulty: 2

41. Weighting Items are approximately standard normal distributed
Most items are around the average (0) available
Weighting was implemented to better take account of item availabilities

42. Sample Test Form

43. Thank you very much for your attention!
Any questions?