1. How does DIALANG use the CEF?
overall Framework and test content
self-assessment instruments
parts of feedback, e.g. the CEF scale
standard setting

2. DIALANG Assessment Framework
early decision to adopt CEF as DIALANG Assessment Framework
guide on the content of test items
context of language use; communicative tasks, purposes, themes, activities; texts; competences; communicative functions
scale of proficiency
+ detailed specifications of activities, tasks, texts, etc from Waystage, Threshold and Vantage
checklists based on CEF categories  coverage
test specifications only partially based on the CEF

3. DIALANG self-assessment instruments
DEVELOPMENT (PART 1):
DIALANG self-assessment instruments are based on CEF descriptors
best statements chosen (BN’s study)
in English; for reading, listening, writing; per skill
slight modifications for lay audience, e.g. splitting and simplifying further
a few new statements created

4. EXAMPLE: development of statements
Council of E’s statements:
(1) Can understand short simple personal letters.
(2) Can understand enough to manage simple, routine exchanges without undue effort.
DIALANG statements:
(1) I can understand short simple personal letters.
(2) I can understand enough to manage simple, routine exchanges without too much effort.

5. DIALANG self-assessment instruments
DEVELOPMENT (PART 2):
review of modified statements by B. North
translation into 13 languages
pilot testing of statements (with test items)
 statistical analyses & calibration
DIALANG Version 1:
three 18-statement self-assessment instruments, one each for reading / listening / writing
system calculates test taker’s self-assessed proficiency on the CEF scale  feedback

6. DIALANG feedback and CEF 1
Test result / Level of proficiency
name of the level (A1- C2) + graph
description
(based on the CEF)

7. DIALANG feedback and CEF 2
Advisory feedback
expanded descriptions of the levels
advice on how to improve the skill tested

8. DIALANG feedback and CEF 3
Self-assessment feedback
self-assessed level (A1 - C2)
comparison of SA and test result (levels)

9. Standard setting in DIALANG
purpose: to enable DIALANG to report test results on the CEF scale (by establishing cut-off scores)

10. WHY DO WE NEED STANDARD SETTING?
ILLUSTRATION
WHY DO WE NEED STANDARD SETTING?
a DIALANG test
TEST SCORE SCALE
110 (MAXIMUM SCORE)
109 … 70 69 68 67 66 65 64 2
1 (MINIMUM SCORE)

11. C2 C1 B2 ? B1 A2 A1 a DIALANG test CEF levels TEST SCORE CEF LEVEL
110 (MAX SCORE)
109 … 70 69 68 67 66 65 64 2
1 (MINIMUM SCORE) C2 C1
B ? B1 A2 A1

12. C2 ? C1 ? B2 ? B1 ? A2 ? A1? a DIALANG test CEF levels 71 70 69 68 67
TEST SCORE
ABILITY SCALE
CEF LEVEL … 71 70 69 68 67 66 65 64 63
.455
.421
.343
.300
.276
.240
.233
.199
.138
C2 ?
C1 ?
B2 ?
B1 ?
A2 ?
A1?

13. Cut-off points? C2 ? C1 ? B2 ? B1 ? A2 ? A1 ? a DIALANG test
Cut-offs /
st. setting
CEF
levels
TEST SCORE
ABILITY SCALE
CEF LEVEL … 71 70 69 68 67 66 65 64 63
.455
.421
.343
.300
.276
.240
.233
.199
.138
C2 ?
C1 ?
B2 ?
B1 ?
A2 ?
A1 ?
Cut-off
points?

14. B2 .275 B1 a DIALANG test Cut-offs / st. setting CEF levels 70 69 68
TEST SCORE
ABILITY SCALE
CEF LEVEL … 70 69 68 67
.421
.343
.300
.276
.275 B2 66 65 64
.240
.233
.199 B1

15. B2 .275 B1 a DIALANG test Cut-offs / st. setting CEF levels persons =
TEST SCORE
ABILITY SCALE
CEF LEVEL … 70 69 68 67
.421
.343
.300
.276
.275 B2 66 65
.240
.233 B1
persons =
persons
items =
items =
items

16. a DIALANG test Cut-offs / st. setting CEF levels .944 .275 -.243
TEST SCORE
ABILITY SCALE
CEF LEVEL … 90
.955
.944
.275
-.243
-1.266 C1 89 67
.924
.276 B2 66 30
.240
- .224 B1 29 15
- .250 A2 14
1.303 A1

17. Standard setting in DIALANG 1
steps in the DIALANG standard setting procedure:
establish a group of experienced language teachers for each language
familiarisation with the CEF scale
judgement of each item, one skill at a time, against the CEF scale ”Is it reasonable to require that a person at level B1 gets this item right?”
2 versions used
teachers / judges mark their ratings on a form

18. Standard setting in DIALANG 2
analysis of the data with a standard setting programme developed by Norman Verhelst at CITO
results of the IRT analyses of the pilot tested items and the standard setting data (on the same items) are both taken into account
series of cut-off points on an ability/difficulty (logit) scale for the skill in question
comparison of cut-off points with a logit (ability) scale from the test to which the cut-offs are to be applied
placing of cut-off points