1. Professor Jim Tognolini
Presentation 5: Analysing Tests and Test Items using Classical Test Theory (CTT)
Professor Jim Tognolini

2. Analysing Tests and Test Items using Classical Test Theory (CTT)
During this session we will
define some basic test level statistics using Classical Test Theory analyses: test mean, test discrimination and test reliability (Chronbach’s Alpha).
define some basic item level statistics from Classical test theory: item difficulty, item discrimination (Findlay Index and Point Biserial Correlation).

3. Test characteristics to evaluate
Difficulty
Discrimination
Reliability
Validity

4. Test difficulty

5. Test discrimination
The ability of a test to discriminate between high- and low-achieving individuals is a function of the items that comprise the test.

6. Methods of estimating reliability
Type of Reliability
Procedure
Test-Retest
Stability Reliability
Give the same test to the same group on different occasions with some time between tests.
Equivalent Forms
Equivalent Reliability
Give two forms (parallel forms) of the test to the same group in close succession.
Split-half
Internal Consistency
Give test once; split test in half (odd/even); get the correlation between the score; correct the correlation between halves using the Spearman-Brown formula.
Coefficient Alpha
Give test once to a group and apply formula.
Interrater
Consistency of Ratings
Get two or more raters to score the responses and calculate the correlation coefficient.

7. Split-halves method
Reliability can also be estimated from a single administration of a test, either by correlating the two halves or by using the Kuder-Richardson Method.
The Split-halves method requires the test to be split into halves which are most equivalent.
To estimate the reliability of the full test the Spearman-Brown Adjustment is usually applied

8. Kuder-Richardson (KR-20 and KR-21) Method

9. Cronbach’s alpha method

10. Ways to improve reliability
Test length
In general the longer the test the higher the reliability (more adequate sampling) provided that the material that is added is identical in statistical and substantive properties
Homogeneity of group
The more heterogeneous the group, the high the reliability. It can vary at different score levels, gender, location, etc.
Difficulty of items
Tests that are too difficult or too hard provide results of low reliability. Generally set tests of item difficulty equal to 0.5. In general with tests that are required to discriminate, spread questions over the range in which the discrimination is required.

11. Ways to improve reliability
Objectivity
The more objective the test (and marking scheme) the more reliable are the resulting test scores.
Retain Discriminating Items
In general replace items with a low discrimination with those that highly discriminate. There comes a point where this practice raises the reliability to such a point that it lowers validity (attenuation paradox).
Increase Speededness of the Tests
Highly speeded tests usually show higher reliability. Don’t use internal consistency estimates.

12. Types of validity
There are many different types of validity. Traditionally there are three main types:
Content Validity (sometimes referred to as curricular or instructional validity)
Criterion Related Validity (types include predictive and concurrent validity)
Construct Validity
Face Validity
Loevinger (1957) argued that “since predictive, concurrent and content validities are all essentially ad hoc, construct validity of the whole of validity from a scientific point of view”

13. Define some basic item level statistics from Classical Test Theory

14. Item difficulty

15. Item discrimination Methods for checking item discrimination include
The Findlay Index (FI)
The Point Biserial Correlation
The Biserial Correlation

16. The Findlay Index (FI)

17. The Findlay Index (FI) – An example

18. The Findlay Index (FI) FI = PRU - PRL
If the number of students in the top group is not equal to the number in the bottom group proportions must be used.
where
PRU = proportion of persons right in upper group
PRL = proportion of persons right in lower group
FI = PRU - PRL

19. Graphical display of the Findlay Index (FI)
Calculate the proportion of the group getting the item correct and then plot this against the mean score for the particular group mean scores for each group.

20. Graphical display of the Findlay Index (FI)

21. The Findlay Index (FI) – An example
Item Type SA E
Total
Item Number 1 2 3 4 5 6 7 8 9 10 11 12
Max Marks 28
Astha 18
Bosco
Chetan 21
Devika 16
Emily
Farhan 24
Gogi
Harshita
Indu
Jagat 22
TOTAL 23 14 20 31
176

22. The Findlay Index (FI) – An example
Item Type SA E
Total
Item Number 1 2 3 4 5 6 7 8 9 10 11 12
Max Marks
Astha 18
Bosco
Chetan 21
Devika 16
Emily
Farhan 24
Gogi
Harshita
Indu
Jagat 22
TOTAL 23 14 20 31
176
Difficulty/Mean
0.9
1.0
0.8
0.5
2.3
0.4
1.4
2.0
1.8
2.4
3.1

23. The Findlay Index (FI) – An example
Item Type SA E
Total
Item Number 1 2 3 4 5 6 7 8 9 10 11 12
Max Marks 28
Astha 18
Bosco
Chetan 21
Devika 16
Emily
Farhan 24
Gogi
Harshita
Indu
Jagat 22
TOTAL 23 14 20 31
176
Difficulty/Mean
0.9
1.0
0.8
0.5
2.3
0.4
1.4
2.0
1.8
2.4
3.1
17.6
P-Value
0.2
0.7

24. The Findlay Index (FI) – An example
Item Type SA E
Total
Item Number 1 2 3 4 5 6 7 8 9 10 11 12
Max Marks 28
Farhan
0.5
0.75
0.7 24
Harshita
Jagat
0.8 22
Chetan
0.3 21
Emily
Astha
0.25 18
Bosco
Devika 16
Indu
Gogi
0.2
P-Value
0.9
1.0

25. The Findlay Index (FI) – An example
Item Type SA E
Total
Item Number 1 2 3 4 5 6 7 8 9 10 11 12
Max Marks 28
Farhan
0.5
0.75
0.7 24
Harshita
Jagat
0.8 22
Chetan
0.3 21
Emily
Astha
0.25 18
Bosco
Devika 16
Indu
Gogi
0.2
P-Value
0.9
1.0 FI
0.0
0.6
-0.1
0.4

26. Guttman scale Item Type SA E Total Item Number 2 5 1 3 6 11 8 9 10 412 7
Max Marks 28
Farhan
0.75
0.7
0.5 24
Harshita
Jagat
0.8 22
Chetan
0.3 21
Emily
Astha
0.25 18
Bosco
Devika 16
Indu
Gogi
0.2
P-Value
1.0
0.9

27. Point-biserial correlation

28. The Guttman structure
If person A scores better than person B on the test, then person A should have all the items correct that person B has, and in addition, some other items that are more difficult.
Louis Guttman

29. The Guttman structure (cont.)

30. Reasons for not obtaining a strict Guttman pattern
The items do not go together as expected and the scores on the items should not be added.
The items are very close in difficulty and the persons are all close in ability.

31. Guttman scale Item Type SA E Total Item Number 2 5 1 3 6 11 8 9 10 412 7
Max Marks 28
Farhan
0.75
0.7
0.5 24
Harshita
Jagat
0.8 22
Chetan
0.3 21
Emily
Astha
0.25 18
Bosco
Devika 16
Indu
Gogi
0.2
P-Value
1.0
0.9

32. Individual reporting 3 11 2 15 14 9 8 1 7 4 13 12 5 10 6

33. Individual reporting 3 11 2 15 14 9 8 1 7 4 13 12 5 10 6