1. ~ Test Construction and Validation ~
Fundamental Points and Practices
Stephen J. Vodanovich, Ph.D.

2. ~ Identifying The Item Domain ~
[a.k.a. Where do the questions come from?]
Test
Item Domain
Specific, defined content area (e.g., course exam, training program)
Expert opinion, observation (e.g., professional literature)
Job analysis (identification of major job tasks, duties)

3. Job Analysis Overview Task Identification KSA Identification Job
(or Job Category)
Task 1
Task 2
Task 3
Task 4
KSA 1
KSA 2
KSA 3
KSA 4
Rate Tasks and KSAs
Connect KSAs to Tasks

4. ~ Sample Task Rating Form ~
Frequency of use
5 = almost all of the time
4 = frequently
3 = occasionally
2 = seldom
1 = not performed at all
Importance of performing successfully
5 = extremely important
4 = very important
3 = moderately important
2 = slightly important
1 = of no importance
Importance for new hire
5 = extremely important
Distinguishes between superior & ad performance
5 = a great deal
4 = considerably
3 = moderately
2 = slightly
1 = not at all
Damage if error occurs
5 = extreme damage
4 = considerable damage
3 = moderate damage
2 = very little damage
1 = virtually no damage 1 2 3 4 5 6 7

5. ~ Sample KSA Rating Form ~
Importance for acceptable job performance
5 = extremely important
4 = very important
3 = moderately important
2 = slightly important
1 = of no importance
Importance for new hire
Distinguishes between superior & adequate performance
5 = a great deal
4 = considerably
3 = moderately
2 = slightly
1 = not at all A B C D E F G

6. Sample Task -- KSA Matrix
To what extent is each KSA needed when performing each job task?
5 = Extremely necessary, the job task cannot be performed without the KSA
4 = Very necessary, the KSA is very helpful when performing the job task
3 = Moderately necessary, the KSA is moderately helpful when performing the job task
2 = Slightly necessary, the KSA is slightly helpful when performing the job task
1 = Not necessary, the KSA is not used when performing the job task
KSA A B C D E F G H
Job Tasks 1 2 3 4 5 6 7

7. ~ Writing Test Items ~ Write a lot of questions
Write more questions for the most critical KSAs
Consider the reading level of the test takers

8. ~ Selecting Test Items ~
Initial review by Subject Matter Experts (SMEs)
Connect items to KSAs
Assess difficulty of items relative to job requirements
Suggest revisions to items and answers

9. Sample Item Rating Form
Connect each item to a KSA or two
Rate difficulty of each item (5-point scale) relative to the level of KSA needed in the job)

10. ~ Statistical Properties of Items ~
Item Difficulty levels. Goal is to keep items of moderate difficulty (e.g., p
values between )
“p-value” is % of people getting each item correct
-4    Mean    

11. R E L I A B I L I T Y A N A L Y S I S - S C A L E (A L L)
Mean Std Dev Cases Q Q
Q Q Q Q Q Q Q Q Q Q Q Q
Answers are scored as correct “1” or wrong “0.” So, the mean is the p value of the items (difficulty level or % or people getting each item correct)
Easy items
Acceptable items

12. ~ Statistical Properties of Items (cont.) ~
Internal Consistency
Item correlations with each other. Goal is to select items that relate
moderately to each other or “hang together” reasonably well (e.g., item x total
score correlations of between , “alpha if item deleted” information)

13. ~ Item-Total Statistics ~
Scale mean if item deleted
Scale variance if item deleted
Corrected item-total correlation
Alpha if item deleted Q Q Q Q Q Q Q Q Q Q Q Q Q Q
Alpha = .8374

14. Kirkland v. Department of Correctional Services (1974)
~ Legal Concerns ~
Kirkland v. Department of Correctional Services (1974)
"Without such an analysis (job analysis) to single out the critical knowledge, skills and abilities required by the job, their importance relative importance to each other, and the level of proficiency demanded as to each attribute, a test constructor is aiming in the dark and can only hope to achieve job relatedness by blind luck”
The KSAs tested for must be critical to successful job performance
Portions of the exam should be accurately weighted to reflect the relative importance to the job of the attributes for which they test
The level of difficulty of the exam material should match the level of difficulty of the job

15. Method 2 (Clinical Interview)
Construct Validation
Method 1
(Paper & Pencil)
Method 2 (Clinical Interview)
Method 3
(Peer observation)
Traits
A B C
A B C
A B C A B C
Mono Method
Hetero Method

16. Hetero-Trait; Mono Method .33 .36 .87
(Paper & Pencil)
Method 2 (Clinical Interview)
Method 3
(Peer observation)
Traits
A B C
A B C
A B C
A (Boredom)
B (Dep)
C (Anxiety)
.89
Reliability Figures
.91
.49
Hetero-Trait; Mono Method
.33
.36
.87
.55
.20
.08
.92
.20
.46
.12
.54
.93
.15
.15
.53
.62
.55
.82
.55
.20
.15
.61
.35
.41
.90
.21
.46
.13
.40
.54
.37
.49
.93
.15
.15
.53
.31
.32
.66
.54
.52
.87
Mono-Trait; Hetero-Method
Hetero-Trait; Hetero-Method