1. Test Design & Construction
RSCH 6109: Assessment & Evaluation Methods
Test Design & Construction
Purpose & Framework
Test Specifications or Blueprint
Item Construction
Field Testing
Evaluation & Revision

2. Evaluating the Value of a Test
RSCH 6109: Assessment & Evaluation Methods
Evaluating the Value of a Test
The value of a test can only be determined relative to its purpose and intended use
Value = Reliability, Validity, Cultural Sensitivity
However, “value” is not a property of the test itself
Reliability, Validity, and Cultural Sensitivity are properties of the information the test provides in the context of the administration, scoring, interpretation, and application to a particular population
Contrast Proper and Improper use of the BDI

3. The Pilot Study RSCH 6109: Assessment & Evaluation Methods
Give the test to a small group from the target audience
Ask them to complete the test and to give feedback about the test
How long did it take you to complete the test?
Anything ambiguous / confusing items or wording?
Anything not covered / suggestions for additional questions?
Talk Out Loud Method / Focus Groups
Open-ended questions about the construct

4. Descriptive Statistics – Item Analysis
RSCH 6109: Assessment & Evaluation Methods
Descriptive Statistics – Item Analysis
Item difficulty
Distribution of item responses
Item discrimination
Correlations between items
Begin the process of eliminating or modifying item content

5. Reliability Studies RSCH 6109: Assessment & Evaluation Methods
Internal consistency (Consistency)
Split-half reliability (Consistency)
Test-retest reliability (Stability)
Inter-rater reliability (Agreement)
Further the process of eliminating or modifying items

6. Construct Validity RSCH 6109: Assessment & Evaluation Methods
What is the test really measuring?
How many sub-scales should there be?
Exploratory factor analysis
Define the subscales that emerge
Consider eliminating or adding items
Higher order factors

7. Content Validity RSCH 6109: Assessment & Evaluation Methods
Does the test really cover the content domain?
Expert review
Comparison to theory
Consider additional items

8. Further Validation RSCH 6109: Assessment & Evaluation Methods
Criterion Validity
Concurrent Validity / MTMM matrix
Predictive Validity
Known Group Differences Validity
Confirmatory Factor Analysis

9. Cultural Sensitivity RSCH 6109: Assessment & Evaluation Methods
Expert review
DTF – Differential Test Functioning
DIF – Differential Item Functioning

10. Final Stages RSCH 6109: Assessment & Evaluation Methods
Create Norm Tables
Create Norm Tables for Sub-groups
Prepare the technical manual
Guidelines for administration
Scoring
Interpretation
Technical properties
Compare the reliability, validity, and cultural sensitivity evidence and technical manual to the Standards