Creating Assessments AKA how to write a test

Creating Assessments All good assessments have three key features: All good assessments have three key features: –Validity –Reliability –Usability

Reliability Next to validity, reliability is the most important characteristic of assessment results. Why? 1. It provides the consistency to make validity possible. 2. It indicates the degree to which various kinds of generalizations are justifiable.

Reliability re·li·a·ble adj. Capable of being relied on; dependable. re·li”a·bil“i·ty or re·li“a·ble·ness n. --re·li“a·bly adv. (American Heritage Dictionary)

Reliability Reliability: the consistency of measurement, i.e. how consistent test scores or other assessment results are from one measurement to another.

Reliability Which is more reliable?

Reliability Classical Test Theory: X = T + e Where:X = observed score T = “true score” e = error

Reliability x = observed score: The score the student receives on the exam. T = “true score”: What the student “really” knows.

Reliability e = error Error variance is the variability that exists in a set of scores and is due to factors other than the one being assessed. –Systematic: errors that are consistent. –Random: errors that have no pattern.

Reliability e = error Positive error (i.e. raises score): –Lucky guesses. –Items that give clues to the answer. –Cheating (students, aides, teachers).

Reliability e = error score Negative error (i.e. lowers score): –Not following directions. –Miss-marking items. –Room climate/atmosphere. –Hunger, fatigue, illness, “need to go potty”. –Assemblies, ball games, fire drills, etc. –Break-up of a relationship.

Circle the figures that are half shaded.

Reliability Determining Reliability: Test-retest method Test-retest method Equivalent forms Equivalent forms Split half method Split half method KR-20 method KR-20 method Interrater reliability Interrater reliability Intrarater reliability Intrarater reliability

Reliability Standard Error of Measurement (SEM)= the estimated amount of variation expected in a score.

Reliability Example: If Sara scored 78 on a standardized test with a SEM of 6 we can be: 68% certain her true score is between 72 and 84 68% certain her true score is between 72 and 84 95% certain her true score is between 66 and 90 95% certain her true score is between 66 and 90 99% certain her true score is between 60 and 96 99% certain her true score is between 60 and 96

Reliability Summation of Reliability: 1. Reliability refers to the results and not to the instrument itself. 2. Reliability is a necessary but not sufficient condition for validity. 3. The more reliable the assessment, the better.

Usability The practical aspects of a test cannot be neglected: –Ease of administration –Time Administration Administration Scoring Scoring –Ease of Interpretation –Availability of equivalent forms –Cost