1. REVIEW I Reliability scraps Index of Reliability Theoretical correlation between observed & true scores Standard Error of Measurement Reliability measure Degree to which an observed score fluctuates due to measurement errors Factors affecting reliability A test must be RELIABLE to be VALID

2. REVIEW II Types of validity Content-related (face) Represent knowledge Use “experts” to establish Criterion-related Evidence of a statistical relationship w/ trait being measured Alternative measures must be validated w/ criterion measure Construct-related Validates unobservable theoretical measures

3. REVIEW III Standard Error of Estimate Validity measure Degree of error in estimating a score based on the criterion Methods of obtaining a criterion measure Actual participation Experts Perform criterion Known valid test Interpreting “r”

4. Criterion-Referenced Measurement PoorSufficientBetter

5. Guidelines for Writing Behavioral Objectives (Mager, 1962) Identify the desired behavior/action by name Define the desired condition Specify the criteria of acceptable performance

6. Criterion-Referenced Testing Mastery Learning Standard Development Judgmental: use experts Normative: theoretically accepted criteria Empirical: cutoff based on available data Combination: expert & norms typically combined

7. Advantages of Criterion-Referenced Measurement Represent specific, desired performance levels linked to a criterion Independent of the %of the population that meets the standard If not met, specific diagnostic evaluations can be made Degree of performance is not important... reaching the standard is Performance linked to specific outcomes Individuals know exactly what is expected of them

8. Limitations of Criterion-Referenced Measurement Cutoff scores always involve subjective judgment Misclassifications can be severe Student motivation can be impacted; frustrated/bored

9. Setting a Cholesterol “Cut-Off” Cholesterol mg/dl N of deaths

10. Setting a Cholesterol “Cut-Off” Cholesterol mg/dl N of deaths

11. Considerations with CRT The same as norm-referenced testing Reliability Consistency of measurement Validity Truthfulness of measurement

12. Statistical Analysis of CRTs Nominal Data Contingency Table Development (2x2 Chi 2 ) Phi Coefficient (PPM for dichotomous variables) Chi-Square Analysis

13. Figure 7.1 (a) FITNESSGRAM Standards (1987) 24 (4%) 21 (4%) 64 (11%) 472 (81%) Did not achieve the standard on the run/walk test Did achieve the standard on the run/walk test Below the criterion VO 2max Above the criterion VO 2max

14. Figure 7.1 (b) AAHPERD Standards (1988) 130 (22%) 23 (4%) 201 (35%) 227 (39%) Did not achieve the standard on the run/walk test Did achieve the standard on the run/walk test Below the criterion VO2max Above the criterion VO2max

15. Meeting Criterion-Referenced Standards Possible Decisions Truly Below Criterion Truly Above Criterion Did not achieve standard Correct Decision False Positive Did achieve standard False Negative Correct Decision

16. Table 7.1 Test-Retest Reliability Example Day 2 Day 1Did not achieve the standard Did achieve the standard Did not achieve the standard 8020100 Did achieve the standard 50250300 130270400 P =.825 K =.576 Phi =.586  2 = 137.13, df = 1, p <.001

17. Table 7-2 Criterion-Referenced Equivalence Reliability Between the 1 Mile Run/Walk and PACER TestsTotal sampleBoysGirls Trial 1 P.76.83.66 K.51.65.33 Trial 2 P.71.76.65 K.43.52.30

18. Figure 7.3 A theoretical example of the divergent group method

19. Examples of Criterion Referenced Standards Cholesterol < 240 mg / dl Systolic Blood Pressure < 140 mmHg Diastolic Blood Pressure < 90 mmHg FITNESSGRAM 1-mile run time for boy age 10 < 11:30

20. CRT Reliability Test/Retest of a single measure Fail Day 1 Pass Fail Pass Day 2

21. CRT Validity Use of a field test and criterion measure Fail Field Test Pass Fail Pass Criterion