1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012

2 Maximizing Predictive Accuracy of District Benchmarks Objective 1. Better understand how performance level setting is key to predictive validity. 2. Better understand how to create performance level bands based on equipercentile equating

3 Comparing District Benchmarks to CST Results Common Methods for Setting Cutoffs on District Benchmarks:  Use default settings on assessment platform (e.g. 20%, 40%, 60%, 80%)  Ask curriculum experts for their opinion of where cutoffs should be set  Determine percent correct corresponding to performance levels on CSTs and apply to benchmarks

4 Comparing District Benchmarks to CST Results There is a better way!

5 Comparing District Benchmarks to CST Results “Two scores, one on form X and the other on form Y, may be considered equivalent if their corresponding percentile ranks in any given group are equal.” (Educational Measurement-Second Edition, p. 563)

6 Comparing District Benchmarks to CST Results  Equipercentile Method of Equating at the Performance Level Cut-points  Establishes cutoffs for benchmarks at equivalent local percentile ranks as cutoffs for CSTs  By applying same local percentile cutoffs to each trimester benchmark, comparisons across trimesters within a grade level are more defensible

7 Equipercentile Equating Method Step 1-Identify CST SS Cut-points

8 Equipercentile Equating Method Step 2 - Establish Local Percentiles at CST Performance Level Cutoffs (from scaled score frequency distribution)

9 Equipercentile Equating Method Step 3 – Locate Benchmark Raw Scores Corresponding to the CST Cutoff Percentiles (from benchmark raw score frequency distribution)

10 Equipercentile Equating Method Step 4 – Validate Classification Accuracy – Old Cutoffs

11 Equipercentile Equating Method Step 4 – Validate Classification Accuracy – Old Cutoffs

12 Equipercentile Equating Method Step 4 – Validate Classification Accuracy – Old Cutoffs

13 Equipercentile Equating Method Step 4 – Validate Classification Accuracy – New Cutoffs

14 Equipercentile Equating Method Step 4 – Validate Classification Accuracy – New Cutoffs

15 Equipercentile Equating Method Step 4 – Validate Classification Accuracy – New Cutoffs

16 Example: Classification Accuracy Biology OldNew 2 nd Semester Proficient or Advanced42%77% Each Level38%55% 1 st Semester Proficient or Advanced30%77% Each Level31%50%

17 Example: Classification Accuracy Biology OldNew 1 st Quarter Proficient or Advanced53%71% Each Level41%46%

18 Example: Classification Accuracy Chemistry OldNew 2 nd Semester: Prof. & Adv. 63%79% 2 nd Semester: Each Level 47%52% 1 st Semester: Prof. & Adv. 74% 1 st Semester: Each Level 49%50% 1 st Quarter: Prof. & Adv. 83%76% 1 st Quarter: Each Level 48%47%

19 Example: Classification Accuracy Earth Science OldNew 2 nd Semester: Prof. & Adv. 48%68% 2 nd Semester: Each Level 43%52% 1 st Semester: Prof. & Adv. 33%66% 1 st Semester: Each Level 38%47% 1 st Quarter: Prof. & Adv. 42%56% 1 st Quarter: Each Level 34%41%

20 Example: Classification Accuracy Physics OldNew 2 nd Semester: Prof. & Adv. 57%87% 2 nd Semester: Each Level 37%57% 1 st Semester: Prof. & Adv. 60%88% 1 st Semester: Each Level 42%50% 1 st Quarter: Prof. & Adv. 65%87% 1 st Quarter: Each Level 47%45%

21 Things to Consider Prior to Establishing the Benchmark Cutoffs  Will there be changes to the benchmarks after CST percentile cutoffs are established?  If NO then raw score benchmark cutoffs can be established by linking CST to same year benchmark administration (i.e. spring 2011 CST matched to benchmark raw scores)  If YES then wait until new benchmark is administered and then establish raw score cutoffs on benchmark  How many cases are available for establishing the CST percentiles? (too few cases could lead to unstable percentile distributions)

22 Things to Consider Prior to Establishing the Benchmark Cutoffs (Continued)  How many items comprise the benchmarks to be equated? (as test gets shorter it becomes more difficult to match the percentile cutpoints established on the CST’s)

23 Summary Equipercentile Equating Method  Method generally establishes a closer correspondence between the CST and Benchmarks  Comparisons between benchmark and CST performance can be made more confidently  Comparisons between benchmarks within the school year can be made more confidently

24 Coming Soon from Illuminate Education, Inc.! Reports using the equipercentile methodology are being programmed to: (1) establish benchmark cutoffs for performance bands (2) create validation tables showing improved classification accuracy based on the method

Contact: Tom Barrett, Ph.D. President, Barrett Enterprises, LLC (office) (cell) 25