1. Making MAP More Meaningful Winter 2008 David Dreher, Project Coordinator Office of Accountability Highline Public Schools

2. Overview Recap of Predicting 2008 WASL Examining 2008 WASL Predictions Moving Forward to 2009

3. RECAP WERA Spring 08 Presentation Predictions Released November 2007 –A “best guess” about each student’s performance on the upcoming WASL based on prior MAP and/or WASL performance –Intended Uses Provide building staff with a level of risk for not meeting WASL standard. School- and District- level 2008 WASL “forecasts” –Theory: Putting MAP scores in context with WASL scores will make MAP more meaningful.

4. Example of Projection and Prediction 7 th Grade Student in Reading MAP Scores Winter 2006Spring 2007Fall 2007Highest MAP 210212216 Expected MAP Growth 3 WASL 2007 Projected MAP Spring 2008 392219 Predicted Prediction Model 2008 WASL Score 2008 WASL Range MAP and WASL399392 - 406 MAP Only402395 - 409 WASL Only397390 - 404

5. WASL Prediction Range Constructed using the SEM values reported in the 2001 WASL Technical Reports. Predicted Range = Predicted WASL Score +/- SEM Grade LevelSEM – ReadingSEM – Math 3, 4, 5, 6712 7,8716 10 12

6. Interpreting Predictions If the prediction range is: –Entirely below 400 (ex.: 380-396): student has less than a 20% chance on the WASL this spring unless we accelerate their learning. –Straddles 400 (ex.: 396-410): student has basically a coin-flip chance on the WASL, even if their prediction is above 400. –Entirely above 400 (ex.: 408-424): student has more than an 80% chance on the WASL in the spring, IF they continue to progress.

9. NWEA’s MAP/WASL Alignment Study Released January 2008 Reading Fall Testing Window GradeNWEA “Meets Standard” Cut Score HPS Accountability “Strategic” RIT Range 3187182-190 4190189-197 5199199-207 6209206-214 7216210-218 8215213-221 9214212-220 10213210-218

10. NWEA’s MAP/WASL Alignment Study Released January 2008 Mathematics Fall Testing Window GradeNWEA “Meets Standard” Cut ScoreStrategic RIT Range 3189183-191 4202198-206 5211207-215 6221219-227 7228223-231 8233229-237 9236233-241 10239237-245

11. Spring 2008: WASL happened... Late Summer 2009: WASL results arrive!

12. Examining the Predictions Reliability Analysis –Repeated the “Backward Look” analysis –“Within Group Look” Analysis of “Exceptional” Performances –Predicted Level Analysis

13. Grade WASL 2008 “Backward Look” (%) WASL 2007 “Backward Look” (%) 480.187.3 593.886.8 692.986.8 785.189.8 897.589.0 1096.186.9 “Backward Look”: Math % = Actual Met / Predicted to Meet Predicted to Meet = Predicted WASL score of 400 or better.

14. Grade WASL 2008 “Backward Look” (%) WASL 2007 “Backward Look” (%) 493.983.1 5113.381.8 6105.283.1 785.282.5 868.981.0 1094.387.7 “Backward Look”: Reading % = Actual Met / Predicted to Meet Predicted to Meet = Predicted WASL score of 400 or better.

15. Benchmark (%) Strategic (%) Intensive (%) GradeEstimated >80%Estimated ~50%Estimated <20% 485.240.32.5 593.147.36.7 687.856.48.9 783.134.42.5 894.851.55.4 1091.651.16.5 “Within Group”: Math % = Actual Met Within Group / Total Number In Group

16. Benchmark (%) Strategic (%) Intensive (%) GradeEstimated >80%Estimated ~50%Estimated <20% 489.345.18.7 591.962.017.1 688.960.816.5 784.245.310.0 886.750.512.6 1092.257.717.2 “Within Group”: Reading % = Actual Met Within Group / Total Number In Group

17. Questions/Comments Procedures for making predictions Results of reliability analyses What about our theory behind doing this? –“Putting MAP scores in context with WASL scores will make MAP more meaningful.”

18. “Meaningful”: Depends on Who You Ask My experience talking with the people who work directly with the kids suggests that the strength of our ability to assess the risk level of their students doesn’t impress them. –“You don’t need a weatherman to tell you which way the wind blows” Bob Dylan

19. What would be more “Meaningful”? Information that would help determine whether things done to help kids pass the WASL worked Data Users Principals, Coaches, Teachers Data Creators Office Of Accountability

20. But...I don’t really know what schools are doing to try to help kids pass the WASL... So...how can I find out?

21. “ Exceptional” Performance Analysis Please see handout Objective: Start conversations that would increase the flow of information from data users back to us in Accountability What are your observations of the data?

22. Expectations for “Above Level” Students were receiving interventions designed to address skills/knowledge deficits Students were receiving interventions designed to familiarize them with WASL format Students benefited from actions taken by the school to improve the WASL testing environment Your ideas?

23. Expectations for “Below Level” Students were ELL or SPED Students were chronically absent or highly mobile Students did not take the WASL seriously Your ideas?

25. Moving Forward Predictions simplified: Use BSI designations only Raising awareness and understanding of NWEA’s Alignment Study Increase understanding of NWEA goals and how to interpret goal-level results Investigate the possible use of MAP data in evaluation of interventions, initiatives, and programs

26. Continue to Solicit Input Data Users Principals, Coaches, Teachers Data Creators Office Of Accountability

27. Contact Information David Dreher, Project Coordinator Office of Accountability Highline Public Schools www.hsd401.org 206-433-2334

30. What they said... Expected ResponsesUnexpected Responses SchResp Supplement/ Content Supplement/ Format Test Environ School/ Class Environ Perception/ Understanding of MAP Did Not Answer the Question BEVXX XX CED DESX X HAZX X MIDXX MOUXXX MCM SHO WCHXXX CASX?? CHIX?? GLOX X

32. What is MAP M easures of A cademic P rogress –Developed by the Northwest Evaluation Association –Norm-referenced assessment –Computerized and adaptive –Performance is reported as a RIT score The RIT Scale –Uses individual item difficulty values to estimate student achievement –A RIT score has the same meaning regardless of grade level –Equal interval scale Highline Public Schools –Three testing windows per year (Fall, Winter, Spring) –Test students in the areas of math and reading –Test students in grades 3-10

33. The Needs of the Data User Building staff were saying things like... –“How can we use MAP data to help us make decisions?” –“How do MAP and WASL performance compare?” –“I want to know what a student’s history is with MAP.” –“What is a RIT score?” –“Giving me a RIT score is like telling me the temperature in Celsius!”

34. Making The Predictions Snooped and found the best indicators of WASL success Applied linear regression models to generate WASL scores for each student Examined the predicted WASL scores

35. Projecting MAP to Spring For the models with “Projected MAP” as one of the factors individual student performance on MAP in the Spring of 2008 was projected. –The amount of expected growth added to a student’s Highest MAP score came from NWEA’s Growth Study

36. Snooping (Reading) R-Values WASL 2007 Reading Scale vs. Grade 34567810 WASL Reading 2006 0.7300.7690.7640.7450.755 WASL Reading 2005 0.727 WASL Reading 2004 0.723 MAP-R Spring 2007 0.7920.7780.7990.7740.7440.7260.755 MAP-R Winter 2007 0.782 0.7960.7670.7800.7110.786 MAP-R Fall 2006 0.7690.7500.7930.768 0.7550.806 High MAP-Read (F06, W07, S07) 0.8040.7790.8160.7840.7910.7460.804 High MAP-Read + High MAP-Math 0.814 High MAP-Read +WASL06 Read scale 0.8030.8350.8240.816 0.780 High MAP-Read + WASL 04_Rscale 0.787

37. Snooping (Math) R-values WASL 2007 Math vs. Grade 34567810 WASL Math 2006 0.8010.8400.8750.8730.890 WASL Math 2005 0.603 WASL Math 2004 0.856 MAP-Math Spring 2007 0.8170.862 0.8650.8880.8960.828 MAP-Math Winter 2007 0.8090.8510.8600.8630.8830.9060.856 MAP-Math Fall 2006 0.7940.8170.8480.8330.8790.9070.865 High MAP-Math (F06, W07, S07) 0.8320.8770.8790.8780.9020.9150.879 High MAP-Math + High MAP-Read 0.846 High MAP-Math + WASL06 Math scale 0.892 0.8960.9100.921 0.930 Highest MAP-M + WASL04 Math scale 0.908

38. What we learned by snooping... Correlations were generally good. –Reading R-value range: 0.711 - 0.835 –Math R-value range: 0.603 - 0.921 Correlations in math were stronger than in reading. “Highest MAP” consistently correlated better than any single MAP score. Correlations were generally strongest when Highest MAP and WASL 2006 factors were combined.

39. Regression Models For students with both MAP and 2006 WASL scores (~95%) WASL 2007 = b0 + b1*Highest MAP + b2*WASL 2006 For students that only had MAP score(s) (~3%) WASL 2007 = b0 + b1*Highest MAP For students that only had WASL 2006 score (~2%) WASL 2007 = b0 + b1*WASL 2006 Where: Highest MAP = The student’s highest score on MAP from the Fall 2006, Winter 2007, or Spring 2007 windows. Typically Spring 2007. WASL 2006 = The student’s raw score from the 2006 WASL Spring testing.

40. Prediction Models For students with both MAP and 2007 WASL scores WASL 2008 = b0 + b1*Projected MAP + b2*WASL 2007 For students with only MAP score(s) WASL 2008 = b0 + b1*Projected MAP For students with only WASL 2007 score WASL 2008 = b0 + b1* WASL 2007 Where: Projected MAP = Projected Spring 2008 MAP score based on the student’s highest score on MAP from the Winter 2007, Spring 2007 or Fall 2008 windows. WASL 2007 = The student’s raw score from the 2007 WASL Spring testing.