Making MAP More Meaningful Winter 2008 David Dreher, Project Coordinator Office of Accountability Highline Public Schools

Overview Recap of Predicting 2008 WASL Examining 2008 WASL Predictions Moving Forward to 2009

RECAP WERA Spring 08 Presentation Predictions Released November 2007 –A “best guess” about each student’s performance on the upcoming WASL based on prior MAP and/or WASL performance –Intended Uses Provide building staff with a level of risk for not meeting WASL standard. School- and District- level 2008 WASL “forecasts” –Theory: Putting MAP scores in context with WASL scores will make MAP more meaningful.

Example of Projection and Prediction 7 th Grade Student in Reading MAP Scores Winter 2006Spring 2007Fall 2007Highest MAP Expected MAP Growth 3 WASL 2007 Projected MAP Spring Predicted Prediction Model 2008 WASL Score 2008 WASL Range MAP and WASL MAP Only WASL Only

WASL Prediction Range Constructed using the SEM values reported in the 2001 WASL Technical Reports. Predicted Range = Predicted WASL Score +/- SEM Grade LevelSEM – ReadingSEM – Math 3, 4, 5, ,

Interpreting Predictions If the prediction range is: –Entirely below 400 (ex.: ): student has less than a 20% chance on the WASL this spring unless we accelerate their learning. –Straddles 400 (ex.: ): student has basically a coin-flip chance on the WASL, even if their prediction is above 400. –Entirely above 400 (ex.: ): student has more than an 80% chance on the WASL in the spring, IF they continue to progress.

NWEA’s MAP/WASL Alignment Study Released January 2008 Reading Fall Testing Window GradeNWEA “Meets Standard” Cut Score HPS Accountability “Strategic” RIT Range

NWEA’s MAP/WASL Alignment Study Released January 2008 Mathematics Fall Testing Window GradeNWEA “Meets Standard” Cut ScoreStrategic RIT Range

Spring 2008: WASL happened... Late Summer 2009: WASL results arrive!

Examining the Predictions Reliability Analysis –Repeated the “Backward Look” analysis –“Within Group Look” Analysis of “Exceptional” Performances –Predicted Level Analysis

Grade WASL 2008 “Backward Look” (%) WASL 2007 “Backward Look” (%) “Backward Look”: Math % = Actual Met / Predicted to Meet Predicted to Meet = Predicted WASL score of 400 or better.

Grade WASL 2008 “Backward Look” (%) WASL 2007 “Backward Look” (%) “Backward Look”: Reading % = Actual Met / Predicted to Meet Predicted to Meet = Predicted WASL score of 400 or better.

Benchmark (%) Strategic (%) Intensive (%) GradeEstimated >80%Estimated ~50%Estimated <20% “Within Group”: Math % = Actual Met Within Group / Total Number In Group

Benchmark (%) Strategic (%) Intensive (%) GradeEstimated >80%Estimated ~50%Estimated <20% “Within Group”: Reading % = Actual Met Within Group / Total Number In Group

Questions/Comments Procedures for making predictions Results of reliability analyses What about our theory behind doing this? –“Putting MAP scores in context with WASL scores will make MAP more meaningful.”

“Meaningful”: Depends on Who You Ask My experience talking with the people who work directly with the kids suggests that the strength of our ability to assess the risk level of their students doesn’t impress them. –“You don’t need a weatherman to tell you which way the wind blows” Bob Dylan

What would be more “Meaningful”? Information that would help determine whether things done to help kids pass the WASL worked Data Users Principals, Coaches, Teachers Data Creators Office Of Accountability

But...I don’t really know what schools are doing to try to help kids pass the WASL... So...how can I find out?

“ Exceptional” Performance Analysis Please see handout Objective: Start conversations that would increase the flow of information from data users back to us in Accountability What are your observations of the data?

Expectations for “Above Level” Students were receiving interventions designed to address skills/knowledge deficits Students were receiving interventions designed to familiarize them with WASL format Students benefited from actions taken by the school to improve the WASL testing environment Your ideas?

Expectations for “Below Level” Students were ELL or SPED Students were chronically absent or highly mobile Students did not take the WASL seriously Your ideas?

Moving Forward Predictions simplified: Use BSI designations only Raising awareness and understanding of NWEA’s Alignment Study Increase understanding of NWEA goals and how to interpret goal-level results Investigate the possible use of MAP data in evaluation of interventions, initiatives, and programs

Continue to Solicit Input Data Users Principals, Coaches, Teachers Data Creators Office Of Accountability

Contact Information David Dreher, Project Coordinator Office of Accountability Highline Public Schools

What they said... Expected ResponsesUnexpected Responses SchResp Supplement/ Content Supplement/ Format Test Environ School/ Class Environ Perception/ Understanding of MAP Did Not Answer the Question BEVXX XX CED DESX X HAZX X MIDXX MOUXXX MCM SHO WCHXXX CASX?? CHIX?? GLOX X

What is MAP M easures of A cademic P rogress –Developed by the Northwest Evaluation Association –Norm-referenced assessment –Computerized and adaptive –Performance is reported as a RIT score The RIT Scale –Uses individual item difficulty values to estimate student achievement –A RIT score has the same meaning regardless of grade level –Equal interval scale Highline Public Schools –Three testing windows per year (Fall, Winter, Spring) –Test students in the areas of math and reading –Test students in grades 3-10

The Needs of the Data User Building staff were saying things like... –“How can we use MAP data to help us make decisions?” –“How do MAP and WASL performance compare?” –“I want to know what a student’s history is with MAP.” –“What is a RIT score?” –“Giving me a RIT score is like telling me the temperature in Celsius!”

Making The Predictions Snooped and found the best indicators of WASL success Applied linear regression models to generate WASL scores for each student Examined the predicted WASL scores

Projecting MAP to Spring For the models with “Projected MAP” as one of the factors individual student performance on MAP in the Spring of 2008 was projected. –The amount of expected growth added to a student’s Highest MAP score came from NWEA’s Growth Study

Snooping (Reading) R-Values WASL 2007 Reading Scale vs. Grade WASL Reading WASL Reading WASL Reading MAP-R Spring MAP-R Winter MAP-R Fall High MAP-Read (F06, W07, S07) High MAP-Read + High MAP-Math High MAP-Read +WASL06 Read scale High MAP-Read + WASL 04_Rscale 0.787

Snooping (Math) R-values WASL 2007 Math vs. Grade WASL Math WASL Math WASL Math MAP-Math Spring MAP-Math Winter MAP-Math Fall High MAP-Math (F06, W07, S07) High MAP-Math + High MAP-Read High MAP-Math + WASL06 Math scale Highest MAP-M + WASL04 Math scale 0.908

What we learned by snooping... Correlations were generally good. –Reading R-value range: –Math R-value range: Correlations in math were stronger than in reading. “Highest MAP” consistently correlated better than any single MAP score. Correlations were generally strongest when Highest MAP and WASL 2006 factors were combined.

Regression Models For students with both MAP and 2006 WASL scores (~95%) WASL 2007 = b0 + b1*Highest MAP + b2*WASL 2006 For students that only had MAP score(s) (~3%) WASL 2007 = b0 + b1*Highest MAP For students that only had WASL 2006 score (~2%) WASL 2007 = b0 + b1*WASL 2006 Where: Highest MAP = The student’s highest score on MAP from the Fall 2006, Winter 2007, or Spring 2007 windows. Typically Spring WASL 2006 = The student’s raw score from the 2006 WASL Spring testing.

Prediction Models For students with both MAP and 2007 WASL scores WASL 2008 = b0 + b1*Projected MAP + b2*WASL 2007 For students with only MAP score(s) WASL 2008 = b0 + b1*Projected MAP For students with only WASL 2007 score WASL 2008 = b0 + b1* WASL 2007 Where: Projected MAP = Projected Spring 2008 MAP score based on the student’s highest score on MAP from the Winter 2007, Spring 2007 or Fall 2008 windows. WASL 2007 = The student’s raw score from the 2007 WASL Spring testing.