PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 1 Effect Size Lens on Local SAT and State Math Test Results NC Accountability Conference February 2008 Presentation by Johnny Wilson, Charles Lanier, and Mike Gallagher, all from the Accountability Services Dept., NCDPI

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 2 Purposes – Overall Session Provide background and access paths to important data. (important = helpful) Show how LEAs and schools can access and compare test results across EOG, EOC and SAT. Entice folks to exploit the usefulness of such comparisons.

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 3 Who’s on First? 1.Johnny Wilson, Reporting Section, SAT Data Report. 2.Charles Lanier, Test Development Section, State Testing Data in the Green Book. 3.Mike Gallagher, Test Development, Effect Size Look at SAT and State Test Results.

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 4 Purposes – Part 1: SAT Get acquainted with the major trends in SAT results for North Carolina. Look at school level variability of SAT results from year-to-year. Show the access path for information at the school and LEA levels.

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 5 SAT Math Score Trends Between 1997 and 2007, the Math gap between NC and the Nation narrowed from 23 pts. to 6 pts. NC’s Math scores fell in 2007 after more than a decade of gains.

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 6 SAT Math Score Trends

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 7 SAT Verbal Score Trends By comparison, NC’s Verbal scores have been up and down over the same period. Between 1997 and 2007, the Math gap between NC and the Nation has narrowed from 15 pts. to 7 pts.

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 8 SAT Verbal Score Trends

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 9 SAT Total Score Trends for NC and the Nation After 10 consecutive years of gains, NC’s mean total SAT scores fell in 2006 and Gap between NC’s mean total score and the nation’s score has narrowed from 53 in 1990 to 13 points in 2007.

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 10 Total Score Trends for NC and the Nation

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 11 NC’s Schools with Declining SAT Scores At the school level, math scores also declined in 2006 & 2007, compared with the previous year. Notably, fewer schools increased math scores in 2007 than in 2006.

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 12 Percent of NC’s Schools with Declining SAT Scores

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 13 Changes in SAT Math Scores Most changes in SAT scores from year to year are not unusual. Low-Volume Schools tend to have larger changes. The larger the test-taking population, the smaller changes in SAT scores tend to be.

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 14 Changes in SAT Math Scores by Test-taking Volume (US)

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 15 Changes in SAT Math Scores by Test-taking Volume (NC)

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 16 Changes in SAT Math Scores All Schools (US)

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 17 Changes in SAT Math Scores All Schools (NC)

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 18 Factors that Might Influence Score Fluctuations Fatigue Factor SAT fee increase in 2005 resulted in decline in repeat test-takers. Difficulty of the test

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 19 NC’s Asian students had the highest SAT score among racial/ethnic groups, followed by White students. Only Asian and White students scored above the state average. NC SAT Math Scores by Race

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 20 NC SAT Math Scores by Race Mean SAT Score

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 21 SAT Mathematics Scores By Race/Gender Mean SAT Score

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 22 For More Information... SAT Reports from 1995 to 2007 include information and data for: the Nation the State 117 Public School Systems Charter Schools North Carolina School of the Arts North Carolina School of Science and Mathematics /

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 23 Sample SAT Report Page Some questions to explore: Are districts closer to the state average in Math than in Critical Reading? How about variability among schools within districts? How about participation rates?

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 24 Question of the Day … SAT puts an “official” question of the day on its web site. Excellent way to get a better understanding of what students will be facing. Why not have 8 th graders try them?

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 25 Purposes – Part 2: Green Book Find out more about getting data from the Green Book, … both in its standard form … and its interactive, electronic form.

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook: A Web-tool for Data

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook: A Web-tool for Data Your

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook Equally accurate Equally reliable More timely More accessible

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook geography students display grade year subject

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook 2. performance 1. participation

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook students geography year grade & subject

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook So far … Questions? Answers? 5% / 95% Rule

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION Electronic GreenBook

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 46 Classic Green Book SAMPLE PAGES FOLLOWING ….. Finding average scale scores for LEAs. Finding standard deviations for EOC and EOG scale scores. ….. Which leads in to … PART 3

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 47 Purposes – Part 3: THE LENS Discuss effect size, a useful tool for a fuller understanding of LEA or school performance across grade and time. Compute some math effect sizes and highlight where the information is obtained. Show the relationship between LEA math performance on SAT, EOC, and EOG.

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 48 The Importance of “Where.” ‘A school (or LEA) is data driven wise if everyone knows where they are and where they are going.” … Donna Snodgrass, et al.

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 49 Where is this LEA? Math ABC Results for (Percent at or above grade level) EOG Grade 5 … (state 66.7) EOG Grade 8 … (state 65.1) EOC Geometry …45.0 (state 63.0) Pass rates and gaps in pass rates leave out a big part of the picture.

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 50 The Average is above Average The percent above a cut score counts a score the same whether it is just above (below) the cut or far above (below) the cut. BUT the average includes all the score information. Year-to-year changes in percent above are affected by where on the distribution the cut is located. BUT the average follows the center of scores.

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 51 Then …Where is this LEA? Math Score Averages for 2007: EOG Grade 5 … (State 354.7) EOG Grade 8 … (State 360.5) EOC Geom. … (State 151.3) SAT … 493 (State 509) SAT % Tested … 72.8 (State 71.0) That’s still difficult to answer with different tests on different scales.

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 52 Effect Size takes care of Differing Scales Example: Calculate the effect size for math SAT: Subtract the overall average from the local average: = -16 Divide by the Standard Deviation: -16 / 110 = -0.1 (rounded to 10ths)

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 53 So … Where is this LEA? Math Effect Sizes for 2007: EOG Grade 5 … EOG Grade 8 … EOC Geometry … -0.4 Average SAT SAT is very close to the state. Grade 5 is low but still close. Geometry and Grade 8 are much lower.

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 54 Questions to pose.. In your LEA (or school) which grades, courses, SAT results are stronger? In your LEA how do schools compare on average EOG, EOC, SAT? How does your LEA (school) compare with similar LEAs (schools)? What is the effect of what you’ve been trying to improve?

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 55 NC math’s great progress NAEP results from 1992 to 2005 Compared with the U.S. average and with other states, North Carolina went from back of the pack to ahead of the pack. NC was somewhat sluggish 2003 to For the NAEP data in the following slides, go to:

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 56 NAEP Scale Score Averages – A Decade of Great Progress Grade 8 Grade 4 North Carolina U.S. Average North Carolina U.S. Average