1. 1 California's API: Recent Developments and Future Prospects Edward H. Haertel Stanford University School of Education CRESST Conference University of California at Los Angeles September 10, 2004

2. 2 The "API" is California's Academic Performance Index School-level summary of student test performance Mandated by Public Schools Accountability Act of 1999 (PSAA) Foundation of California's (pre-NCLB) public-school accountability system

3. 3 Plan for This Morning Calculating the API Coping with change Lessons learned

4. 4 Calculating the API

5. 5 Calculating the API in 1999 It used to be so simple... Index based just on SAT-9 scores Elem/Middle: Reading, Language Arts, Spelling, Math High School: Reading, Language Arts, Math, Science, Social Studies

6. 6 Subject Area Weights Reading 30% Lang Arts 15% Spelling 15% Math 40% Reading 20% Lang Arts 20% Math 20% Science 20% Soc St 20% Elem/MiddleHigh School

7. 7 Precursors to Standards-Based Performance Levels 1-19 20-39 40-59 60-79 80-99 200 500 700 875 1000 SAT-9 National Percentile Rank "Progressive Weighting Factor"

8. 8 Performance Levels on California Standards Tests (CSTs) Far Below Basic Below Basic Basic Proficient Advanced 200 500 700 875 1000 CST Performance Level "Weighting Factor"

9. 9 Growth Targets Statewide Perf. Target = 800 Annual Growth Target = 5% of distance from Base-Year API to 800 Initially, schools at/above 800 need only remain at that level

10. 10 Numerically Significant Subgroups "Comparable Improvement" require- ment mandated by PSAA of 1999 Groups defined by Race/Ethnicity and by socioeconomic disadvantage Student in soc. dis. group if either (1) eligible for free/reduced price meal program or (2) highest parent educational level is "not a high school graduate."

11. 11 Defining "Numerically Significant" How large must a subgroup be to count as numerically significant? Fewer than 30 students not num. sig. 30-99 students num. sig. only if subgroup constitutes at least 15% of total enrollment 100 or more students are num. sig. For purposes of API growth calculation, subgroup must be num. sig. in both base year and growth year

12. 12 Defining "Comparable Improvement" Growth target is defined in terms of difference between subgroup's score in base year and in growth year Subgroup growth target is 80% of schoolwide target

13. 13 Coping with Change

14. 14 "IF YOU WANT TO MEASURE CHANGE, DON'T CHANGE THE MEASURE." --Al Beaton

15. 15 Base API and Growth API May... Oct... Feb... May... Oct... Feb... testing Growth Base testing Growth Base API API API API Release Release Release Release School Year testing cycle Biannual Accountability Cycle

16. 16 The "Scale Calibration Factor" The SCF is a constant added to both the Base API and the Growth API within an accountability cycle. It is chosen to make the mean Base API for that cycle equal to the mean Growth API for the previous cycle. There are new SCFs each year for Elementary schools, Middle schools, and High schools

17. 17 The "Scale Calibration Factor" May... Oct... Feb... May... Oct... Feb... testing Growth Base testing Growth Base API API API API Release Release Release Release School Year testing cycle Adjusted to equal state-level means via SCF

18. 18 Changes in the Measure Changes in the tests included Changes to accommodate heterogeneity of California schools Evolving decision rules for special situations Changes due to NCLB Act of 2001 Refinements in data acquisition/ quality control

19. 19 Changes in the Tests Included 1999-00SAT-9 only 2000-01SAT-9 only 2001-02add California Standards Test (CST) in ELA 2002-03add CST-Math (G2-11); add CST-Hist/SocSt (G10- 11); drop SAT-9 Soc St; add CAHSEE G9-10

20. 20 Changes in the Tests Included 2003-04add CST-Science (G9-11); incorporate CAPA; CAHSEE now G10-11; add CAT-6; drop SAT-9 2004-05??? (2004-05 Base API to be posted March 2005)

21. 21 Changes in the Tests Included Future changes will include: reducing CAT-6 to brief survey at just two or three grade levels adding CST-Science (G5) adding cumulative Hist/Soc Sci (G8) expansion of CAHSEE to include G11, G12 retests... ?

22. 22 Heterogeneity of California Schools Schools that cut across usual Elem/Middle/High boundaries Small and very small schools Alternative School Accountability Model (ASAM) schools

23. 23 Most Common Grade Spans for California Schools Data from "Accountability Update – Spring 2004" at http://www.cde.ca.gov/ta/ac/ay/

24. 24 Small (and Very Small) and ASAM Schools First, schools with N < 100 did not receive API Then, schools with N between 11 and 99 received API with an asterisk Now, due to inclusion of API as alternative indicator in California's NCLB accountability, all schools must receive API, including ASAM schools and those with N < 11

25. 25 Evolving Decision Rules denominator (total enrollment) for testing participation rate calculation Parent opt-outs students tested with Testing accommodations Testing modifications California Alternate Performance Assessment (CAPA)

26. 26 Evolving Decision Rules Testing irregularities Student testing irregularities Adult testing irregularities Decision rules for deciding which has occurred Appeals processes Growth calculations when API is invalidated for one year

27. 27 Evolving Decision Rules Mobility Exclusions Students new to district Students new to school Special rules for "feeder school" patterns Non-universal indicators High School Students not enrolled in a math or science course Changing rules for out-of-level testing

28. 28 Evolving Decision Rules Partial records Counting students taking some but not all subtests G4 or G7 writing tests with no CST-ELA blank test forms Students taking high school math or science courses earlier/later than standard grade sequence

29. 29 Impact of NCLB...... has been massive, requiring changes in what subgroups are included, how participation rate is calculated (exclusions, averaging), what schools get APIs, what tests are given at each grade, rules for out-of-level testing, data release timelines, design of score reports, treatment of students designated proficient on CAPA,...

30. 30 Data Acquisition/Quality Control School-level analog to standardized testing conditions at student level Requires looking at API as product of complex system, dependent on Government officials, State Board of Ed Actors in school and district offices Testing contractors Teachers

31. 31 Data Acquisition/Quality Control Data sources have changed over time for Total school enrollment numbers Free/Reduced Price Meal Participation numbers New "2004 STAR Pre-Identification Data Review"

32. 32 Lessons Learned

33. 33 Lessons Learned Long-term stability is probably unattainable Complexity increases, and increases, and increases Technical and policy considerations often point to divergent courses of action

34. 34 Lessons Learned Seeing Like a State* Fairness is a matter of degree Comparison of API (successive cohort) vs. matched individual growth indices Inequities in accounting for educational challenge Paradoxical effects *Title of 1998 book by James C. Scott

35. 35 Lessons Learned With hard work, a system can be created that is trusted and respected, despite its imperfections By and large, scores improved steadily over the entire time California used the SAT-9* An enormous "behind the scenes" effort is required to maintain a reliable and valid accountability system *See David Rogosa's paper, "Four-peat: Data Analysis Results from Uncharacteristic Continuity in California Student Testing Programs" at http://www.cde.ca.gov/ta/ac/ap/researchreports.asp

36. 36 What Can Policy Makers Do? Create and use a stable technical advisory committee Plan ahead, but stay flexible Resist pressures to use tests for new and different purposes Remember that the school-level index is only part of an accountability system and only part of an education indicator system