1. Determining Validity For Oklahoma’s Educational Accountability System Prepared for the American Educational Research Association (AERA) Oklahoma State Department of Education April 15, 2004

2. Education Perspective for Oklahoma, 2002-2003 All Local Education Agencies (LEAs) in the State of Oklahoma participate in Title I programs. There are 541 total LEAs 44 LEAs with a student enrollment of 2,500 or greater 201 LEAs with a student enrollment between 499 and 2,500 296 LEAs with a student enrollment of less than 500

3. Oklahoma Demographics, 2003 Total school districts: 541 Independent (K-12): 430 Elementary (K-8): 111 Total School Sites: 1791 Teachers: 47,259

4. Oklahoma Demographics, 2003 Average Daily Membership: 618,358 Special Education (K-12): 13.6%83,812 Alternative Education (K-12): 2.5%15,312

5. Oklahoma Demographics, 2003 American Indian/Alaskan: 17.9% Asian/Pacific Islander: 1.5% Black/Non-Hispanic: 10.9% Hispanic: 7.0% White Non-Hispanic/Other: 62.6%

6. State Standards, Assessments & Accountability Timeline 1990 - Oklahoma Education Reform and Funding Act of 1990 1991 - Initial development of core curriculum with statutory requirement for review and revision every 3 years

7. 1992 - Legislation mandated the development of Criterion-Referenced Tests (CRTs) in seven (7) content areas for Grades 5, 8, and 11 1993- First review and revision of core curriculum, Priority Academic Student Skills (PASS) 1995- Implementation of CRTs in Grades 5, 8, and 11 State Standards, Assessments & Accountability Timeline

8. PASS revisions – 1997, 2000, 2003 Revisions have provided more specific, detailed and clear standards PASS includes the Oklahoma School Testing Program (OSTP) “blueprints” for the CRTs State Standards, Assessments & Accountability Timeline

9. 1989 - Low-performing, high challenge schools identification process adopted in state law; includes sanctions and technical assistance 1999 - Academic Performance Index (API) adopted in state law; includes API Awards State Standards, Assessments & Accountability Timeline

10. Oklahoma’s Accountability System To guide instruction To measure achievement To promote a climate of change Proficiency for all Content and Performance Standards Assessments Additional Indicators Decision Rules Sanctions and Rewards

11. Validity Questions for Oklahoma’s Accountability System Do our standards help guide instruction? Do our assessments and performance indicators measure performance standards (i.e. achievement)? Do our decisions rules accurately identify schools? Do our sanctions and rewards promote change towards intended outcomes?

12. Key Elements in Determining Validity and Reliability 1.Ensure that the system has provided the intended outcomes. 2.Conduct research on additional information to corroborate findings. 3.Analyze design and implementation of each component of the system. 4.Conduct Analysis on several levels.

13. Sources of External Evidence Studies and research using additional indicators to corroborate findings or results of the system. Surveys on attitudes and opinions. Outside reviews for components of system.

14. Sources of Internal Evidence Ongoing review of policies and procedures. Data analysis for various levels such as State level, district level, and for particular types of schools. Evidence of Quality Control measures and Data Audits. Trend analysis over time.

15. PASS - external reviews have included: 2001-02: Achieve, Inc. 2001-02: Southwest Educational Development Laboratory (SEDL) 2001-02: Technical Issues in Large Scale Assessment (TILSA) 1997-2003: “Quality Counts” report cards by Education Week Oklahoma’s Steps Towards Validation Studies and Reviews

16. Studies and Reviews TILSA Review: Curriculum and Assessment Alignment Study (2001-02) with Dr. Norman Webb, University of Wisconsin Results of the study enabled Oklahoma to conduct more in-depth alignment of standards and assessments Provided common language to facilitate communication in building curriculum and designing aligned assessments

17. Research Study (2002-03) by Dr. John Poggio, University of Kansas Study to determine levels of cognitive complexity of multiple choice test items Studies and Reviews

18. Implementation Survey (January 2002) After initial implementation of the API an attitude and opinion survey was collected. Additional surveys specifically on Adequate Yearly Progress (AYP) will be collected in October 2004. Studies and Reviews

19. Study on School Improvement Schools –Collecting additional local LEA assessments from additional grades. Purpose: –To corroborate the results of the decision rules. –To look at trends of schools being identified as school improvement. Studies and Reviews Ongoing Study

20. Grants and Projects No Child Left Behind (NCLB) Enhanced Assessment Grant Oklahoma selected as lead state in collaborative curriculum/assessment alignment project with Council of Chief State School Officers (2003) Continues the alignment work done in 2001 Includes vertical alignment of curriculum and assessments (Grades 3-8) Includes alignment of special education alternate assessments

21. Grants and Projects NCLB English Language Development Enhanced Assessment Grant - 2003 Collaborative multi-state project to develop assessment materials for English Language Proficiency Tests

22. Grants and Projects Project with Council of Chief State School Officers (SCASS) - 2003 Develop survey tools to determine the level of implementation of a state’s standards-based language arts curriculum To provide support for teachers in meeting NCLB requirement to increase student achievement

23. Policies A few examples of changes or development as a result of reviewing the system. Developed policy for appeals of Adequate yearly progress. Developed a Technical Advisory Committee to review and offer expertise in regards to technical issues of the State testing program. Implemented new rules and regulations regarding data audits and security.

24. AYP Statistics 367 (20.5%) of Oklahoma Schools did not make AYP. 198 (36.5%) of Oklahoma School Districts did not make AYP. 1.7 is the average number of subgroups that did not make AYP by school. 1.5 is the average number of subgroups that did not make AYP by district.

25. AYP Statistics Total number of subgroups not meeting specific performance targets, ordered from greatest to least. –Reading target –Math target –Additional Indicator (attendance or graduation) –Participation target

26. AYP Statistics Total number of subgroups not meeting specific performance targets, ordered from greatest to least. –Reading target –Math target –Additional Indicator (attendance or graduation) –Participation target