1. School Improvement and Educational Accountability M. David Miller University of Florida

2. A major shortcoming of current accountability systems for purposes of making valid inferences about school quality is due to severe limitations on the data that are generally included in systems (Linn, 2007)

3. The growing consensus is that in the era of standards based reform, a major missing piece of the puzzle is data. Every state must quickly develop a robust system of student data and information that allows us to not only report on assessment data for state and NCLB accountability purposes, but to use those data for diagnostic and instructional decision making (Wilhoit, 2006)

4. Accountability Systems and Multiple Uses  NCLB / AYP  Individual Uses Diagnostic/ Instructional Diagnostic/ Instructional Graduation Graduation Retention/Promotion Retention/Promotion Course Credit Course Credit

5. School Use of Assessments  Designs for School (e.g., matrix sampling) fewer items per student fewer items per student broader coverage broader coverage noncomparable student level scores noncomparable student level scores reduced testing time reduced testing time

6. Diagnostic Testing  Student v Group level  Level of scores - Subscores v Benchmark  Reliability  Validity

7. Florida Content Focus Clusters Content Area Content Focus Cluster Reading Words and phrases in context Main idea, plot, and purpose Comparisons and cause/effect Reference and research Mathematics Number sense, concepts and operations Measurement Geometry and spatial sense Algebraic thinking Data analysis and probability Science Physical and chemical sciences Earth and Space sciences Life and environmental sciences Scientific thinking

8. Spearman-Brown projected subscore reliabilities Subscores ρ=.85 ρ=.90 ρ=.95 3.65.75.86 4.59.69.83 5.53.64.79

9. Reading benchmarks – Main idea, plot, and purpose  Chronological order  Details and Facts  Author’s Purpose  Character Development  Plot Development

10. Mathematics benchmarks – Algebraic Thinking  Functions  Graphic Patterns  Linear Equations  Solving Equations*  Equations*  Inequalities  Solving Inequalities  Solving Equations  Translating Inequalities

11. Science benchmarks – Scientific Thinking  Experimental Design  Drawing Conclusions*  Observation  Communicating Results  Understanding Models  Predictable Events  Using Machines  Problem Solving  Scientific Applications

12. Spearman-Brown projected benchmark reliabilities Benchmarks ρ=.85 ρ=.90 ρ=.95 15.27.36.56 25.18.26.43 40.12.18.32

13. Spearman-Brown projected factors for test length with benchmark reliabilities=.70 Benchmarks ρ=.85 ρ=.90 ρ=.95 156.313.811.83 2510.636.643.09 4017.1110.634.96

14. Diagnostic Validity and Test Construction  Adding items that fit overall test blueprint and information function  Dropping items that do not fit overall test

15. Additional Indicators to Interpret Accountability Data (Conditions of Education)  Participation  Student Effort and Progress student attitudes and aspirations student attitudes and aspirations student effort – absenteeism student effort – absenteeism  Contexts of Education course taking and standards course taking and standards learning opportunities learning opportunities special programs special programs teachers teachers school characteristics and climate school characteristics and climate

16. Successful and Unsuccessful Schools In current accountability systems, student intake and instructional effectiveness are confounded to some unknown degree, calling into question any inferences about school effectiveness from these data (Raudenbush, 2004)

17. Summary  More information needed to understand causes of success or lack of success  Value-added adjustments?  Diagnostic information reasonable in this context given issues of reliability and validity ?