1. Implementing Virginia’s Growth Measure: A Practical Perspective Deborah L. Jonas, Ph.D. Executive Director, Research and Strategic Planning Virginia Department of Education May 2012

2. Potential Uses for Student Growth Percentiles School improvement (including accountability) and program evaluation One component of comprehensive performance evaluation ▫Consistent with the Code of Virginia requirement to incorporate measures of student academic progress in evaluations (§ 22.1-295). ▫Growth percentiles may form the basis of one of multiple measures of student progress, when available and appropriate. Planning for differentiated teacher professional development Communications with students and parents 2

3. Current Policy and Program Initiatives Implementing 2011 Virginia Board of Education Guidelines for Uniform Performance Standards and Evaluation Criteria for teachers statewide, which: ▫includes a comprehensive model for teacher performance evaluation; ▫recommends that evaluations include multiple measures of student academic progress; and ▫recommends that one measure of student academic progress be the state provided student growth percentiles when available and appropriate. Incorporating growth measures into Elementary and Secondary Education Act waiver application. Student growth percentile training for administrators. 3

4. Overview of our work Began working with nationally recognized experts in calculating student growth percentiles in spring 2010. Discovered that a standard development/modeling approach would not work in Virginia. Developed policy solutions—some of which may be informed by growth data; many can be informed by additional research. 4

5. Today’s topics Varied test progressions in statistical models Imperfect model fit—ceiling effects Changing tests in 2011-2012 Other 5

6. Test progressions Not all students progress through Virginia’s tests using the same pathway, e.g., ▫Grade 6, 7, Algebra I ▫Grade 6, 7, 8, Algebra I This impacts statistical model building and the “peer” or comparison group. States using or moving to end-of-course tests may need to consider how to model unique student progressions through courses and tests—they do have an impact. 6

7. Model fit—Impact of floor and ceiling effects Compromises validity of some statistical estimates.* Important issues: ▫Impact on individual student data. ▫Impact on grouped data (teachers, grade-level schools, districts). Policy should be developed to account for these issues. 7 *See Koedel, C. & Betts, J. (2009). Value-added to What? How a ceiling in the testing instrument influences value- added estimation. http://economics.missouri.edu/working-papers/2008/WP0807_koedel.pdf.http://economics.missouri.edu/working-papers/2008/WP0807_koedel.pdf

8. Concrete example of ceiling effect impact* Scaled Score Grade 3 Scaled Score Grade 4 Scaled Score Grade 5 SGP Grade 5 Student A600 67 Student B554600 80 Student C522600 88 8 Scaled Score Range 200 - 600

9. Concrete example of ceiling effect impact Scaled Score Grade 3 Scaled Score Grade 4 Scaled Score Grade 5 SGP Grade 5 Student A600 67 Student B554600 80 Student C522600 88 9 This is the maximum possible growth score for students with a perfect score for three consecutive years. Growth is still considered high (in Virginia), how high is not clear.

10. Concrete example of ceiling effect impact—a more extreme case Scaled Score Grade 3 Scaled Score Grade 4 Scaled Score Grade 5 SGP Grade 5 Student A600 24 Student B568600 50 Student C522600 59 10

11. Concrete example of impact of ceiling effects—a more extreme case Scaled Score Grade 3 Scaled Score Grade 4 Scaled Score Grade 5 SGP Grade 5 Student A600 24 Student B568600 50 Student C522600 59 11 This is the maximum possible growth score for students with a perfect score for three consecutive years, and may not be a valid measure of student growth. This occurs because the majority of students who earn perfect scores for two consecutive years also earn a perfect score in the third year. NOTE: The impact of significant floor effects would be similar.

12. What if you have floor or ceiling effects in your tests? Statistical adjustments may be possible, depending on the magnitude of the ceiling effects (Koedel & Betts, 2009). ▫Statistical adjustments that may be acceptable for research purposes are not always acceptable for accountability purposes. ▫More statistical work is needed in this area. Policy for Growth Data ▫Virginia established a cut-off for which students will receive and not receive growth data. ▫Resulted in the state providing no data for a substantial percentage of students. Policies related to growth data cut-offs will be re-evaluated when new tests are in place. 12 Koedel, C. & Betts, J. (2009). Value-added to What? How a ceiling in the testing instrument influences value-added estimation. http://economics.missouri.edu/working-papers/2008/WP0807_koedel.pdf.http://economics.missouri.edu/working-papers/2008/WP0807_koedel.pdf

13. Changing tests and growth data Virginia has not adopted the Common Core State Standards ▫Implemented new mathematics assessments this year, 2011-2012. ▫New English assessments will be implemented in 2012-2013. New tests complicate growth model development, training, interpretation, and timing of various use-cases. 13

14. Considerations when tests change Growth-to-Standard calculations ▫It is not clear how many years of data are needed to measure growth-to- standard reliably. ▫The amount of data needed depends at least in part on the magnitude of changes in student outcomes each year. Use of growth data in school and educator accountability ▫Virginia has not included growth-to-standard data in proposed school or educator accountability due to changing tests. ▫Statewide training is focused on data interpretation and use of multiple measures to support teachers work to ensure students can meet new and higher expectations. ▫Regularly analyzing data and considering policy and programmatic implications. Public information becomes an even greater challenge after tests change. 14 Stay tuned……there’s more to learn!

15. A few other considerations Lack of growth data for students assessed with alternative tests and the impact on use in evaluation and accountability. Impact of floor/ceiling effects on other data derived from state assessments (e.g., Lexile ® and Quantile ® data). Public information requests for data and Freedom of Information Act/privacy laws and regulations. Data interpretations when growth is calculated for assessments when more than one year is between last and most recent tests. 15 Lexile ® and Quantile ® are trademarks of MetaMetrics, Inc., and are registered in the United States and abroad. Copyright © 2012 MetaMetrics, Inc.

16. 16 Contact Information Deborah L. Jonas, Ph.D. Executive Director for Research and Strategic Planning Virginia Department of Education Deborah.Jonas@doe.virginia.gov