UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing Practical Considerations for Choosing an Accountability Model Pete Goldschmidt American Educational Research Association Annual Meeting San Francisco, CA - April 7-11, 2006 If you choose to use this title slide, simply delete the previous slide (the one-line title version). This will be slide 1 of your presentation.

2/28 Practical considerations - 1 Need to decide what the accountability system is trying to do. Accountability models should hold schools accountable for those things that schools are responsible. Generally we consider those things to be student outcomes. Outcomes can be multi-faceted, but emphasis is on academic performance. Academic performance usually limited to a few subjects. Academic performance usually measured by a large scale assessment. When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

3/28 Accountability Means That We Intend to Hold Schools Responsible for Student Outcomes Student outcomes accumulate over time. Student achievement in grade 4 is a function of achievement in grade 3 and grade 2 (etc) as well as school processes, family background, innate ability, peers, luck, and error. (Hanusheck, 1979). To get an unbiased estimate the effect of school processes, would need a measure of ability prior to any schooling. Unlikely to have those data, but to reduce bias estimate marginal effect of schooling on the change in achievement from one grade to the next (fixed info incorporated into previous test score). By only looking at incremental change reduce bias from not having true a priori measures. When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

4/28 Practical considerations - 2 An accountability model should only be based on results that reflect the effects of internal factors (factors schools control). Simple aggregate static measures of student performance judge schools based on both internal and external factors – but are overly influenced by external factors. When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

5/28 Concerns - 1 Static models assumes current achievement in solely a function of current schooling processes. Aggregating individual student variables inflates their importance – correlations between aggregate performance and school enrollment characteristics about.75. When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

6/28 Directly Comparing the Relationships Among Indicators Reveals… To adjust the slide numbering, do the following: 1.Go to the VIEW menu, MASTER, and select SLIDE MASTER 2.In the lower right, change the number 28 to your number of slides 3.Do not change the character. It generates the auto-numbers.

7/28 Concerns - AYP is a Simple Aggregate Static Measure AYP, by construction also penalizes large (and) heterogeneous schools. AYP as an accountability model can categorize schools, but does so very imprecisely. Policy makers want the accountability model to provide more than simply a categorization (and at least the correct categorization) but potentially also use the results to inform school improvement. It ignores the fact that the accumulation of both external and internal factors over time affect current student performance. Although P = progress, no inferences regarding progress can be made as AYP does is heavily influenced by external factors – factors that are outside of school’s control. When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

8/28 Validating AYP Validate AYP results with additional accountability model – good idea Too often focus is on choosing the “right” model – as decided by: identifying fewer schools as not making AYP AYP results do not match growth or value added results. The less an accountability model relies on static aggregate information to rank schools, the less its results will match AYP – and the more likely it will be that school traditionally thought of a being good, will not look as favorable When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

9/28 Growth and value added models address different questions (states should explicitly address this rather than have multiple sets of results that may confuse stakeholders) e.g. If you are interested in how everyone at the school did this year. If you are interested in how this cohort of students is dong this year. If you are interested in how students improved from last year. If you are interested in how second grade (3rd etc) improved. Achievement gaps. How will we get to 100% proficiency? When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT” Policy – What should results address?

10/28 Concerns - 3 Accountability Model Will Not Facilitate Meeting Ultimate Goal in No model can ensure that a state will meet goals e.g. we may have accounting rules in business, and the goal of every business – say Delta, for example, aims to make a profit, but simply setting a trajectory for growth in revenue (and decrease in costs) by year does not guarantee it will happen When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

11/28 Practical considerations – moving beyond AYP - Data Is more data better? Yes Allows for longer growth trajectories. Generate more precise estimates of growth. Reduces effect of initial status on growth (or picks up that initial status is not finitely determinant of future.

12/28 Data But, More data, more missingness (although it is beneficial to tradeoff additional missingness for more occasions). Time span may not be relevant for grade level. Models using more occasions are more consistent than models using fewer, but models of different occasions are only moderately correlated.

13/28 Data and policy questions Expectations. Accounting for external factors. Gaps and changes in gaps. Interested in cohort to cohort improvement. Cohort models have less stringent data requirements than panel models (e.g. vertical scaling). Cohort results may be confounded with external factors (e.g. changes in student background). Interested in individual student growth. Need to consider metric and its interpretation. More likely to address confounding issues.

14/28 Practical way to combine results? High Growth Second cut First cut Low Growth First cut Not Met AYPMet AYP

15/28 Considering the relationship Note: Schools rank by different criteria. Status (x-axis): A > B > E > C > D; Gain (y-axis): B > C > A > E > D; Conditional gain (regression line): B > C > D > E > A. The vertical line and the horizontal line represent district average initial status and district average gain, respectively.

16/28 Model subgroups directly (less data) Focusing on achievement gaps and the likelihood of meeting the target in and utilizing an accountability model not intended for evaluation. Can use a longitudinal binomial growth model that simply models the probability over time that a subgroup will be proficient. Does not require a vertically equated metric. Provides a clear picture of current status. Provides a direct estimate of progress over time. Demonstrates where subgroups are and are going.

17/28 Data Structure by subgroups Year 1GirlBoy Low SES215/245234/257 Not Low SES300/345300/330 Year 2GirlBoy Low SES220/249230/260 Not Low SES304/351304/326 Year 3GirlBoy Low SES215/232243/260 Not Low SES3006/ /330

18/28 Binomial longitudinal model

19/28 Binomial longitudinal model

20/28

21/28 Decomposing changes in school performance (more data) Cohort and individual student performance. Change Cohort Performance. Change in individual student performance.

22/28 Cohorts year 1 (less data)

23/28 Cohorts year 2

24/28 Panel and cohorts

25/28 Longitudinal Cohort & Panel Growth Model (more data).

26/28 Relationship between initial status, cohort, and individual student growth

27/28 Practical applications Some capacity and assumed stakeholder understanding affects state growth model proposals Regression based growth models Value tables Percent of expected growth Percent achieving a year’s growth All accountability models depend on value – some models explicitly assign value to results (although this can be somewhat arbitrary, e.g. year’s growth, expected growth, or points for changing categories) When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

28/28 Discussion A aggregate measure leads to an ecological fallacy. Static measures ignore accumulated effects on performance over time. AYP is an aggregate static measure. Cohort improvement models address school improvement but not directly student growth. Panel growth models follow individual students. How handle external factors? Use initial status Use student background Time frame No model assures meeting the target and growth and assuming many factors remain constant – still unlikely a majority of schools will reach 100% proficiency in

29/28 next presentation Pete Goldschmidt voice If you choose to use this end slide, simply delete the previous slide (with no contact information). ©2006 Regents of the University of California