1. LISA A. KELLER UNIVERSITY OF MASSACHUSETTS AMHERST Statistical Issues in Growth Modeling

2. Background We are all well aware of the need to measure growth under today’s accountability framework Growth measurements are used for a variety of purposes Monitoring :  student performance  teacher performance  School performance

3. Background To accomplish these goals, a variety of growth models have been developed and are in use You will hear about them today Unfortunately, the rush to get these models into use has resulted in using models in ways for which they have not been validated (in some cases)

4. Purpose The purpose of my presentation today is to discuss some of the statistical issues inherent in growth modeling, and a couple practical ones too  [As an aside, my goal is to keep it fairly understandable, too!]

5. The Term “Growth” There are many growth models in use today and they are all called “Growth” models Each model is defining “growth” in a different way Some of the models are not even measuring what is commonly thought of as growth This imprecision of language can lead to misleading interpretations of the results of growth models

6. The Term “Growth” When models are developed, care should be taken to ensure that the users of the models are clear about what information the model provides Although the names of the models are usually somewhat descriptive, the models are collectively referred to as growth models, which can be misleading

7. Examples Student Growth Percentile:  “Growth” is defined as how a student’s performance compares to an academic peer group  For example, a SGP of 0.75 indicates that this year, a student performed better than 75 percent of students that performed similarly to this student in the past.  Does not provide information regarding progress relative to curriculum

8. Examples Value Tables:  Growth is defined as changes in student’s performance classification between administrations  For example, last year a student was basic, this year the student was proficient. This student exhibited growth  The value of the growth is determined by the state  This measure provides information regarding student’s progress relative to curriculum

9. The Term Growth Clearly growth means something different in these two models, but both are called growth models. It is important to be careful to define what is meant by growth

10. Choosing a Model The choice of model should align with the questions you want to answer with the growth information. Be sure to pick a model that defines growth in a way that is meaningful for your goals.

11. Vertical Scale Many growth models require that tests are on a vertical scale A vertical scale means that the scores from different grades are comparable While a vertical scale seems like an obvious choice, they are not easy to construct, and are even more difficult to maintain

12. Vertical Scale One of the fundamental difficulties in developing a vertical scale is the existence of a constant construct that spans multiple grades For example: Is Grade 4 Math the same construct as Grade 5 math. If we administer the same item to a 4 th and 5 th grader, does it mean the same thing?

13. Vertical Scale Creating a vertical scale requires establishing a unidimensional scale across grades 3-8. This is the statistical version of the previous problem. This is an empirical question: does the dimensionality of the construct change across grades? If so, creation of a vertical scale is suspect.

14. Vertical Scale Even if we can produce a meaningful scale, scale drift is difficult to manage  Scales drift with changes in curriculum/instruction, exposure of test items, changes in student population This issue is magnified in the cross-grade scale As the number of links increases, so does the magnitude of error

15. Missing Data Regardless of model used, multiple years of assessment data are required for each student This creates a lot of missing data for students who move Decisions of how to handle the missing data are required

16. Missing Data Do you impute data for missing values? Do you ignore students who don’t have complete data? Do you choose a model that is not affected by missing data? In any case, the results will be different. Which results are (more) correct?

17. Background Variables? Most growth models do not include student background variables for policy reasons From a statistical point of view, the accuracy of the model might be improved by including background information Important to see what effect including/excluding background variables might have on the results in your state

18. Error Bands Scale scores on tests often include error bands to indicate likely scores a student would obtain given a repeated test administration Stakeholders can understand this concept, and much good work has been done in this area Provides important information regarding the affect of measurement error

19. Error Bands Growth scores are often reported without error bands, even next to scale scores reported with error bands What is the message? Growth scores are free from error? We don’t know the amount of error? We don’t want YOU to know the amount of error?

20. Error Bands The inclusion of error bands is essential for policy makers to understand the stability of the measures that are provided High stakes decisions are often linked to growth measures (e.g. teacher effectiveness) We should demand error bands on growth measures just as with scale scores

21. Complexity vs. Simplicity Choosing a growth model is difficult Complex models are often more able to accurately capture the complexity of the nature of educational data  For example, hierarchical models are sometimes sought since they can model the nested nature of the data

22. Complexity vs. Simplicity As a result, the complex models might be more statistically sound The disadvantage is that the stakeholders might not always understand the model  This is an issue IF stakeholders are not trained to interpret and use results appropriately

23. Complexity vs. Simplicity Simpler models often require simplifying assumptions regarding the nature of the data* Following previous example, the nesting of data within classrooms/schools/districts might be ignored Advantage: Stakeholders usually can understand and interpret the results easily  * note: this is not always the case

24. Conclusions Although there are many statistical issues inherent in growth models, we need to use them. No model is perfect, but be sure: You know what growth means You understand the amount of error in the measure used You train stakeholders to use and interpret results appropriately