1 Outcome Measures for School Evaluation Coalition for Excellence in Science and Math Education

2 Outcome measures fall into two distinct types: Growth and Performance Performance displays how closely a school’s achievement compares to a standard Growth displays the change of a school’s achievement relative to other schools Both give valuable information Attempts to combine the two types by some subjective estimates of relative performance are open to criticism Types of Measures

3 Growth tends to be biased in favor of schools starting from low initial performance. The relationship is not strong, but is significant

4 Growth in New Mexico schools, as the annual change of scale scores tends to be chaotic. A sharp rise in one year is often followed by a steep decline in the next year. There are methods of smoothing the irregularity, by averaging or by ordinary least squares trends, as here,

5 Sources of Influence Most of the important demographic variables as well as the important performance variables, are mutually correlated – a condition known as “Multi-colinearity” With correlated variables, the total influence of all the variables together can be quite easily determined, but the effect of individual variables is more complicated In the next two slides a transformation has been applied that yields completely uncorrelated variables that have the same total effect as the correlated variables, and correctly shows the individual contribution of each. In some circumstances we really want to know the contribution of two or more variables acting together. Another method finds this, and also part of each that works independently of the other. If you see a legend on a pie chart for one slice that says something like “A and B Together”, It has been done to display the effect of the correlation.

6 The current score is strongly associated with the previous performance, and to a lesser but not negligible extent, with school demographics

7 Growth, as annual change in score, is almost independent of school demographics. The association of “Growth” with prior score is less striking than the association of “Current Score” with prior score.

8 EFFECTS OF TWO VARIABLES ACTING TOGETHER  In reality, demographic variables tend to be strongly correlated  The influence of the correlation can be far more important than the effects of that part of each variable that is not correlated with the other  Minority status and poverty together are far more potent than either alone

9 The next slide will show the fractional increase in canonical combined score variance explained by current mean scale scores, average growth of mean scale scores, growth of scores at the 25 th percentile and growth of scores at the 75 th percentile are progressively added to the ensemble of performance variables. Details vary widely with year, grade level student subgroups, but invariably, only a few of the many possible performance variables are needed to completely characterize school performance. In many cases, scale scores alone are adequate. Multiple demographic variables behave similarly Unnecessary variables increase the chance of human or machine errors.

10 Because of multi-colinearity, the total contributions of entire groups of variables are correctly shown, but the individual contributions of variables within the group may not be correct.

11 Implications Growth in NM schools tends to be chaotic, and might be questionable as a major indicator Growth vs, time can be smoothed by any of several methods A broadly accepted smoothing method needs to be established Prior performance is a strong predictor of current performance and is a moderate predictor of recent growth School demographics is a modest predictor of current performance, but not of growth There is no need for many outcome measures. Usually 4 or 5 are adequate to display a complete picture of school standing Outcome measures and demographic variables should be analyzed to establish the minmum necessary Additional measures add complexity and chance of error without increasing accuracy