The Meaning of Change over Cohorts Mark D. Reckase Michigan State University

An Amazing Data Set It would seem that everyone should be interested in the amount of increase in achievement results from the educational system. For example, how much does the average student increase in achievement from one year of instruction? How much do these gains change over years of instruction? It is amazing how little data is available to answer these questions. EQAO has created an absolutely wonderful data set.

Enhancements to this Study Although I like the study, I think that there are a number of pieces of information that are missing. The distributions should be presented at each grade level along with means and standard deviations and the proportion in each category. The correlations between scores at each grade level are important and should be reported.

Other Enhancements The characteristics of the tests should be described. The process for placing students into Academic and Applied are needed. It is interesting that about 1/3 of students are applied. How is it determined that there should be that many and how are they identified?

The Meaning of the Standard The standard used in the study is roughly equivalent to the Proficient standard used in the U.S. There is an additional complication that there are French and English tests for Academic and Applied. This is a more complex system than we see in the U.S.

Marginal Distributions The presentation shows the percents in many subcategories, but I would like to see the proportions passing. I have computed them from the data. Grade369 Percent Exceeding Academic Percent Exceeding Applied

The Marginals Make a Lot of Difference It is not possible to get a lot of students successful over the years if there are not many successful in one of the years. Academic must have more of those than Applied Note the increase in the exceeding group for Applied at Grade 9. This sets up a particular pattern.

Misleading Slide

There are fewer Applied than Academic so the bars must be lower in general for Applied. But, note the Never Met Standard Group. Having the percents add to 100% for each group is difficult to interpret.

Pathways The percent of students by Pathways are much more interesting and the percent across categories is a good way to go. However, I would rather see the results compared to chance results from the marginals.

Pathways -- Applied

Pathways – Applied Compared to Chance Result 369ObservedChance

Pathways -- Applied Persistent Difficulty is a problem and the 33% is higher than expected by chance marginals. Sustained Strength seems small at 13%, but it is better than predicted by the marginals of 4%. New Strength is misleadingly high because it is less than predicted by the marginals.

Pathways -- Academic

Pathways – Academic Compared to Chance Result 369ObservedChance

Pathways -- Academic The good news is that Sustained Strength is better than predicted by the marginals. The bad news is that Persistent Difficulty is as well. This is consistent with correlations in performance over years. Returning Strength is lower than would be expected by chance as are the other strength categories.

Pathways There is more going on in these data then is immediately apparent. More detailed analyses my reveal even more than my simple attempts. The same is true for gender and SEN.

Attitudes There does not seem to be much of an interaction of attitudes by group. It is interesting that only about 30% like mathematics or think they are good at it. I dont know if this is good or bad. It would be useful to have longitudinal data on these questions. I hope they will continue to be collected.

Final Comments This is a powerful data set. It would be possible to spend years doing research on it to get all of the information that is there. It is not a vertically scaled data set so the analyses done here use social moderation. This limits the result somewhat. These initial results are just wetting my appetite for more.