1. Advanced Quantitative Research ED 602

2. You know, Mary Stevens has really blossomed this year. She is doing much better. Actually, this whole fifth grade group is a great group academically. It would be wonderful if I could be confident enough that those statements were true that I can report them to the school board. I’ve never been to a school Christmas party before where they served martinis. I vote that Judy hosts the party again next year.

3. What just happened? Bob summarized qualitatively information he had gathered on an individual student and a group. Kay doesn’t know how Bob gathered data so she has to think about how to support those two statements to a degree that she can get others to believe they are true statements.

4. How can Kay proceed? If Kay wants to talk about Mary Stevens (anonymously of course), she can select any number of characteristics of Mary and show growth. She might pick a test score: Mary moved from a 4.7 grade equivalency on some standardized assessment to a 5.5 equivalency in just 6 months. Sounds great. The cool thing about talking about a single student is that you can use any characteristics that you want to “tell Mary’s story.” Name the characteristic and show an exemplar. (Sound familiar.) Accumulate exemplars and tell the story.

5. Evidence We like stories that are convincing. Lots of accumulated exemplars will do the trick. It doesn’t matter if the exemplars are quantitative or qualitative as long as they add up to a good story. Accumulate enough information and people begin to go: “Wow, that is overwhelming evidence.”

6. What will Kay do about showing the whole class is good? If there are 60 students in the class we could do the same thing we did with Mary, 60 times. Tell each student’s story. There might be a few who weren’t doing that well but we could make the case that in general everyone’s story looked pretty good. Does this sound like portfolios?

7. So, let’s get started. With Mary, Kay selected characteristics of Mary’s performance for which there were exemplars. If we looked for the positive characteristics for each of the 60 students we could make 60 individual cases (stories) but then it would be very difficult to support the original premise that the whole class was doing better academically. For someone to know you would have to tell 60 stories.

8. To solve this Kay must do 2 things: Choose the same characteristics to measure for each student. Choose exemplars (measures) of those characteristics that are clearly related to academics.

9. Great! Kind of … Let’s use the same standardized test that Kay used to discuss Mary. Kay now has 60 test score differences. How can she tell anyone about this? She has 60 differences to report to make the case that the class is well academically. Summaries—not raw data. No one wants to know the raw data about a group. No one wants to read 60 stories.

10. Summaries Average (summary) difference in the scores of the pretest and the posttest. How much did the class grow over a 6 month period of time as a group? Now that Kay has the summary difference she has evidence to support what she thinks it means.

11. What do we have so far? One approach to describing an individual is to accumulate evidence around specific characteristics. One approach to describing a group is to accumulate evidence around the same characteristics from each individual and then summarize the evidence.

12. Problems with Group Summary What do you do if the characteristic that you want to summarize for the group isn’t quantitative? Well, the first answer is that you do what you learned to do last semester. Another approach is to figure out how to transform qualitative data into quantitative.

13. Qualitative Transformation Putting qualitative observations into ordered categories. You already know how to do this—rubrics It turns out that qualitative transformation is complicated if you want to use the numbers for anything beyond group description. We need to study this.

14. The Bigger Problem … One the most important uses of summaries of group characteristics is to compare groups. Should be simple … If the summary number for one group is higher or lower than another group then there is a difference between the two groups. Well maybe …

15. What if? If Mrs. Johnson and Mr. Smith taught the same curriculum using the same instructional style to two groups of students who were demographically similar, would you expect the summary of the measures of student learning to be exactly the same in the two classes? Can you think of any circumstance where you would expect the measures to be exactly the same? Why is that? 15

16. What if? What if Mrs. Johnson and Mr. Smith taught differently? Since you already wouldn’t expect the measures of learning to be the same, how would you know if something other than random influences caused the differences? 16

17. Group Differences We solve this problem by asking a single question: Is the difference between the groups so big that it is really unlikely that the difference could have appeared randomly? If the difference is really unlikely to appear randomly then we say it didn’t appear randomly and the difference between the two groups means something.

18. Ok, let’s do this in Excel

19. (from the Excel Files) Inquiry Science 19

20. These are all the same question: Is the difference between the groups so big that it is really unlikely that the difference could have appeared randomly? Could the two groups have appeared randomly within the same population? What is the probability that the group mean differences could have appeared by chance? Is there sufficient evidence to reject the null hypothesis? Is the difference between the groups statistically significant?

21. What Did We Learn? When we want to describe individuals we do this by choosing specific characteristics and finding exemplars. When we want to describe groups we have to use the same characteristic for each individual and then figure out how to summarize the exemplars. When we need to compare groups we use the group summaries to figure out the likelihood of groups being really different.

22. Why Do It? We are thinking about these things so that we can convince others of the veracity of statements we make about the individuals and the groups we are describing. Otherwise it is just unsubstantiated opinion. No one cares. A research secret—this is much harder to do qualitatively than quantitatively.

23. You know, Mary Stevens made a one year gain in 6 months on the MAT test. The class as a whole showed statistically significantly greater improvement than last years class. Maybe we can take these same data and compare over the past 4 years since we implemented the Japanese Lesson Design. If I have another martini I’m pretty sure I won’t be able to say statistically significant.