Experiments

The essential feature of the strategy of experimental research is that you… Compare two or more situations (e.g., schools) that are as similar as you can make them (e.g., % of students who are poor, size of school, quality of teaching staff), but these schools are different in the few factors (e.g., reading materials) whose effects you are trying to find out, or test. If the only main way that the schools differ is the reading materials used, and if one school has consistently higher reading achievement than the others, then it is very likely that the reading materials make the difference. What ELSE could it be? After all….  What ELSE could it be? After all…. 

The factor that makes the difference in reading achievement can’t be class size---because this is the same across all the schools.The factor that makes the difference in reading achievement can’t be class size---because this is the same across all the schools. The factor that makes the difference in reading achievement can’t be teacher quality---because this is the same across all the schools.The factor that makes the difference in reading achievement can’t be teacher quality---because this is the same across all the schools. The factor that makes the difference in reading achievement can’t be the percentage of students who are poor---because this is the same across all the schools.The factor that makes the difference in reading achievement can’t be the percentage of students who are poor---because this is the same across all the schools. Therefore, the factor that makes the difference in reading achievement must be the ONE major way the schools are different----reading curriculum.

Not all experiments give information that you can trust. The validity (accuracy) of the information is weakened by OTHER factors. So, you have to carry out experiments in a way that RULES OUT these other factors. For example… 1. Maybe students in schools with the highest reading achievement were already reading at a higher level. Therefore, these students don’t have to learn very much in order to come out first. In other words, maybe students’ pre-skills (and not the curriculum materials) account for higher achievement. To rule out this possibility, an experiment should have pre- test measures of the variables whose change we are interested in. For instance, we should measure reading skills in all of the schools BEFORE new reading materials are introduced, to see if some classes are ahead of other classes.

So, if an article says that a program or method produced more change or is more effective than another program or method, ask if the researchers did pre-tests to show that students all started at the same level. If they did not do pre-tests, then you cannot trust the claim about effectiveness. The group with the highest achievement may have started at a higher level.

Is the Research Strategy Appropriate for the Research Questions? Experiment 2. How do you know how much students learned? The only way to find out is to do post-tests (e.g., at the end of the semester and year) to see where students’ reading achievement ended up. And you have to see how much students changed (learned, achieved) between the pre- test (reading proficiency at the start) and the post-test (reading achievement at the end). And you need to report how much difference there is between the pre-test and post-test scores. 2. How do you know how much students learned? The only way to find out is to do post-tests (e.g., at the end of the semester and year) to see where students’ reading achievement ended up. And you have to see how much students changed (learned, achieved) between the pre- test (reading proficiency at the start) and the post-test (reading achievement at the end). And you need to report how much difference there is between the pre-test and post-test scores. If researchers do NOT do post-tests, or do not REPORT the size of differences between the pre-test and post-test, then you don’t know HOW MUCH students learned as a result of the curriculum that was used between the pre-test and the post- test. Therefore, you can’t trust claims that one curriculum produced more achievement than the others.

Is the Research Strategy Appropriate for the Research Questions? Experiment 3. Were measures objective? You don’t rely on a physician’s subjective opinion about your health. “You look fine to me.” Maybe the physician is right. But what if the physician’s opinion is wrong? Just so, you want objective evidence of (in this experiment) reading achievement. It is nice to know how much teachers think their students learned, but you can’t judge effectiveness based on opinion. Reading is a real thing. It can be measured objectively---for example, how many words students read correctly per minute.

Is the Research Strategy Appropriate for the Research Questions? Experiment So, if researchers do NOT use objective measures of (for example) reading and math achievement (e.g., counting correct words students read or problems students solve per minute), but instead use the subjective opinions of teachers (“Students improved a great deal.”) then you cannot trust claims about effectiveness----because the researchers did not give evidence of effectiveness. They only gave evidence of how teachers see things. But that was not the question.

4. Were instruments and measurement methods tested for validity (they measure what they are supposed to measure) and reliability (different persons measuring the same thing would come up with about the same scores)? Validity. Do you think a good measure of reading proficiency is whether students can name the parts of a book and can turn the pages? If your school principal is selecting a new reading program, is that the information you want the principal to use? “Students taught with Weird Phonics learned to turn the pages. So, I’ll use that program with 200 students in my school.” I don’t think so.

Turning pages is not what “READING” means. You want to know if a program teaches students to: (1) say the sounds that go with the letters; (2) accurately and quickly read single words; (3) accurately and quickly read passages; (4) learn many new vocabulary words each week; (5) answer both simple and hard comprehension questions about what they read.

So, if researchers use instruments and measures that do not measure what they OUGHT to measure, then the findings have nothing to do with the research question. The question is whether a curriculum teaches students to read. But the findings have to do with turning pages. Invalid measure.The question is whether a curriculum teaches students to read. But the findings have to do with turning pages. Invalid measure. The question is whether a certain teaching procedure increases math achievement. But the findings have to do with how much students enjoyed being taught with the method. Invalid measure.The question is whether a certain teaching procedure increases math achievement. But the findings have to do with how much students enjoyed being taught with the method. Invalid measure. The question is whether peer tutoring is effective at helping students to spell better. But the findings have to do with changes in self-esteem. Invalid measure.The question is whether peer tutoring is effective at helping students to spell better. But the findings have to do with changes in self-esteem. Invalid measure.

Instruments and measures may measure what they are supposed to measure (e.g., they measure reading skills) but they do not provide accurate (valid) information on what they measure. They are like a stop watch that gives the wrong time. Therefore, the findings of research may LOOK as if one curriculum produces higher achievement than another, when in fact the information (data, numbers) is incorrect. Therefore, it is essential to use instruments and measures that have already been tested and have been shown to provide accurate information. You cannot trust the claims of authors who do NOT discuss how their tests, instruments, or measures were validated.

5. Is the sample size, composition (kinds of participants), and selection/allocation of participants to groups appropriate for the type of study and for how the findings might be used? Poor. The researcher claims that a teaching method is effective. However, the method was used in only one class, or in one school. There is no way to tell if it would be effective anywhere else. Poor. The researcher claims that a method is not effective. However, it was used in only one class, or in one school. There is no way to tell if it would have been effective elsewhere. [More] [More]

Poor. The researcher claims that an instructional procedure or curriculum is more effective than another. However, the comparison groups (one class used one curriculum; the other class used a different curriculum) were not created by random assignment of children to the groups, or by matching the groups to make sure they had, for example, the same percentage of boys and girls. Therefore, the groups may have been different in other ways besides the curriculum; e.g., more children in one group got reading instruction at home, and this (not the curriculum) caused differences in achievement. [More]

So, if research involves comparing (for example) achievement from one group/sample (class, school) to another, but the researchers did NOT make sure (through random assignment of students to the samples, or through matching) that the groups have the same characteristics, then you can’t trust that differences in achievement are caused by the curriculum or teaching method.

There are several different experimental designs you can use.

Classic Design: Pre-test, Post-Test, Control Group, with Random Allocation If the experimental group and control group are equivalent in most ways, then differences in outcomes (the amount of change in each group between pre-test and post-test) is, logically, an effect of the intervention. If the experimental group and control group are equivalent in most ways, then differences in outcomes (the amount of change in each group between pre-test and post-test) is, logically, an effect of the intervention. E C Pre-test Post-test Pre-test Post-test

Time Series Time Series Sometimes, it is not possible to have comparison groups. But you may have several years of data on the SAME sample (school, district) before and then after an intervention. If there is a significant change in the outcome (dependent) variables, such as student achievement, it is likely to be the effects of the intervention Year Year Eclectic Programs Systematic, Explicit Programs Eclectic Programs Systematic, Explicit Programs Percentage Passing State Proficiency Tests Percentage Passing State Proficiency Tests

Equivalent Time Samples Design. Same Group, Reversal of Treatment and Baseline A useful design within a class, for example, involves collecting pre-intervention data, then instituting the intervention, then removing it for a short time, and then re-instituting it. If student behavior changes along with changes in the intervention, it provides evidence that the intervention has an effect. After all, the students are the same. More  More 

Equivalent Time Samples. Same Group, Reversal of Treatment and Baseline A = Teacher provides NO praise when students are on task and participating. B = Teacher provides immediate and enthusiastic praise when students are on task and participating. The outcome (dependent variable) is the number of times during a 30 minute lesson that students volunteer answers to questions. A1 B1 A2 B2 A1 B1 A2 B2