Rigor: Contemporary Standards for Experimental and Quasi- Experimental Research Russell Gersten Director,Instructional Research Group & Professor Emeritus, University of Oregon

Purpose: Explain in colloquial terms the shifts in requirements for experimental research. Indicate why this can reduce the gap between research and practice.

Caveats These reflect my understanding of current IES standards and experts on research design. They do not necessarily reflect the views of the US Dept of Education, IES or OSEP. Single subject research is not included. Nor qualitative/ case studies.

Major Issues/Shifts 1.Increased value of Randomized Controlled Trials. 2.Increased stress on problems with confounds. 3.Unit of analysis must match unit of assignment. 4.Increased concern that we control for number of outcome measures. 5.Differential attrition.

Forces That Led to These Shifts Concern in Congress for low quality of much educational research. National Academy of Sciences report indicating that there is no reason that educational research can’t be as scientific as research in public health, engineering, and biology. Successful use of RCTs in terms of studies on welfare programs and other social programs.

Value of Randomized Controlled Trials Although Donald Campbell pushed quasi experiments in 1960s, he realized their flaws as early as Key flaw: Never sure there is not some other factor causing the results. Example: Salk vaccine QED. Schools have become much more amenable.

Increased Stress on Problems with Confounds Studies with one teacher per condition or one school per condition are no longer considered valid. Other potential confounds seriously assessed.

Unit of Analysis Must Match Unit of Assignment Reason: students’ scores tend to “cluster” at a given school or class. Thus, if schools or classes are randomly assigned and student is unit of analysis: Analysis is much too “liberal” and may spread disinformation. Use of hierarchical linear modeling allows for precision.

Results Studies need to be larger and involve many more teachers and students. They become much more costly. US Department of Education is able to support these large scale studies with higher budgets. Studies become more difficult for one university or institution to conduct on their own.

Increased Concern that we Control for Number of Outcome Measures With 20 outcome measures, it is very likely that one will have a significant effect even though intervention is not effective. Even a problem with 5 or 8 outcome measures. Thus, use the Benjamini-Hochberg correction. Use of composite measures.

Differential Attrition Often ignored until past 5 years. Can lead to spurious effects. Examples from various substance abuse programs etc drove this point home. Ideal would be to include the students or teachers who leave… or make sure they were equivalent for both groups (i.e. chance attrition).

Benefits of Standards to Researchers Higher budgets to conduct studies. Increased openness to RCTs in public schools. Opportunity to collaborate with other organizations. More accurate results.

Benefits of Standards to the Profession Fewer spurious findings. –i.e. things that work for two carefully trained graduate students but are not feasible in real world of education. More accurate information goes to teachers, administrators, university faculties, and parents.

Downsides Many cherished beliefs likely to be challenged. Must deal with realities of current work force in special education rather than “ideal”. Opens up the realities of classroom implementation in special education and general education.