Experimental Research Designs Chapter 9 This multimedia product and its contents are protected under copyright law. The following are prohibited by law: • Any public performance or display, including transmission of any image over a network; • Preparation of any derivative work, including the extraction, in whole or in part, of any images; • Any rental, lease, or lending of the program. Copyright © Allyn & Bacon 2008

Experimental Research Purpose To make causal inferences about the relationship between the independent and dependent variables Characteristics Direct manipulation of the independent variable Control of extraneous variables Eliminate the variable from the study Statistically adjust for the effect of the variable Copyright © Allyn & Bacon 2008

Experimental Research Difficulties Difficult to carry out in educational settings Difficult to control all factors that might affect the outcome Consumers should know the limitations associated with a study to judge the usefulness of the findings Copyright © Allyn & Bacon 2008

Experimental Research Interesting websites A discussion of experimental designs A non-technical discussion of experimental designs Some examples of experimental and quasi-experimental designs Copyright © Allyn & Bacon 2008

Experimental Validity Two issues Internal validity The extent to which the independent variable, and not other extraneous variables, produce the observed effect on the dependent variable External validity The extent to which the results are generalizable Copyright © Allyn & Bacon 2008

Copyright © Allyn & Bacon 2008 Internal Validity Internal validity is discussed in terms of factors, called threats, that reduce the level of confidence in any causal conclusions Eleven (11) specific threats History Extraneous events have an effect on the subjects’ performance on the dependent variable The crash of the stock market, 9-11, the invasion of Iraq, etc. Copyright © Allyn & Bacon 2008

Copyright © Allyn & Bacon 2008 Internal Validity Eleven (11) specific threats (continued) Selection Groups that are initially not equal due to differences in the participants in those groups Positive and negative attitudes, high and low achievers, etc. Maturation Participants’ maturation over the course of the study Pretesting The effect of having taken a pretest Copyright © Allyn & Bacon 2008

Copyright © Allyn & Bacon 2008 Internal Validity Eleven (11) specific threats (continued) Instrumentation Poor technical quality (i.e., validity, reliability) or changes in instrumentation Treatment replications Insufficient replications of the treatment Subject attrition Differential loss of participants from groups Statistical regression The natural movement of extreme scores toward the mean Copyright © Allyn & Bacon 2008

Copyright © Allyn & Bacon 2008 Internal Validity Eleven (11) specific threats (continued) Diffusion of treatment The treatment is given, usually inadvertently, to the control group Experimenter effects Different characteristics or expectations of those implementing the treatments across groups Copyright © Allyn & Bacon 2008

Copyright © Allyn & Bacon 2008 Internal Validity Eleven (11) specific threats (continued) Subject effects The effects of being aware one is involved in a study Four types Hawthorne effect John Henry effect Resentful demoralization Novelty effect Copyright © Allyn & Bacon 2008

Copyright © Allyn & Bacon 2008 Internal Validity Control of effects through the choice of specific research designs Interpretation of the results is tempered by the existence of internal validity concerns Internal validity Conceptual definition of internal validity Copyright © Allyn & Bacon 2008

Copyright © Allyn & Bacon 2008 External Validity Five factors affecting external validity Participants Representativeness of the sample in comparison to the population Consistency of the results across subgroups within the sample Personal characteristics of the participants Participant’s awareness of being involved in a study Copyright © Allyn & Bacon 2008

Copyright © Allyn & Bacon 2008 External Validity Five factors affecting external validity Situations - characteristics of the setting Specific environment Special situation Particular school Time - explanations can change over time Treatments - specific way in which an experimental treatment is conceptualized, operationalized, and administered Copyright © Allyn & Bacon 2008

Copyright © Allyn & Bacon 2008 External Validity Five factors affecting external validity Measures Different instruments measure content or constructs differently Measures chance across studies Control of both types of threats through sampling procedures Generalization of results is tempered by external validity concerns Copyright © Allyn & Bacon 2008

Copyright © Allyn & Bacon 2008 Experimental Designs Four types of designs Pre-experimental Quasi-experimental True experimental Factorial Copyright © Allyn & Bacon 2008

Copyright © Allyn & Bacon 2008 Experimental Designs Notation R indicates random selection or random assignment O indicates an observation Test Observation score Scale score X indicates a treatment A, B, C, ... indicates a group Copyright © Allyn & Bacon 2008

Pre-Experimental Designs No pre-experimental design controls internal validity threats well Single group pretest only A X O Internal validity threats History, maturation, attrition, experimenter effects, subject effects and instrumentation are viable threats Useful only when the research is sure of the status of the knowledge, skill, or attitude being changed and there are no extraneous variables affecting the results Copyright © Allyn & Bacon 2008

Pre-Experimental Designs Single group pretest posttest A O X O Internal validity threats Maturation and pretesting are threats History and instrumentation are potential threats Useful when subject effects will not influence the results, history effects can be minimized, and multiple pretests and posttests are used Copyright © Allyn & Bacon 2008

Pre-Experimental Designs Non-equivalent groups posttest only A X O B O Internal validity threats Selection is a threat History, maturation, and instrumentation are potential threats Useful when groups are comparable and participants can be assumed to be about the same at the beginning of the study Copyright © Allyn & Bacon 2008

Quasi-Experimental Designs Types Non-equivalent pretest-posttest, experimental control groups A O X O B O O Non-equivalent pretest-posttest, multiple treatment groups A O X1 O B O X2 O Useful when participants are in pre-existing groups (e.g., classes, schools, teams, etc.) Copyright © Allyn & Bacon 2008

Quasi-Experimental Designs Threats to internal validity Selection is the major concern Controls for statistical regression Likely to control for most other threats provided the groups are not significantly different from one another See Table 9.3 for specific threats related to each design Copyright © Allyn & Bacon 2008

True Experimental Designs Important terminology Random assignment Participants are placed into groups using a random procedure This ensures equivalency of the groups Random selection of subjects Participants are chosen from a population using random procedures This ensures generalizability to the population from which the participants were selected (i.e., external validity) Copyright © Allyn & Bacon 2008

True Experimental Designs Types Randomized posttest only experimental control groups R A X O R B O Randomized posttest only multiple treatment groups R A X1 O R B X2 O Randomized pretest-posttest experimental control groups R A O X O R B O O Copyright © Allyn & Bacon 2008

True Experimental Designs Types (continued) Randomized pretest-posttest multiple treatment groups R A O X1 O R B O X2 O Threats to internal validity Controls for selection, maturation, and statistical regression Likely to control for most other threats See Table 9.3 for specific threats related to each design Copyright © Allyn & Bacon 2008

Copyright © Allyn & Bacon 2008 Factorial Designs Research designs containing two or more independent variables A study of the effects of two instructional strategies on male and female students’ math achievement A study of two counseling approaches on middle and secondary students’ self-esteem Examples of factorial designs Copyright © Allyn & Bacon 2008

Copyright © Allyn & Bacon 2008 Factorial Designs Types of effects Main effects There is a main effect for each independent variable For the first example above, there is one main effect for instructional strategy and one main effect for math achievement For the second example above, there is one main effect for counseling approach and one main effect for school level Copyright © Allyn & Bacon 2008

Copyright © Allyn & Bacon 2008 Factorial Designs Types of effects (continued) Interaction effect A different effect for the level of the first independent variable across the levels of the second independent variable For the first example above, the first instructional strategy could be effective for males but not females, whereas the second instructional strategy could be effective for females but not males Copyright © Allyn & Bacon 2008

Copyright © Allyn & Bacon 2008 Factorial Designs Interaction effect (continued) For the second example above, the first counseling approach could be effective for secondary students but not so for middle school students, while the second strategy could be effective for middle school students but not so for secondary students In both examples one cannot state the effectiveness of the instructional strategy or counseling method without qualifying it relative to the gender or school level respectively Copyright © Allyn & Bacon 2008

Evaluating Experimental Designs Criteria for evaluating experimental research The primary purpose is to test causal hypotheses There should be direct intervention of the independent variable There should be clear identification of the specific research design Copyright © Allyn & Bacon 2008

Evaluating Experimental Designs Criteria for evaluating experimental research The design should provide maximum control of extraneous/confounded variables The intervention should be described and implemented as planned The number of participants is dependent on equal to the number of intervention replications Copyright © Allyn & Bacon 2008

Single Subject Designs Designs in which the effect of an experimental treatment is studied for one participant Repeated measurement of the dependent variable before, during, and after implementing treatment Not restricted to one (1) participant, but rarely involves more than three (3) participants Used extensively in studies involving exceptional children or counseling Copyright © Allyn & Bacon 2008

Single Subject Designs Characteristics Reliable measurement Repeated measurement Clear description of the conditions Baseline and treatment conditions One variable investigated Notation A indicates a baseline condition without treatment B indicates a treatment condition Copyright © Allyn & Bacon 2008

Single Subject Designs Two specific designs A B A Multiple observations are made during initial baseline time frame; multiple observations during treatment implementation time frame; treatment withdrawn and multiple observations during the second baseline time frame Variations on this design include A B A B (i.e., including a second treatment phase) Limitations Complicated statistical analysis of the data Interpretation of specific outcomes (e.g., a lasting effect of treatment that does not diminish in the second baseline observations) Copyright © Allyn & Bacon 2008

Single Subject Designs Two specific designs (continued) Multiple baseline designs Extension of the A B A design to include more than one subject, behavior, or setting These designs enhance the generalizability of the results Copyright © Allyn & Bacon 2008

Single Subject Designs Criteria for evaluating single subject research Reliable measurement of the target behavior Target behavior is clearly defined in operational terms Sufficient measurements are made during each time frame to establish stability Copyright © Allyn & Bacon 2008

Single Subject Designs Criteria for evaluating single subject research Full descriptions of the procedures, participants, and settings are provided Use of one (1) standard treatment Control of experimenter and/or observer effects Results should have practical significance Copyright © Allyn & Bacon 2008