Presentation is loading. Please wait.

Presentation is loading. Please wait.

Experimental Design Brian Mennecke College of Business Iowa State University Brian Mennecke College of Business Iowa State University.

Similar presentations


Presentation on theme: "Experimental Design Brian Mennecke College of Business Iowa State University Brian Mennecke College of Business Iowa State University."— Presentation transcript:

1 Experimental Design Brian Mennecke College of Business Iowa State University Brian Mennecke College of Business Iowa State University

2 The Source…  The source of much of this information comes from Campbell & Stanley… –Campbell, D. T. and J.C. Stanley (1963). Experimental and Quasi-Experimental Designs for Research. Chicago: Rand McNally College Publishing Company.  The source of much of this information comes from Campbell & Stanley… –Campbell, D. T. and J.C. Stanley (1963). Experimental and Quasi-Experimental Designs for Research. Chicago: Rand McNally College Publishing Company.

3 The Goal of Research  When one conducts science, the goal is to seek out the truth. –Question: How does one identify truth?  Experimentation is one mechanism for identifying causation, which is a step toward understanding how one set of factors influence another set of factors  When one conducts science, the goal is to seek out the truth. –Question: How does one identify truth?  Experimentation is one mechanism for identifying causation, which is a step toward understanding how one set of factors influence another set of factors

4 Causation and Positivism  Positivism is a research perspective that has as its premise that inferences about cause can be made.  David Hume espoused the conditions by which inference could be made; these include… –Contiguity between the cause and effect –Temporal precedence –Constant conjunction (i.e., when the effect is seen, the cause is always present)  Positivism is a research perspective that has as its premise that inferences about cause can be made.  David Hume espoused the conditions by which inference could be made; these include… –Contiguity between the cause and effect –Temporal precedence –Constant conjunction (i.e., when the effect is seen, the cause is always present)

5 But how do we know something is true?  Some propositions are not true; how do we know when something is true or not?  One approach is to test for validity. –Validity is a term used to describe whether the conclusions one draws about a proposition are true or false  Some propositions are not true; how do we know when something is true or not?  One approach is to test for validity. –Validity is a term used to describe whether the conclusions one draws about a proposition are true or false

6 Types of validity  Internal Validity: How sure are we that the cause leads to the expected results? In other words, is it appropriate for us to infer that the relationship between variables is causal  External Validity: How sure are we that we can generalize the finding of causation to other populations, settings, or variables?  Construct Validity: How sure are we that the variables we are using actually measure the concept (i.e., the construct) that we are seeking to measure?  Statistical Conclusion Validity: Do the statistical tests that we perform accurately measure the relationships between the variables under study?  Internal Validity: How sure are we that the cause leads to the expected results? In other words, is it appropriate for us to infer that the relationship between variables is causal  External Validity: How sure are we that we can generalize the finding of causation to other populations, settings, or variables?  Construct Validity: How sure are we that the variables we are using actually measure the concept (i.e., the construct) that we are seeking to measure?  Statistical Conclusion Validity: Do the statistical tests that we perform accurately measure the relationships between the variables under study?

7 Threats to Internal Validity (Campbell & Stanley)  History: events that occur between the first and second measurement that are unrelated to the experiment but that could affect the results.  Maturation: Changes in the participants that occur as a function of the passage of time and not specific to the experiment.  Testing: The effects of taking a first test on the scores of a second test.  Instrumentation: Changes in the measurement instrument or changes or the observers make changes in the obtained measurements.  Statistical regression (toward to mean): Groups having extreme scores on the pretest (or selected on the basis of extreme scores) will tend to have scores closer to the mean on the posttest.  Selection: Biases resulting in differentials selection of respondents for the comparison groups.  Experimental mortality: Differential loss of respondents from the comparison groups.  Selection-maturation interaction, other interaction effects:  History: events that occur between the first and second measurement that are unrelated to the experiment but that could affect the results.  Maturation: Changes in the participants that occur as a function of the passage of time and not specific to the experiment.  Testing: The effects of taking a first test on the scores of a second test.  Instrumentation: Changes in the measurement instrument or changes or the observers make changes in the obtained measurements.  Statistical regression (toward to mean): Groups having extreme scores on the pretest (or selected on the basis of extreme scores) will tend to have scores closer to the mean on the posttest.  Selection: Biases resulting in differentials selection of respondents for the comparison groups.  Experimental mortality: Differential loss of respondents from the comparison groups.  Selection-maturation interaction, other interaction effects:

8 Threats to External Validity  Reactive or interaction effects of testing: The pretest itself might be a learning experience such that by taking the pretest students gain information that will affect posttest results  Interaction of selection and the experimental variable: Different groups may respond differently to the experimental variable.  Reactive effects of experimental arrangements: Subjects respond differently because they know they are in an experiment (i.e., the Hawthorne effect)  Multiple treatment interference: Multiple treatments applied to the same respondents; the effects of prior treatments cannot be erased.  Reactive or interaction effects of testing: The pretest itself might be a learning experience such that by taking the pretest students gain information that will affect posttest results  Interaction of selection and the experimental variable: Different groups may respond differently to the experimental variable.  Reactive effects of experimental arrangements: Subjects respond differently because they know they are in an experiment (i.e., the Hawthorne effect)  Multiple treatment interference: Multiple treatments applied to the same respondents; the effects of prior treatments cannot be erased.

9 What is the basis for asking questions about causation?  The source for all questions pertaining to research experimentation is theory –Why is theory important?  Theory should always drive research because it defines expectations about the relationships that exist between variables.  The source for all questions pertaining to research experimentation is theory –Why is theory important?  Theory should always drive research because it defines expectations about the relationships that exist between variables.

10 Before we get started…  Some definitions: –Construct: An idea or concept that you are attempting to measure Latent Construct: A construct that cannot be measured directly (e.g., group cohesion) –Independent Variable: Variables that are presumed to be the cause of an effect being studied; independent variables are manipulated to examine their impact on results –Dependent Variables: Variables that are observed to understand the result of causation. –Hypothesis: A statement of a possible explanation for causation. An hypothesis is tested by drawing conclusions from an experimental examination of the variables that are expected to be related  Some definitions: –Construct: An idea or concept that you are attempting to measure Latent Construct: A construct that cannot be measured directly (e.g., group cohesion) –Independent Variable: Variables that are presumed to be the cause of an effect being studied; independent variables are manipulated to examine their impact on results –Dependent Variables: Variables that are observed to understand the result of causation. –Hypothesis: A statement of a possible explanation for causation. An hypothesis is tested by drawing conclusions from an experimental examination of the variables that are expected to be related

11 Types of Experimental Designs  Pre-experimental designs: One group designs and designs that compare pre-existing groups  Quasi-experimental designs: Experiments that have treatments, outcome measures, and experimental conditions but that do not use random selection and assignment to treatment conditions.  True experimental designs: Experiments that have treatments, outcome measures, and experimental conditions and use random selection and assignment to treatment conditions. This is the strongest set of designs in terms of internal and external validity.  Pre-experimental designs: One group designs and designs that compare pre-existing groups  Quasi-experimental designs: Experiments that have treatments, outcome measures, and experimental conditions but that do not use random selection and assignment to treatment conditions.  True experimental designs: Experiments that have treatments, outcome measures, and experimental conditions and use random selection and assignment to treatment conditions. This is the strongest set of designs in terms of internal and external validity.

12 Pre-Experimental Designs  Design 1: One-Shot Case Study: A single group is studied once after some intervention/treatment that is presumed to cause change. –For example, a training program is implemented and participants are given a posttest at the conclusion of the training. XO  Design 1: One-Shot Case Study: A single group is studied once after some intervention/treatment that is presumed to cause change. –For example, a training program is implemented and participants are given a posttest at the conclusion of the training. XO

13 Pre-Experimental Designs  Design 2: One-Group Pretest-Posttest Design: One group, not randomly selected nor randomly assigned, is given a pretest, followed by a treatment/intervention, and finally a posttest. There is no comparison group. Generally done with intact groups. –For example, a classroom teacher gives her students a pretest then implements an instructional strategy followed by a posttest. O1 XO2  Design 2: One-Group Pretest-Posttest Design: One group, not randomly selected nor randomly assigned, is given a pretest, followed by a treatment/intervention, and finally a posttest. There is no comparison group. Generally done with intact groups. –For example, a classroom teacher gives her students a pretest then implements an instructional strategy followed by a posttest. O1 XO2

14 Pre-Experimental Designs  Design 3: The Static-Group Comparison: One group which has experienced a treatment/intervention (X) is compared to another group that has not had the intervention. The groups are not randomly selected nor randomly assigned and are generally pre-existing groups. There is no pre-observation/pretest. –For example, comparison of GRE scores for students who attended a rural high school versus those who attended an urban high school. X1O X2O  Design 3: The Static-Group Comparison: One group which has experienced a treatment/intervention (X) is compared to another group that has not had the intervention. The groups are not randomly selected nor randomly assigned and are generally pre-existing groups. There is no pre-observation/pretest. –For example, comparison of GRE scores for students who attended a rural high school versus those who attended an urban high school. X1O X2O

15 True Experimental Designs  Design 4: Pretest-Posttest Control Group Design: One group is administered a treatment while the other is not; all groups are observed before and after the treatment is administered. –For example, 50 freshman students are randomly selected to participate in a tutoring study. Half are randomly assigned to a tutor for their first semester and half are not. All students are given a pretest at the beginning of the term and a posttest at the end of the term. RO1 XO2 RO1O2  Design 4: Pretest-Posttest Control Group Design: One group is administered a treatment while the other is not; all groups are observed before and after the treatment is administered. –For example, 50 freshman students are randomly selected to participate in a tutoring study. Half are randomly assigned to a tutor for their first semester and half are not. All students are given a pretest at the beginning of the term and a posttest at the end of the term. RO1 XO2 RO1O2

16 True Experimental Designs  Design 5: Solomon Four-Group Design: This design involves four experimental groups. Two of the groups parallel the structure of Design 4 while the remaining two groups include no pre-test (so that the effects of the pretest can be evaluated). –For example, 100 freshman students are randomly selected to participate in a tutoring study. 25 are randomly assigned to a tutor for their first semester and given a pretest. 25 are randomly assigned to a group where no tutor is assigned and they are given a pretest. Another 25 are randomly assigned to a tutor but not given a pretest. The remaining 25 are randomly assigned to a group where no tutor is assigned and they are not given a pretest. Whew! RO1 XO2 RO1O2 R XO2 RO2  Design 5: Solomon Four-Group Design: This design involves four experimental groups. Two of the groups parallel the structure of Design 4 while the remaining two groups include no pre-test (so that the effects of the pretest can be evaluated). –For example, 100 freshman students are randomly selected to participate in a tutoring study. 25 are randomly assigned to a tutor for their first semester and given a pretest. 25 are randomly assigned to a group where no tutor is assigned and they are given a pretest. Another 25 are randomly assigned to a tutor but not given a pretest. The remaining 25 are randomly assigned to a group where no tutor is assigned and they are not given a pretest. Whew! RO1 XO2 RO1O2 R XO2 RO2

17 True Experimental Designs  Design 6: Posttest Only Control Group Design: One group is administered a treatment while the other is not; all groups are observed after the treatment is administered BUT not before the treatment. –For example, students are randomly assigned to two groups of 50 each. The experimental (treatment) group receives a new teaching method during a special class session. The second group (the control) receives a traditional teaching method during a special class session. No pretest is used for each group. Issues such as existing grades, SAT scores, and other factors are examined as covariates. R XO2 RO2  Design 6: Posttest Only Control Group Design: One group is administered a treatment while the other is not; all groups are observed after the treatment is administered BUT not before the treatment. –For example, students are randomly assigned to two groups of 50 each. The experimental (treatment) group receives a new teaching method during a special class session. The second group (the control) receives a traditional teaching method during a special class session. No pretest is used for each group. Issues such as existing grades, SAT scores, and other factors are examined as covariates. R XO2 RO2

18 Quasi-Experimental Designs  Design 7: The Time-Series Experiment: This design involves periodic measurements of some group or individuals and the introduction of a change into the conditions during the series. –For example, studying a group of workers over time and taking several measures of productivity during this period. At some point a new work process is introduced and measures of productivity are taken over several weeks following the intervention. O1 O2 O3XO4 O5 O6  Design 7: The Time-Series Experiment: This design involves periodic measurements of some group or individuals and the introduction of a change into the conditions during the series. –For example, studying a group of workers over time and taking several measures of productivity during this period. At some point a new work process is introduced and measures of productivity are taken over several weeks following the intervention. O1 O2 O3XO4 O5 O6

19 Quasi-Experimental Designs  Design 8: Equivalent Time-Samples Designs: This design involves periodic introduction of treatments followed by measurements with the treatments varied consistently over time. –For example, to study the effect on student discussions of having an observer appear in a classroom. At time period one, an observer is present and a measure of discussion level is made. At time two, no observer is present and a measure of discussion level is made. At time three an observer is present, a measure is taken. At time four an observer is not present, a measure is taken. Etc. X1 O X2OX1O X2 O  Design 8: Equivalent Time-Samples Designs: This design involves periodic introduction of treatments followed by measurements with the treatments varied consistently over time. –For example, to study the effect on student discussions of having an observer appear in a classroom. At time period one, an observer is present and a measure of discussion level is made. At time two, no observer is present and a measure of discussion level is made. At time three an observer is present, a measure is taken. At time four an observer is not present, a measure is taken. Etc. X1 O X2OX1O X2 O

20 Quasi-Experimental Designs  Design 9: The Equivalent Materials Design: This design involves giving equivalent samples of materials to subjects, imparting interventions, and then making observations. –For example, subjects are asked to complete a survey instrument about their opinions related to current events. The students are then split into two groups and given two different sets of (falsified) survey results indicating how other students answered the survey. Both groups are then asked to complete the survey again to observe how they respond. Experimental Materials A(O) X0 O Experimental Materials B(O) X0 O  Design 9: The Equivalent Materials Design: This design involves giving equivalent samples of materials to subjects, imparting interventions, and then making observations. –For example, subjects are asked to complete a survey instrument about their opinions related to current events. The students are then split into two groups and given two different sets of (falsified) survey results indicating how other students answered the survey. Both groups are then asked to complete the survey again to observe how they respond. Experimental Materials A(O) X0 O Experimental Materials B(O) X0 O

21 Quasi-Experimental Designs  Design 10: Nonequivalent Control Group: This design involves an experimental and control group with both given pretests and posttest; however, these groups are not randomly selected because they constitute naturally assembled groups (e.g. classrooms). The assignment of X (the treatment) to one group or the other is randomly selected by the researcher. –For example, four sections of a course are chosen to participate in a study of teaching methods. Half are randomly assigned a new teaching method and half are not. All are given pretests at the beginning of the term and all are given posttests at the end of the semester. O X O O  Design 10: Nonequivalent Control Group: This design involves an experimental and control group with both given pretests and posttest; however, these groups are not randomly selected because they constitute naturally assembled groups (e.g. classrooms). The assignment of X (the treatment) to one group or the other is randomly selected by the researcher. –For example, four sections of a course are chosen to participate in a study of teaching methods. Half are randomly assigned a new teaching method and half are not. All are given pretests at the beginning of the term and all are given posttests at the end of the semester. O X O O

22 Quasi-Experimental Designs  Design 11: Counterbalanced Designs: In this design all subjects receive all treatments but in a different order. Each treatment occurs once at each time period and once for each treatment group. A Latin-square design is a type of counterbalanced design in which four treatments are applied to four naturally assembled pools of subjects. –For example, consider a study of the effect of different training methods on learning. Subjects are placed into four groups (A,B,C, D) for different training methods, X1-X4. Group AX1OX2OX3OX4O Group BX2OX4OX1OX3O Group CX3OX1OX4OX2O Group DX4OX3OX2OX1O  Design 11: Counterbalanced Designs: In this design all subjects receive all treatments but in a different order. Each treatment occurs once at each time period and once for each treatment group. A Latin-square design is a type of counterbalanced design in which four treatments are applied to four naturally assembled pools of subjects. –For example, consider a study of the effect of different training methods on learning. Subjects are placed into four groups (A,B,C, D) for different training methods, X1-X4. Group AX1OX2OX3OX4O Group BX2OX4OX1OX3O Group CX3OX1OX4OX2O Group DX4OX3OX2OX1O

23 Quasi-Experimental Designs  Design 12: The Separate Sample Pretest-Posttest Design: Often used with large populations (i.e., in public opinion studies) where the researcher cannot randomize or segregate subgroups for different treatments. Two equivalent groups are identified, one sample is measured prior to the treatment and a different (but equivalent) sample is measured after the treatment. This design is also called the "simulated before and after" design. –For example, 100 community members are randomly surveyed concerning their opinions about local government policies. A PR campaign is then conducted for six weeks. A follow-up survey is then conducted with 100 different residents who are randomly selected. RO X RXO  Design 12: The Separate Sample Pretest-Posttest Design: Often used with large populations (i.e., in public opinion studies) where the researcher cannot randomize or segregate subgroups for different treatments. Two equivalent groups are identified, one sample is measured prior to the treatment and a different (but equivalent) sample is measured after the treatment. This design is also called the "simulated before and after" design. –For example, 100 community members are randomly surveyed concerning their opinions about local government policies. A PR campaign is then conducted for six weeks. A follow-up survey is then conducted with 100 different residents who are randomly selected. RO X RXO

24 Quasi-Experimental Designs  Design 13: The Separate Sample Pretest-Posttest Control Group Design: This design is similar to Design 12; however, a control group is added to the design. –For example, consider the PR campaign described in Design 12. In this case, the same design is used, but, in addition, the measurements are made in a similar nearby city where no PR campaign is run. RO X RXO RO  Design 13: The Separate Sample Pretest-Posttest Control Group Design: This design is similar to Design 12; however, a control group is added to the design. –For example, consider the PR campaign described in Design 12. In this case, the same design is used, but, in addition, the measurements are made in a similar nearby city where no PR campaign is run. RO X RXO RO

25 Quasi-Experimental Designs  Design 15: Recurrent Institutional Cycle Design (A "Patched-Up" Design): This is an approach used in field research. A researcher begins with an inadequate design and then adds features to control for one or more sources of invalidity. The result is an "inelegant accumulation of precautionary checks." The researcher is aware of rival interpretations (sources of internal invalidity) and incrementally identifies other data that would rule out rivals. The design exploits contextual features to refine the research as it progresses. –For example, this design would combine a longitudinal and cross sectional structure. One group will be exposed to X and measured at the same time as a second group that is just about to be exposed to X. A comparison of the two groups would be able to be made because it is equivalent to a static group comparison. The second group would be remeasured (posttest), which would make the design comparable to the one group pretest-posttest design. Group AXO1 Group B O1 XO2  Design 15: Recurrent Institutional Cycle Design (A "Patched-Up" Design): This is an approach used in field research. A researcher begins with an inadequate design and then adds features to control for one or more sources of invalidity. The result is an "inelegant accumulation of precautionary checks." The researcher is aware of rival interpretations (sources of internal invalidity) and incrementally identifies other data that would rule out rivals. The design exploits contextual features to refine the research as it progresses. –For example, this design would combine a longitudinal and cross sectional structure. One group will be exposed to X and measured at the same time as a second group that is just about to be exposed to X. A comparison of the two groups would be able to be made because it is equivalent to a static group comparison. The second group would be remeasured (posttest), which would make the design comparable to the one group pretest-posttest design. Group AXO1 Group B O1 XO2

26 My Research Agenda  So, what type of research approach do you think I use?

27 General Research Themes  Geographic Information Systems (GIS) and Location Intelligence –Studies of the use of GIS as a decision support tool –The use of GIS in Businesses and Organizations –Location intelligence and the use of location in decision making –Perceptions of space and geography  Geographic Information Systems (GIS) and Location Intelligence –Studies of the use of GIS as a decision support tool –The use of GIS in Businesses and Organizations –Location intelligence and the use of location in decision making –Perceptions of space and geography

28 General Research Themes  Studies of Teams, Collaboration, and technology –Virtual Teams –Team History –Individual Characteristics  Studies of Teams, Collaboration, and technology –Virtual Teams –Team History –Individual Characteristics

29 General Research Themes  Virtual Worlds –The application of VW to education and learning –Perceptions of avatars, space and location in VWs –Legal, tax, and social issues in VWs –Communication and collaboration in VWs  Virtual Worlds –The application of VW to education and learning –Perceptions of avatars, space and location in VWs –Legal, tax, and social issues in VWs –Communication and collaboration in VWs

30 General Research Themes  Mobile Commerce, Computing, and Virtual Teams –Mobile Device Interfaces –Impressions of Mobile Device Users –Applications of Mobile Devices  Mobile Commerce, Computing, and Virtual Teams –Mobile Device Interfaces –Impressions of Mobile Device Users –Applications of Mobile Devices

31 General Research Themes  Applications of Conjoint to IS Research –Human Resources –Information Systems Analysis –IT Planning  Applications of Conjoint to IS Research –Human Resources –Information Systems Analysis –IT Planning

32 General Research Themes  IT Adoption and Implementation –User Acceptance of Mobile Devices –The Use of Mobile Devices in Commerce  IT Adoption and Implementation –User Acceptance of Mobile Devices –The Use of Mobile Devices in Commerce

33 General Research Themes  The Application of IT for Training and Learning –The Application of Technology in Education –The Role of Communication Technology in Learning  The Application of IT for Training and Learning –The Application of Technology in Education –The Role of Communication Technology in Learning

34 A Recent Study  Question: What is the impact of video conference technology and training methodology on student learning  IV: –Training Mode: Enactive Mastery Vicarious Experience –Communication Media Face to Face Video Conferencing  Question: What is the impact of video conference technology and training methodology on student learning  IV: –Training Mode: Enactive Mastery Vicarious Experience –Communication Media Face to Face Video Conferencing

35 Results

36


Download ppt "Experimental Design Brian Mennecke College of Business Iowa State University Brian Mennecke College of Business Iowa State University."

Similar presentations


Ads by Google