1 COMM 301: Empirical Research in Communication Lecture 10 Kwan M Lee

2 Experiments Things to know by the end of the lecture: –Know advantages and disadvantages of experiments –Know how to achieve experimental control to get causal inference Factors influencing control Specific techniques to ensure control –Know the various experimental designs, in terms of Understand logic behind each design structure, how they look like advantages and disadvantages How to apply

3 Experiments Experiment –method that allows us to evaluate the influence of some independent variables on some dependent variables –while controlling for other intervening variables

4 Experiments Advantages vs. Disadvantages Advantages –Allow causal inference –Replication possible by someone else. Disadvantages –Lack ecological isomorphism –Weak generalizability

5 A note on causality (remember lect2 ?) 3 requirements for causal relationships –Temporal ordering: cause precedes effect in time –Meaningful correlation: must have a theoretical foundations for observed correlations –No alternative causes (hypotheses): correlation cannot be explained by other factors Control: the use of techniques for systematically ruling out alternative causes –Alternative hypotheses exercise E.g. no wash on an exam day  1) no wash is lucky vs. 2) more time spend on reading

6 Factors influencing the amount of control in experiment Manipulation of independent variable Ensuring group equivalence Control of intervening variables

7 Manipulation of independent variable Active manipulation –Researcher determines participants’ level of exposure to independent variable. –Give some assurance that if dependent variable changes, it is due to the independent variable –E.g., default experiment setting Passive observation –Cannot be under control of researcher –Natural setting E.g.,Comparison of two existing classes which have different teaching methods

8 Ensuring group equivalence Experiments use different groups of participants treatment group (treatment) control group (no treatment) comparison groups (when two or more treatment groups) Group equivalence assumption –Must ensure that before imposing the treatment, the groups are equivalent with regard to the dependent variables. –How to insure?  see next slides

9 How to ensure group equivalence Random assignment –Participants assigned to treatment or control, or comparison groups by randomized method. –No guarantee of getting equivalent groups But it usually works! –Can use statistical testing to find out how likely the group is non-equivalent. Comparisons of variables not related to dependent variables

10 How to ensure group equivalence (cont.) Pretesting –Groups are measured on the dependent variable(s) before any treatment. –Expectation is that the groups will be similar If not, non-equivalent with regard to DVs

11 How to ensure group equivalence (cont.) Matching Participants in the groups are matched on characteristics important to dependent variable(s) (i.e. matched on intervening variables) –Constancy matching All participants in all groups kept uniform on the characteristic thought to influence the dependent variable. e.g. holding gender constant: test only males –Pairing Each participant in a group is matched with other participants in other group on the variable(s) thought to affect the dependent variable(s) e.g. for each male (female) in treatment group, there is a male (female) in the control group – i.e., equal # of males and females

12 Control of intervening variables Eliminate the influence of: –participants (through ensuring group equivalence) –settings (through ensuring settings are different only with regard to manipulations) –individual researchers use of double-blind procedures use of scripts. –other factors influencing internal validity (e.g., maturation, attrition.)

13 Experimental designs Experimental research designs are categorized along four dimensions –Research settings –Amount of control –Number of independent variables –Different subjects vs. Same subjects in cells

14 Setting Experimental design setting –Laboratory Research setting created by the researcher, maintains strong control over the setting Sacrifice ecological isomorphism –Field Naturally occurring research setting, with little researcher control Enhance ecological isomorphism e.g. new training program at a company

15 Amount of Control – High Control Pretest posttest control group design –Good control for group equivalence Randomized assignment to treatment and control groups Pre-testing –Handle most threats to validity well. –Limitation: test sensitization –See Graph in p. 86

16 Pretest-posttest control group design T: treatment group C: control group R: Random Assignment Pr: Pretest X : Treatment (intervention) - : No treatment (intervention) Po: Posttest

17 Amount of Control – High Control Posttest only control group –Randomized assignment to treatment and control groups for group equivalence –No pre-testing, sacrifice some group equivalence –Attempts to counter test sensitization in pretest posttest control group design

18 Amount of Control – High Control Solomon four-group design –Combines pretest posttest control group and posttest only control group –Highest level of control Ensures group equivalence through randomized assignment and pretesting Allows assessment of test sensitization’s impact (T1 vs. T2; C1 vs. C2; C1 [Pr vs. Po]) –Limitations: Costs, especially in subjects Difficulties in interpreting contradicting results

19 Solomon four-group design

20 Solomon four-group design interpretation exercise Group 1RPr1 (60) XPo1 (82) Group 2R XPo2 (74) Group 3RPr2 (62) Po3 (70) Group 4RPo4 (62) The numbers in the parentheses represent the average score of the group. You can assume that a difference of 5% or more represents some sort of effect or difference beyond random chance Were the four groups equivalent? Why? –Random assignment –Pr1 = Pr2 = Po4 Does the treatment have an effect? –Po1 vs Po3; Po2 vs. Po4; Po1 vs. Pr1 Do you see prestest effects (test sensitization)? –Po3 vs. Po4; Po1 vs. Po2; Pr2 vs. Po3

21 Amount of Control – Moderate Control High control designs use random assignment. Random assignment not always possible. Shift to moderate control designs. –Pretest posttest nonequivalent group design –Interrupted and multiple interrupted time series

22 Amount of Control – Moderate Control Pretest posttest nonequivalent group –Identical to pretest posttest control group but without random assignment into groups. –Typically, groups already formed naturally E.g. classes –Ability to make causal inferences weakened. Differences in the posttest may be due to initial differences between the groups. –many possible alternative Hs T:PrxPo C:Pr-Po

23 Amount of Control – Moderate Control Interrupted time series design –Only a single group available The group must serve as both T and C groups series of pretests, establish baseline measure administer treatment series of posttests –Advantages can handle pretest sensitization (if all pretest scores are similar) and maturation (if there is little change from one test to the next) can test the persistency of the treatment effect Pr1Pr2Pr3Pr4 xPo1Po2Po3Po4 Multiple interrupted time series design Pr1Pr2Pr3Pr4 xPo1Po2Po3Po4 Pr1Pr2Pr3Pr4 Po1Po2Po3Po4

24 Amount of Control – Low Control Experimental designs, low control –At times, circumstances do not allow random assignment nor even moderate control by pretest –Not an experiment in a strict sense –Usually for Exploratory studies Single group post test only design Single group pre-post test design Posttest only nonequivalent control group design