1. Experimental Designs Introduction to Research Methods
School of Communication Studies
James Madison University
Dr. Michael Smilowitz

2. Experimental Designs The material to follow draws heavily from:
Reinard, J.C. (1997). Introduction to Communication Research. Boston: McGraw - Hill.
Smith, M.J. (1988). Contemporary Communication Research Methods. Belmont, CA: Wadsworth.

3. independent variables.
Experiment Defined:
A functional research design whose purpose is to study the relationship among certain input variables on other outcome variables.
Another term that fits here is:
independent variables.
Input variables are either:
1. Manipulations constructed by the experimenter.
2. Naturally occurring differences among communicators.
Another term that fits here is:
dependent variables.
The research is done to assess (measure) the effect of the input variables on the output variables.

4. Experimental designs differ from surveys in this important regard:
Surveys only assess the associations between naturally occurring existing variables.

5. Experimental designs differ from surveys in this important regard:
Surveys only assess the associations between naturally occurring existing variables.
The variables in a survey are not manipulated by the research. Survey designs can only establish if the studied variables are correlated, or otherwise can be categorized, to indicate that the variables are someway related to each other. Survey designs cannot indicate whether there is a causal relationship among the variables.

6. Experimental designs differ from surveys in this important regard:
Surveys only assess the associations between naturally occurring existing variables.
Example1:
A researcher surveys the workers in an organization with questionnaires to measure their productivity and group cohesiveness.
The researcher finds a curvilinear relationship, but doesn’t know if the cohesiveness caused the productivity, or if the productivity caused the cohesiveness, or if the two are no more than coincidentally related.

7. Experimental designs differ from surveys in this important regard:
Surveys only assess the associations between naturally occurring existing variables.
Example2:
A survey of voters on measures of newspaper reading and party identification results in the finding that independents read the paper more than either democrats or republicans.
But we can’t conclude that reading the paper more often leads individuals to identify themselves politically as independents. Nor can we conclude that once someone decides to be an independent that individual begins to read the paper more often.

8. Control is necessary for making arguments for causal relationships.
By controlling the input variables, researchers can find evidence to support claims of causal relationships by determining whether the input variables are responsible for the observed effects.
BUT! As Smith puts it, experiments by themselves do not necessarily establish causal relationships. An experiment only increases the confidence we might have in the predictive relationship of input variables on specified output variables.

9. control confounding effects.
Providing for control is the most difficult part of experimental designs.
The objective of good experimental design is to:
control confounding effects.
Confounding refers to the confusing (mixing) of variation from from one source with variation from other sources so that it is impossible to know whether the effects are due to the impact of either variable separately or in some combination.
Control refers to the methods researchers use to remove, or hold constant, the effects of the confounding effects (often times called nuisance variables.)

10. Methods for controlling nuisance variables
There are seven methods to be discussed:
1. Elimination and removal
2. Matching
3. Randomization
4. Blocking
5. Using subjects as their own controls.
6. Counter balancing
7. Statistical control

11. Methods for controlling nuisance variables
1. Elimination and removal
Removes the nuisance variable from the experimental setting.
In a study of self disclosure, the presence of other people might affect the intimacy of shared information.
Therefore, the researcher has subjects converse in a private room.

12. Methods for controlling nuisance variables
2. Matching (sometimes called stratified grouping)
Involves pairing subjects on variables that they equally share and assigning one to the experimental group and one to a control group.
Let’s say we’re interested in the effect of sex appeal used in a commercials. We have reason to believe age, sex, and educational level will affect the results.
To control the factors that might affect the results, two groups are created, each with the same age, sex, and educational level distributions.

13. Methods for controlling nuisance variables
3. Randomization
Involves assigning subjects so that each and any event or condition is equally likely to belong to any experimental or control condition. Subjects are selected at random from the population following strict rules, and then randomly assigned to the experimental or control conditions.
In short, randomization involves using the rules of chance to balance groups of subjects.

14. Methods for controlling nuisance variables
3. Randomization
Involves assigning subjects so that each and any event or condition is equally likely to belong to any experimental or control condition. Subjects are selected at random from the population following strict rules, and then randomly assigned to the experimental or control conditions.
In short, randomization involves using the rules of chance to balance groups of subjects.
Its quite an assumption, but the laws of statistical probability indicate that with proper randomization, the probability of selected groups being relatively similar in respect to confounding factors is far greater than without randomization.

15. Methods for controlling nuisance variables
4. Blocking
Adds what might otherwise be a nuisance variable into the design as another independent variable of interest.
Blocking permits drawing conclusions about the impact of each independent variable separately or as part of an interaction with the other independent variables and the blocking nuisance variable.
Requires sizeable samples to permit the subdivisions.
For example, in a study of the effects of regional dialects on credibility, the analysis should consider the geographical regions of the receivers.

16. Methods for controlling nuisance variables
5. Using subjects as their own controls.
Pre-testing and post-testing can measure the effect of the treatment variables, by measuring the average amount of change, despite any differences between subjects.
Let’s say a researcher is interested in studying the effects of a training program to reduce communication anxiety. The researcher divides the subjects into two groups, pre-tests both groups with a communication apprehension instrument, administers the training program to only one group, and then measures both groups again with the same communication apprehension instrument. Whatever confounding, nuisance variables that affect the dependent measure would be present both in the pre-test and the post-test.

17. Methods for controlling nuisance variables
6. Counter balancing
Rotates the sequence in which experimental treatments are introduced to subjects in an effort to control for extraneous variables, such as fatigue or cumulative learning effects. The effects of the nuisance variables do not go away, but are distributed equally across all conditions -- hence, the term counterbalanced.

18. Methods for controlling nuisance variables
6. Counter balancing
Rotates the sequence in which experimental treatments are introduced to subjects in an effort to control for extraneous variables, such as fatigue or cumulative learning effects. The effects of the nuisance variables do not go away, but are distributed equally across all conditions -- hence, the term counterbalanced.
In an experiment on conflict management in families, the researchers hypothesize that different patterns of interaction occur depending on the topic of the conflict. The researchers gather a number of families, assign each family three topics to discuss, and ask them to tape record their conversations over a month's period whenever they are all seated together at the dinner table. The topics are: (1) determining what television programs are to be watched; (2) how much allowance should be given to the family's adolescents; (3) how much time must be spent studying and completing homework. To make sure that the sequence in which the families discuss the assigned topics does not affect the results, the families are divided into three groups, and each group is given the topics in different orderings.

19. Methods for controlling nuisance variables
7. Statistical control
If one can measure a nuisance variable, it is possible to use statistical tools to hold it constant. The analysis of covariance and partial correlation are two common methods. These statistical methods compute the variation associated with each nuisance variable and separate it’s variation from the remaining total variation in the experiment.

20. EXPERIMENTAL VALIDITY
Issues of experimental validity concern the design factors that prevent researchers from drawing conclusions about their results. There are two types of concerns: internal invalidity and external invalidity.
Internal Experimental Validity
Contamination in the experimental procedures that is responsible for the observed effects.
External Experimental Validity
Refers to the degree to which experimental results may be generalized to other similar circumstances.

21. Threats to Internal Experimental Validity
History
Events not controlled by the researcher that occur during the experiment between the pre-testing and post-testing.
Researchers are conducting a study to determine whether watching national broadcast news influences voter preference. There are three groups in the study, who are assigned to watch network news for a week, not watch network news, and a group not given any instructions at all. During the week, however, several major newspapers publish some really disparaging facts about the candidates.

22. Threats to Internal Experimental Validity
Selection
Sampling biases that result from the methods for selecting or assigning subjects to experimental conditions.
Examples?

23. Threats to Internal Experimental Validity
Maturation
Changes that occur naturally over time (including fatigue or suspicion) even if subjects are left alone.
Researchers are interested in the effects of conflict training on student=s ability to manage conflict. The researchers assess entering freshman’s familiarity with conflict management principles, and provide them with a conflict management workshop. The researchers do not measure the students again until the beginning of their sophomore year. But, the improvement in their scores could likely be the result of students maturing, having more experiences at college, and learning conflict management skills on their own. What could be done in this study to reduce maturation effects?

24. Threats to Internal Experimental Validity
Testing
Repeated measurements with the same instrument leads people to become test wise.
If you take the same test often enough, will you perform better?

25. Threats to Internal Experimental Validity
Instrumentation
Changing the measuring instruments from the pre-test to the post-test, or the manner by which the instruments are administered (used) confounds the results.
Opposite problem of testing. Difficult to construct identical but different instruments.
Instrumentation threats can result from differences in the instructions. And if coding data, changes in who is doing the coding also causes instrumentation threats.

26. Threats to Internal Experimental Validity
Experimental Mortality
The results of an experiment are likely to be biased if subjects drop out of the experiment before its completion.
Let’s say we are doing a study on successful compliance strategies in weight loss programs. Over a six month period, many subjects become disappointed about their progress. At the end of the six months, comparing the average of the remaining subjects weights with the initial average will necessarily show inflated levels of improvement, leading the experimenters to think the selected compliance strategies are more successful than what they actually are.

27. Threats to Internal Experimental Validity
Statistical Regression
Statistical regression holds that people will Anaturally regress towards the mean if they are simply left
Applies when subjects are selected for their particularly high (or low) scores.

28. Threats to External Experimental Validity
In the strictest sense, the findings of an experiment are accurate indicators only of the sample groups from which they are drawn.
But with careful sampling and good experimental design, researchers are inclined to regard their results as characteristics of the populations from which the samples are drawn.
Threats to external validity do not mean that the experimental variables did not produce the alleged effects -- internal validity assesses that.

29. Threats to External Experimental Validity
Interaction of test and experimental variables (also called pretest sensitization)
This threat occurs when the pre-testing of subjects makes them more (or less) sensitive to the experimental variable.
In a study on sex differences in self-disclosure during actual conversation, the researchers want to control for individual differences among the male and female subjects regarding their own willingness to self-disclose. To do so, they decide to pre-test subjects. But the pre-test “tips” subjects about the purpose of the study and they behave differently than typical.

30. Threats to External Experimental Validity
Interaction of the selection of subjects and the experimental variables.
This threat refers to the situation in which the selection of the particular sample groups limits the ability to generalize to the larger population because of the particular characteristics of the experimental variables.
Let’s say a group of communication researchers are studying public speaking styles. Could the results be generalized if the researchers randomly select their subjects from students in their communication classes?

31. Threats to External Experimental Validity
Reactive Arrangements
The external validity of a study is threatened if subjects are reacting differently because of the experimental arrangements rather than the experimental variable alone.
The classic “Asch” and “Hawthorne” studies are examples of the problems of reactive arrangements.

32. Threats to External Experimental Validity
Multiple Treatment Interference
This threat results when repeated experimental treatments leads subjects to react in ways quite different than that which would be expected in the general population.
Imagine a group of researchers wanting to study the effects of programs that reduce communication anxiety. They select a group of subjects, and expose them to public speaking training, self-esteem enhancement programs, participation in social support groups, and training in self-monitoring skills. What percentage of the general population would engage in all these activities?

33. Controlling with Design
Reinard (1997) offers a reprint of Stanley & Stanley’s (1966) Table of Experimental Design Validity Factors. The following slides are based on that table.
In the table, a minus indicates a definite weakness, a plus indicates that the factor is controlled, a question mark indicates a possible source of concern, and a blank indicates that the factor is not relevant.

34. Controlling with Design

35. Controlling with Design

36. Controlling with Design

37. Controlling with Design

38. Controlling with Design

39. Controlling with Design