EXPERIMENTS AND EXPERIMENTAL DESIGN ETHICAL ISSUES IN RESEARCH

If the stimulus is given, then the outcome occurs Logic of causality If X, then Y and If not X, then not Y Or, colloquially: If the stimulus is given, then the outcome occurs If the stimulus is not given, then the outcome does not occur

Basic requirements of a true experiment: - At least 2 groups: one experimental & one control group - Random assignment of subjects to groups. Every case has: (1) the same chance of being selected (i.e. equal probability) (2) the chance of selection does not change (i.e. constant probability) - Manipulation of the IV. - Measurement of the DV. - Except for the experimental manipulation, the 2 groups = treated exactly alike Manipulation checks: series of experiments in which same theoretical concept = operationalzied differently; ask subjects (orally/written) what they felt/thought during or immediately after the experimental maipulation Can also be used to see if subjects understood/recalled essentaial directions/facts related to manipulation A) evaluation check after the IV = manipulated, but before DV = measured (manipulation = fresh and so si R memory; But: may not be feasable; OR reactivity (macip check may alter subjects subsequent behavior (DV)

Results   controlling for pre-existing subject differences; differences are to be neutralized/distributed approximately evenly among the groups  random assignment of subjects to treatment & control groups - making sure that the events occurring within each experimental group are exactly the same, except for the manipulated independent variable  controlling for extraneous factors & experiences during the experiment

Randomization for groups

Tests Use tests of statistical significance to determine the likelihood that the results occurred by chance rather than being caused by the treatment variable (i.e. significant differences btw. the groups). A result is statistically significant at the 0.05 level: it means that only 5% of the time (5 times in 100 if we were to repeat the experiment) would differences btw. the experimental & control groups occur by chance when the experimental variable actually has no effect.

Tests Differences between the mean values for T and C The differences between slopes for T and C

Mean values for T and C

Slope

Staging of an experiment  1. Introduction: Instructions and Cover Story 2. Manipulation of the IV 3. Measurement of the DV (verbal reports; behavioral observation) 4. Debriefing (explain purpose of experiment; explore subjects’ reactions; alleviate subjects’ anxiety about their performance/ participation)

Validity Internal Validity: the approximate truth about inferences regarding the tested causal relationships in the experimental setting   External Validity: the degree to which the conclusions in the study would hold for other persons in other places & at other times (i.e. how generalizable are the results of the study)

Issues to consider Pre-testing of experiments   Laboratory experiments vs. experiments in natural settings (field experiments) Trade-off btw. increase in external validity & control over extraneous variables & manipulation of the IV Subjects’ reaction to the social meaning of the situation: often a problem for laboratory experiments

Types of experiments Pre-test – post-test control group design Post-test-only control group design The Solomon four group design Factorial design Quasi-experimental design Experimental Design in Survey

Pre-test – post-test control group design Tt1 X Tt2 R Ct1 -- Ct2 R = random assignment of subjects to groups T = Treatment group, C = Control group t1 = first measurement, before experimental treatment (pretest), t2 = 2nd measurement, after experimental treatment (post-test) Possible problem regarding external validity: testing-treatment interaction. The effect of the IV may be different when a pretest is present than when it is not (the subjects have been sensitized to the experimental treatment through pre-testing; thus, in real life the effect would not occur). Depends on experimental situation: in education settings, should not matter; in studies of attitude change, however, or persuasion, pretest ma alert subjects and make them more receptive/resistant).

Post-test-only control group design   X Tt1 R -- Ct1 Many variations on this experimental design: See www.socialresearchmethods.net - various experimental designs.

The Solomon four group design Tat1 X Tat2 Cat1 -- Cat2 R X Tbt1 -- Cbt1

Outcome I T =with manipulated IV C = control group

Outcome II

Source: www.socialresearchmethods.net Factorial design extension of the basic experimental design, in which 2 or more IV are manipulated (each manipulated IV is called a factor). Factorial design has several important features: great flexibility for exploring or enhancing the “signal” (treatment) in studies; efficient; the only effective way to examine interaction effects. A level = subdivision of a factor: in this example, time in instruction has two levels and setting has two levels.

The Main Effects and Interaction Effects A main effect is an outcome that is a consistent difference between levels of a factor. Because of the way we combine levels in factorial design, it also enables us to examine the interaction effects that exist between factors. An interaction effect exists when differences on one factor depend on the level of another factor.

Explanation of effects, I A main effect is an outcome that is a consistent difference between levels of a factor. For instance, we would say there’s a main effect for setting if we find a statistical difference between the averages for the in-class and pull-out groups, at all levels of time in instruction. The first figure depicts a main effect of time. For all settings, the 4 hour/week condition worked better than the 1 hour/week one. It is also possible to have a main effect for setting (and none for time). the outcome might be a test of achievement in the subject being taught. 4 groups; 2 are trained in class; 2 outside; and they are divided by number of hours of training. In the green cells, you have the average results of testing after training is completed. Groups are equal in size. In the orange/brownish cells (marginal values) you have the average testing grade by setting (rows), and by time (column), respectively

Explanation of effects, II

Explanation of effects, III

Explanation of effects, IV An interaction effect exists when differences on one factor depend on the level you are on another factor. It's important to recognize that an interaction is between factors, not levels.

Explanation of effects, V The point: margins are the same (no separate effect of time, and no separate effect of setting – no. are equal) while we have these effects if we take the data for groups into account (as opposed to margins)

Quasi-experimental design (QED) QED takes an experimental approach without having full experimental control; lacks some feature of true experiments, generally randomization - Separate sample pretest-posttest design (both groups receive the treatment, but one is tested before the treatment, and the other afterward) - None-equivalent control group design; control group but without random assignment; - Interrupted time series-design: series of observations before and after the treatment; - Multiple time series design: extends the one above by adding a series of observations on a non-equivalent control group.

Experimental Design in Surveys Differently worded questions: As part of a larger survey: direct a question with one wording to a randomly selected sub-sample of respondents; and a differently worded question to the remaining respondents. Vignettes: Short, detailed and concrete descriptions of situations that contain references to the factors that are thought to be important in decision making. To test experimentally which factors affect the subject’s decision making, a number of versions of the vignettes are developed by systematically varying the details of the situation. The different versions are then randomly assigned to respondents.   Constructing different sets of questionnaires

ETHICAL ISSUES IN RESEARCH Basic question: For what to to whoe the researchers in the social sciences are resposible? Human subjests Peers Society

Ethics of treatment of human subjects - Ethical considerations preclude withholding treatment from a control group: education, correction, clinical psychology, medicine frequently the case.   Institutional review boards assess the ethics of the proposed research (including on risk vs. benefits). Overall guiding principle: potential benefits of research should be weighed against potential costs (risks).

General rules: - Researcher should not expose participants to substantial risk of physical or psychological harm, unless the benefits of participation exceed the risks, and the subject knowingly chooses to participate; - Subjects should be informed that their participation is voluntary throughout the entire research; should be told about any aspects of the research that may influence their willingness to participate; - If deception is necessary- appropriate debriefing as soon as possible after their participation; - Confidentiality and anonymity: researchers should use all possible means to protect the confidentiality of information participants provide (identity).

Ethics of data collection and analysis - Honesty and accuracy in collection & analysis of data, and reporting of results - Scientific misconduct, if identified, kills researcher’s career

Ethics of responsibility to society Awareness of potential use and misuse of scientific results by others