Chapter 3 Experiments, Quasi-Experiments, and Field Observations Winston Jackson and Norine Verberg Methods: Doing Social Research, 4e

© 2007 Pearson Education Canada3-2 The Rationale of the Experiment John Stuart Mill — Method of Difference — the experiment is the key tool for discerning causal relations A well-designed experiment should provide clear evidence of a cause-effect relation Indicate whether or not the treatment variable (e.g., studying with the radio on) will bring about a change in some dependent measure (grade performance), other things being equal

© 2007 Pearson Education Canada3-3 Rationale (continued) Internal validity: The extent to which the researcher can demonstrate that the treatment variable is having an impact on the dependent variable, and that other sources have been controlled External validity: The extent to which the researcher can extrapolate the study findings to other groups in general

© 2007 Pearson Education Canada3-4 Key Elements in Experimental Designs A. Dependent variable – the effect in a cause- effect relationship B. Independent variable – the variable the researcher manipulates to determine whether and how it will change the dependent variable  the cause in a cause-effect relationship

© 2007 Pearson Education Canada3-5 Key Elements (cont’d) Kinds of independent variables: i. Treatment Variables – variable studied ii. Control Variables – major influences intentionally controlled for in the experiment iii. Confounding Variables – variables that can unintentionally obscure or enhance results iv. Random Variables – vary without control, but are taken into account in study design (e.g., randomization)

© 2007 Pearson Education Canada3-6 Key Elements (cont’d) C. Levels – often two or three levels 2 x 2: two levels of the treatment variable and two levels in a control variable

© 2007 Pearson Education Canada3-7 Pseudo-Experimental Designs Have limited scientific merit Also called pre-experimental designs Share some elements of classic experiment, however, they do not permit clear causal inferences Two types: Same group: pretest/post-test design Exposed/comparison group design

© 2007 Pearson Education Canada3-8 A. Same Group: Pretest/Post-Test

© 2007 Pearson Education Canada3-9 Threats to Internal Validity i.History – concurrent events ii.Maturation – changes in the individual subject iii.Testing – possible of response bias iv.Instrument Decay – unreliable measurement v.Statistical Regression – extreme scores

© 2007 Pearson Education Canada3-10 B. Exposed/Comparison Group Measures are taken at only one point in time. Problem: groups may not have been similar initially. The result may, or may not, be due to the treatment variable.

© 2007 Pearson Education Canada3-11 More Threats to Internal Validity vi. Selection – Subjects selecting themselves into the study vii. Mortality – Subjects selecting themselves out of the study

© 2007 Pearson Education Canada3-12 Classic Experimental Designs Two types: Between-Subjects Design Within-Subject Design Both types of design allow a researcher to demonstrate causal inference

© 2007 Pearson Education Canada3-13 A. Between-Subjects Design

© 2007 Pearson Education Canada3-14 Between-Subjects Design (cont’d) Involves a control and an experimental group The experimental group is exposed to treatment intervention The control group is exposed to neutral treatment

© 2007 Pearson Education Canada3-15 Key to Experimental Design Construct treatment and control groups to be as similar as possible before the experiment begins. This is done by: Randomization – each subject has an equal chance of being assigned to either group (provides control over both known [control] and unknown [random] factors) Precision matching – matching subjects between groups Combination of the above two methods

© 2007 Pearson Education Canada3-16 Key to Experimental Design (cont’d) Blocking – Group subjects according to some controlled variable before randomly assigning them to a group Baseline stability – Taking measures of the variable prior to introducing treatment

© 2007 Pearson Education Canada3-17 Analyzing the Data TABLE 3.2PERCENT WANTING TO ATTEND UNIVERSITY BY EXPOSURE AND NON-EXPOSURE TO CD-ROM PERCENTAGE WANTING TO ATTEND UNIVERSITY GROUPTIME 1TIME 2DIFFERENCE Treatment – 57 = 16 Control – 55 = 6 Estimated impact of CD-ROM:10

© 2007 Pearson Education Canada3-18 Demonstrating a Causal Relation 1. Changes in treatment variable occur prior to changes in the dependent variable 2. The treatment and dependent variables are associated: as the treatment variable goes up, the dependent varies systematically 3. Nothing but the treatment variable has influenced the dependent variable

© 2007 Pearson Education Canada3-19 Ruling out Confounding Effects Ensure that context is the same Balance the background characteristics Neutralize confounding (sources of spuriousness) variables Deal with random variables

© 2007 Pearson Education Canada3-20 B. Within-Subject Designs In the between-subjects design, the control for known and unknown factors is achieved through randomization In the within-subject design, the control for known and unknown factors is achieved by exposing a subject to the different treatments Since the subject is the same person, background characteristics, attitudes, and intelligence are all perfectly controlled Also called control by constancy

© 2007 Pearson Education Canada3-21 Within-Subject Design (cont’d) Subjects are exposed to the various treatments Subjects’ own scores when exposed to different treatments are compared Importance of having a baseline measure and returning to the original condition The within-subject ABBA design: A – measure dependent variable under original condition B – measure dependent variable under treatment condition B – continue treatment condition and measure dependent variable A – measure dependent variable after returning to original condition

© 2007 Pearson Education Canada3-22 Hawthorne Effect Refers to any variability in the dependent variable that is not the direct result of variations in the treatment variable Hypothesis: worker productivity would increase as lighting intensity was increased When lighting increased, productivity increased HOWEVER, when lighting was later decreased, productivity did not decrease. WHY? Interpretation: something other than treatment variable influenced workers – perhaps they worked faster because they knew were being observed

© 2007 Pearson Education Canada3-23 Quasi-Experimental Designs Approximation of experimental design: done in situations where it is not possible to: use random assignment control the nature or timing of the treatment Example: Henry & Ginzberg: Racial Discrimination in Employment (See Box 3.4, text pp )

© 2007 Pearson Education Canada3-24 Racial Discrimination in Employment Two job applicants matched with respect to age, sex, education, physical appearance (dress), and personality were sent to apply for the same advertised job. Only difference: one was White, one was Black Results Both offered job 5.0% White offered job13.4% Black offered job 4.5% Neither offered job77.1%

© 2007 Pearson Education Canada3-25 Field Experiments Researcher intervenes in a natural settings Direct observations, “real” behaviour Researcher intervention Greeting stranger Proxemics: norms surrounding personal space and the conditions under which such space will or will not be violated Examples: cutting-through behaviour, greeting behaviour, helping behaviour

© 2007 Pearson Education Canada3-26 Naturalistic Observational Studies Observe and record behaviour that occurs in a natural setting with those being observed unaware that they are being studied Do not attempt to alter social environment No intervention, simply record behaviour Tally sheets are designed, then used to record the behaviour Andrew Harrell’s Grocery Cart Safety study

© 2007 Pearson Education Canada3-27 Samples of Student Research Projects Dressing for winter Parking violations Gender and smoking Professor/student participation: gender Seat belt compliance Speeding Antigonish Buying healthy food ABM behaviour Termination of conversations Drinking patterns Smoking behaviour in teens Stop sign Tipping

© 2007 Pearson Education Canada3-28 Steps in Doing Study 1. Restrict observations 2. Review of literature 3. Develop hypotheses 4. Define terms 5. Develop a tally sheet (See Figure 3.5, p. 90) 6. Transfer data to master table (see Figure 3.6, p. 90) 7. Creating tables (Tables 3.4, 3.5, 3.6, p. 91) 8. Writing the report

© 2007 Pearson Education Canada3-29 Field and Observational Studies: An Assessment Weak on generalizations Strong on validity (real behaviour) Making causal inferences a challenge Multivariate a problem Probing strong with participant observation, in-depth interviews, and focus groups Probing weak with naturalistic observational

© 2007 Pearson Education Canada3-30 Advantages and Disadvantages of Experimental Designs Advantages: Ease of making clear causal inferences Disadvantages: Low external validity: poor on generalization to a larger population Concerns about the artificiality of lab Poor on probing, poor on multivariate Experiments cannot study all topics