Field Experiments

Typical Format Researcher manipulates something in the real world, exposing randomly-assigned groups of people to different treatments Possibly the most fruitful area of research in social sciences, but also greatest ethical challenges since the experiments can be so powerful and have real-world effects

Field Experiments in Practice Most medical clinical trials are field experiments Since the experiment takes place outside of the laboratory, researchers cannot completely control randomization Some subjects selected for treatment may refuse, others may switch from control to treatment or from treatment to control – Example: Dartmouth Spine Study

As Treated vs Intent to Treat As treated (AT) analysis compares subjects given the treatment they received. – Does not consider treatment group subject assigned, only which they received Intent to Treat (ITT) compares subjects in the groups to which they were randomized, regardless of whether they received or adhered to the allocated treatment. – Required in medical trials

Intent to Treat: Adherence Problem 1: Adherence: Subjects who did not adhere to treatment or switched to control should be included – Analyze data in group to which subject allocated, not what they actually received

Intent to Treat: Loss to Follow-Up Problem 2: Loss to Follow-up: Subjects who did not complete trial or could not be contacted to provide data should be included – Cannot be deleted from analysis since they are not missing due to random factors – Missing data problem

Applications in Social Sciences Many field experiments manipulate the information that people receive Information is a less powerful and shorter- term manipulation, making the ethical issues less significant

The Effects of Choice and Enhanced Personal Responsibility for the Aged: A Field Experiment in an Institutional Setting Ellen Langer and Judith Rodin Journal of Personality and Social Psychology 1976, 34(2)

Research Question Objective helplessness as well as feelings of helplessness and hopelessness may contribute to psychological withdrawal, physical disease, and death. Objective control and feelings of mastery may contribute to physical health and personal efficacy.

Study Design Conducted in nursing home for generally upper income residents Four floors in the home. Two floors selected for study based on similarity in the residents’ physical and psychological health and prior socioeconomic status.

Randomization? Residents assigned to floors based on availability Residents on two different floors exposed to different “treatments”

Treatment: A Meeting on Lounge of Each Floor Both floors held meeting directed by the same staff member, who read from two different scripts Script 1: Responsibility-Induced Message [Treatment] Script 2: No Responsibility [Control]

Treatment Group: Responsibility Message “Many of you don't realize the influence you have over your own lives here. Take a minute to think of the decisions you can and should be making.” “You should be deciding how you want your rooms to be arranged—whether you want it to be as it is or whether you want the staff to help you rearrange the furniture. It’s your life and you can make of it whatever you want.”

Treatment Group (continued) “If you are unsatisfied with anything here, you have the influence to change it. It's your responsibility to make your complaints known, to tell us what you would like to change, to tell us what you would like.”

Treatment Group (continued) “Also, I wanted to take this opportunity to give you each a present from the Arden House. The plants are yours to keep and take care of as you'd like.” [A box of small plants was passed around, and patients were given two decisions to make: first, whether or not they wanted a plant at all, and second, to choose which one they wanted. All residents did select a plant.]

Treatment Group (continued) “One last thing, I wanted to tell you that we're showing a movie two nights next week, Thursday and Friday. You should decide which night you’d like to go, if you choose to see it at all.”

Control Group “I was surprised to learn that many of you don't know about the things that are available to you; that many of you don’t realize all you're allowed to do here.” “We want your rooms to be as nice as they can be, and we've tried to make them that way for you. We want you to be happy here.”

Control Group “If you have any complaints or suggestions about anything, let a nurse know what they are. Let us know how we can best help you. You should feel that you have free access to anyone on the staff, and we will do the best we can to provide individualized attention and time for you.”

Control Group “Also, I wanted to take this opportunity to give you each a present from the Arden House. The plants are yours to keep. The nurses will water and care for them for you.” [The nurse walked around with a box of plants and each patient was handed one.]

Control Group “One last thing, I wanted to tell you that we're showing a movie next week on Thursday and Friday. We'll let you know later which day you're scheduled to see it.

Dependent Variables Pre-post within and between subjects design Questionnaire given to each subject one week before and three weeks after meetings” How much control do you have over your life? (0=none, 8=total) How happy are you? How active do you feel? Alertness rated by research assistant

Dependent Variables Nurses rated each resident on – Happiness – Alertness – Dependence – Sociability – Activity – Eating and Sleeping Habits Nurses’ ratings blind (did not know whether resident was in treatment or control group)

Dependent Variables Residents could enter competition to guess number of jelly beans in a jar Tenth day after treatment measured how far residents moved in their wheelchairs during one day Note: Many dependent variables measured – Often guarantees at least something turns up significant

Results

Behavioral Results Responsibility group – More likely to attend movie – More likely to enter jelly-bean guessing contest No evidence of more wheelchair movement by responsibility group (floors too clean to measure movement?)

Conclusions 71% of control group residents more debilitated over 3-week study 93% of treatment group showed improvement Having personal control over one’s environment and taking care of a plant made subjects happier, healthier, more active

Concerns? Would be interesting to see in a 2x2 design whether taking care of the plant had an effect separate from message received

The Effects of Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment ALAN S. GERBER and DONALD P. GREEN American Political Science Review Vol. 94(3): 2000

Field Experiment on Voter Turnout Conducted in city of New Haven, Connecticut Test effect of encouraging people to vote by – canvassing (appearing at a person’s house), – telephone call – direct mail (up to three) 2 x 2 x 4 factorial design Canvassing x telephone call x direct mail

Problem The probability that a given person in the experiment votes is: Y=a + b 1 X 1 + b 2 X 2 + e Y=1 if subject votes X 1 =1 if subject is difficult to contact X 2 =1 if subject is actually contacted, 0 otherwise Cannot assume b 1 =0 Creates correlation between X 1 and e

Solution: Instrumental Variables Use the intent to treat as an instrumental variable – Uncorrelated with regression error due to random assignment to intent to treat group – Correlated with endogenous variable since being contacted during experiment is a function of being in the intent to treat goup

Field Experiments of Political Participation (Green, Gerber, Shachar) Contact a random sample of registered voters either by mail or face-to-face in 1998 – We know whether the subjects voted in 1996 Encourage people to vote Check polling data to see if they voted in the next election (1998 and 1999)

Results

Natural Experiment (Lassen) Question: Do more informed people vote, or do people vote become more informed? – What is the causal relationship between information and voting? Research Design: Find a natural experiment in Copenhagen, Denmark, where 4 out of 15 districts carried out a pilot project on decentralization (PCD) before a vote on whether to decentralize