Copyright © Allyn & Bacon (2007) A Second Look at Field Research Graziano and Raulin Research Methods: Chapter 13 This multimedia product and its contents are protected under copyright law. The following are prohibited by law: (1) Any public performance or display, including transmission of any image over a network; (2) Preparation of any derivative work, including the extraction, in whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2007) Why Do Field Research? Many reasons for doing field research Many reasons for doing field research –Test external validity of laboratory studies Do the laboratory findings really indicate what happens in the natural world? Do the laboratory findings really indicate what happens in the natural world? –Study events that only occur in the field Evaluate the effectiveness of social programs Evaluate the effectiveness of social programs –Improve generalization across settings By testing your hypotheses in those settings By testing your hypotheses in those settings Field research is becoming more common Field research is becoming more common

Copyright © Allyn & Bacon (2007) Difficulties in Field Research More difficult to control variables in field studies More difficult to control variables in field studies –Sometimes we have no control over the independent variable (e.g., a natural disaster) –Often we cannot randomly assign participants to conditions Even though we may not have complete control, we can often get sufficient control to allow us to draw valid conclusions Even though we may not have complete control, we can often get sufficient control to allow us to draw valid conclusions

Copyright © Allyn & Bacon (2007) Quasi-Experimental Designs Similar to, but not quite equal to, an experimental design Similar to, but not quite equal to, an experimental design –Approximates the control of an experiment –Researcher does not control key variables Manipulation of the independent variable Manipulation of the independent variable Assignment of participants to conditions Assignment of participants to conditions Two types of quasi-experimental designs Two types of quasi-experimental designs –Nonequivalent control-group design –Interrupted time-series design

Copyright © Allyn & Bacon (2007) Nonequivalent Control-Group Design First introduced in Chapter 10 as the pretest- posttest natural control group design First introduced in Chapter 10 as the pretest- posttest natural control group design The more similar the groups, the closer this design is to a true experiment The more similar the groups, the closer this design is to a true experiment

Copyright © Allyn & Bacon (2007) Nonequivalent Control-Group Design Sometimes participants are, for all practical purposes, randomly assigned by nature (essentially a natural experiment) Sometimes participants are, for all practical purposes, randomly assigned by nature (essentially a natural experiment) Other times, the groups are clearly different, although careful analysis and cautious interpretation may still produce valid conclusions Other times, the groups are clearly different, although careful analysis and cautious interpretation may still produce valid conclusions

Copyright © Allyn & Bacon (2007) Nonequivalent Control-Group Designs Sometimes it is possible to interpret the results from a nonequivalent control- group study, even when the groups are clearly different prior to the intervention Sometimes it is possible to interpret the results from a nonequivalent control- group study, even when the groups are clearly different prior to the intervention Only (b) and (f) represent data suggesting an experimental effect Only (b) and (f) represent data suggesting an experimental effect

Copyright © Allyn & Bacon (2007) Interrupted Time-Series Design A single group of participants is tested repeatedly before and after a manipulation or a natural event A single group of participants is tested repeatedly before and after a manipulation or a natural event –Essentially a single-group, pretest-posttest design with multiple before and after measures The multiple measures allow us to detect many confounding variables The multiple measures allow us to detect many confounding variables –Regression to the mean, history, maturation, etc.

Copyright © Allyn & Bacon (2007) Interrupted Time-Series Design

Copyright © Allyn & Bacon (2007) Relaxation Training and Disruptive Behavior

Copyright © Allyn & Bacon (2007) Adding a Control Group The interrupted time-series design does not need a control group The interrupted time-series design does not need a control group But adding a control group makes the design even more powerful But adding a control group makes the design even more powerful –Guerin and MacKinnon (1985) added two natural control groups in their study of the effects of a child car seat law in California Older children not covered by the law in California Older children not covered by the law in California Children of the same age in Texas Children of the same age in Texas

Copyright © Allyn & Bacon (2007) Guerin and MacKinnon (1985)

Copyright © Allyn & Bacon (2007) Program Evaluation One of the fastest growing areas of research One of the fastest growing areas of research –Demand to evaluate the effectiveness of costly programs before they are renewed –Program evaluation can provide field tests of laboratory theories Program evaluation uses the designs and controls already covered, with a few twists Program evaluation uses the designs and controls already covered, with a few twists –Making the designs work in the field is both a research and political challenge

Copyright © Allyn & Bacon (2007) Practical and Ethical Problems in Evaluations In program evaluation, the “participants” have not volunteered for research In program evaluation, the “participants” have not volunteered for research –Instead, they are a part of the program –Raises ethical issues of consent Including a no-treatment control group may raise both ethical and political questions Including a no-treatment control group may raise both ethical and political questions Treatment staff may be threatened by the evaluation or irritated by the extra workload Treatment staff may be threatened by the evaluation or irritated by the extra workload

Copyright © Allyn & Bacon (2007) Control in Program Evaluation Selecting appropriate measures Selecting appropriate measures –Normally need several dependent measures to evaluate all of the factors expected to change –Should include at least one satisfaction measure Avoiding bias in the evaluation Avoiding bias in the evaluation –Use objective measures whenever possible –Measure community-wide effects, which are less susceptible to response set biases Use strongest research design possible Use strongest research design possible

Copyright © Allyn & Bacon (2007) Program Evaluation Designs Randomized control-group design Randomized control-group design –Excellent control, but may be impossible Nonequivalent control-group design Nonequivalent control-group design –Natural control groups may add much to an otherwise weak design Single-group time-series design Single-group time-series design –Best alternative if no control group is available Pretest-posttest design Pretest-posttest design –A weak design; not recommended

Copyright © Allyn & Bacon (2007) Survey Research A widely used research technique A widely used research technique –Provides information about people’s attitudes, experience, and knowledge –Used extensively today by researchers, politicians, and news organizations First introduced in the 1830s in England to study the impact of the Industrial Revolution on people’s lives First introduced in the 1830s in England to study the impact of the Industrial Revolution on people’s lives

Copyright © Allyn & Bacon (2007) Types of Surveys Status surveys Status surveys –Descriptive survey about the current status of the population sampled –Descriptive information can guide policy and inform policy makers Survey research Survey research –Seeks to identify relationships among the variables studied in the survey

Copyright © Allyn & Bacon (2007) Steps in Survey Research 1. Determine what area of information is to be sought. 2. Define the population to be studied. 3. Decide how the survey is to be administered. 4. Construct the first draft of the survey instrument; edit and refine the draft.

Copyright © Allyn & Bacon (2007) Steps in Survey Research 5. Pretest the survey with a subsample; refine it further. 6. Develop a sampling frame and draw a representative sample. 7. Administer the final form of the instrument to the sample. 8. Analyze, interpret, and communicate the results.

Copyright © Allyn & Bacon (2007) The Survey Instrument Group-administered survey Group-administered survey –Written survey with instructions included –Administered in groups, through the mail, or on the Internet Individual survey schedule Individual survey schedule –Administered in person or over the phone –Can be rather complex, provided there is a clear procedure for the interviewer to follow –Allows clarification more easily than in group- administered surveys

Copyright © Allyn & Bacon (2007) Developing a Survey Instrument Use clear and explicit instructions Use clear and explicit instructions Types and number of questions will depend on the purpose and the type of survey planned Types and number of questions will depend on the purpose and the type of survey planned Types of items Types of items –Open-ended items –Multiple-choice items –Likert-scale items

Copyright © Allyn & Bacon (2007) Sample Open-Ended Questions How do you handle interpersonal difficulties with your co-workers? How do you handle interpersonal difficulties with your co-workers? What are the most important values to instill into today’s children? What are the most important values to instill into today’s children? If you were president, what issues would you make your top priority? If you were president, what issues would you make your top priority? What situations are particularly stressful for you? What situations are particularly stressful for you?

Copyright © Allyn & Bacon (2007) Sample Multiple-Choice Questions How frequently do you take a sick day from work? How frequently do you take a sick day from work? a) never b) once or twice a year c) 3 to 5 times a year d) 6 to 12 times a year e) at least once a month Identify the issue that you believe is most critical to this country’s future. Identify the issue that you believe is most critical to this country’s future. a) the economy b) education c) integrity in government d) national defense e) some other issue

Copyright © Allyn & Bacon (2007) Sample Likert-Scale Questions Rate each item on the scale shown to indicate your level of agreement: I believe in the Bill of Rights. I believe in the Bill of Rights. strongly agree agree uncertain disagree strongly disagree I think that everyone should vote. I think that everyone should vote. strongly agree agree uncertain disagree strongly disagree Most politicians cannot be trusted. Most politicians cannot be trusted. strongly agree agree uncertain disagree strongly disagree

Copyright © Allyn & Bacon (2007) Issues in Sampling Need to define the population and then develop a strategy for adequately sampling from the population Need to define the population and then develop a strategy for adequately sampling from the population –Identify each member of the population –Sample from the comprehensive list The more representative the sample, the more valid the conclusions from the survey The more representative the sample, the more valid the conclusions from the survey

Copyright © Allyn & Bacon (2007) Sampling Procedures Nonprobability samples Nonprobability samples –Convenience sample gathered quickly, but with little interest in drawing strong conclusions Probability samples Probability samples –Simple random sampling (Every person has an equal chance of being included) –Stratified random sampling Random sampling within clearly defined strata (subdivisions of the population) Random sampling within clearly defined strata (subdivisions of the population)

Copyright © Allyn & Bacon (2007) Sample Size Sample size is based on several factors Sample size is based on several factors –Costs and time constraints –Degree of precision needed Larger samples provide more precise estimates of population parameters Larger samples provide more precise estimates of population parameters More heterogeneous populations require larger samples More heterogeneous populations require larger samples The larger the sample, the more narrow the confidence intervals The larger the sample, the more narrow the confidence intervals

Copyright © Allyn & Bacon (2007) Research Design of Surveys Cross-sectional designs Cross-sectional designs –Administer the survey once to a sample Longitudinal designs Longitudinal designs –Repeatedly survey the same participants –Allows us to see changes in attitudes, experience, and knowledge over time –Difficult to get participants to submit to such long-term research

Copyright © Allyn & Bacon (2007) Field research is becoming more common Field research is becoming more common Field research need not be low constraint Field research need not be low constraint Types of field research Types of field research –Quasi-experimental designs –Program evaluation –Survey research Field studies: ultimate test of a hypothesis Field studies: ultimate test of a hypothesis –Is this how nature really works? Summary