Research Design Week 6 Part February 2011 PPAL 6200

Outline Observation vs. Experiment Subjects, Factors, Treatments Good and Bad Experiments Randomized Comparative Experiments –Their Logic Cautions about Experiments Matched Pairs and other Block Designs

Observation vs. Experiments An Experiment is a special kind of research study that imposes or applies a treatment in order to observe responses so as to determine if the treatment causes a change in response An observational study observes variation in individual variables of interest but does not attempt to manipulate them in order to influence or produce responses

In many cases social scientists are interested in, it is probably not ethical to conduct genuine experiments.

Subjects, Factors, Treatments The book describes a “study” as “an experiment when we actually do something to people, animals,… in order to observe the response.” Not sure. Another view is that a study is any systematic investigation.

Some key words we have to remember –Subject –Explanatory Variables or Factors, or Independent Variables –Response Variables, or Dependent Variables –Treatment (can be a combination of Factors)

Experiments Good and Bad The benefit of experiments is that they allow for “control”, we try to manipulate one or some variables (the treatment) so as to produce change in another variable (the response) while holding everything else constant If all the variables that can impact the response are not controlled, the results can be compromised.

Three types of Control Physical control Statistical control Logical

The Double Blind Randomized Comparative Experiment This is the “gold standard” in research design often used in medical and pharmaceutical research. You are interested if treatment “x” can cure a disease. You draw a random sample of the population of those suffering the disease and someone separate from the researchers randomly assigns individuals to either the experimental or control group. You then administer the treatment to the experimental group but provide either a placebo, or more likely, an already approved treatment to the control group so that they also don’t know if they are receiving the treatment or not. Neither researchers nor subjects know who is in the treatment group and who is in the control group You then compare the results for the two groups

The Logic of a Randomized Comparative Experiment Randomized selection and assignment to treatment and control groups mean all variations among the cases ought to be equally represented in each group Therefore, any differences in average response must be due either to the treatment or (as occasionally discovered pure random chance). However, if you use enough subjects random chance ought not to play a role.

As we noted before, there is always a chance that results are the product of pure chance. A result that is so strong that it is unlikely to be the product of pure random chance is said to be statistically significant.

Matched pairs and other block designs Sometimes the gold standard is not possible but other experimental designs are. Essentially this is like a randomized comparative experiment except that the assignment of individuals is done one pair at a time so randomization is not as thorough.

Some things not in the book In order to develop a social science project we need to look at some of the steps involved in moving from an idea to a completed project.

The research development process Stage 1 A Problem or Observation A Question Background Research –Generally a literature of previous research on topics related to your question.

Stage 2 A Hypothesis –A statement that something (the explanatory or independent variable[s]) causes or influences variation in something else (the dependent or response variable), and possibly how this is done. To do this you must have a conceptual definition of your variables

Stage 3 Operationalizing your variables –How are you going to measure your variables –How are you going to collect those measurements

How you go about collecting data depends on the problem and question you began with In many cases the literature review you have conducted will reveal not only what people have found on topics related to the question you have, but also how people have gone about studying these topics as well.

Common Research Designs Small “n” vs. Large “n” designs Small “n” –Small “n” comparative Mill’s Methods of Agreement and Disagreement –The Case Study –Small “n” elite Position of individuals –Participant Observer Researcher’s own senses –Key point, in all of these control is logical

Large “n” Designs –Survey Research (Opinions) –Econometric (National Accounts and Policies) –Demographic. –Quantitative Content Analysis –All of these can involve either original data collection or the secondary analysis of previously collected data. –Key point, in all of these control is exercised statistically.

If you do collect original data… Two things you should be concerned about The validity and reliability of your operationalizations –Pilot Studies The comparability of your work to that of others Make life simpler for yourself, look for questions that have been previously used by other researchers which have been field tested and validated, and to which comparisons of findings can be made.

Stage 4 Executing your design

Stage 5 Analyzing your data

Stage 6 Drawing your conclusions

Stage 7 Writing your report

A funny thing called ethics As public servants are required to work in the public interest, this means respecting whatever codes of research ethics are in place in your work place The gold standards of ethics –informed consent

The Tri-Council Policy Sets out the basic policies that should govern research on human subjects at all institutions that receive funding from the Federal government, such as York University. Sets out a general standard of ethical conduct built around “informed consent” of all subjects who participate in research.