1. 28 April 2009 @ Crawford School 1 Causality and Causal Inference Semester 1, 2009 POGO8096/8196: Research Methods Crawford School of Economics and Government

2. 228 April 2009 @ Crawford School This week Research design and causal thinking Research design and causal thinking What is a research design? What is a research design? Conditions of causality Conditions of causality Research designs for causal thinking Research designs for causal thinking True experimental design True experimental design Quasi-experimental design Quasi-experimental design Correlation design Correlation design (Design without a control group) (Design without a control group)

3. 328 April 2009 @ Crawford School What is a research design? Narrow definition [today’s topic] Narrow definition [today’s topic] Research design refers to the logical method by which we propose to test a hypothesis. Research design refers to the logical method by which we propose to test a hypothesis. Broad definition Broad definition Research design refers to a whole proposal for a research project, including the review of the literature, research questions and hypotheses, details of data collection, methods of data analysis, expected outcomes, a budget proposal, etc. Research design refers to a whole proposal for a research project, including the review of the literature, research questions and hypotheses, details of data collection, methods of data analysis, expected outcomes, a budget proposal, etc.

4. 428 April 2009 @ Crawford School Why do we need it? In empirical research, we often want to test a causal theory (or more specifically, a causal hypothesis) stating that X affects Y. In empirical research, we often want to test a causal theory (or more specifically, a causal hypothesis) stating that X affects Y. But … How do we know that X causes Y, but not the other way around? How do we know that it is X that causes Y, but not something else (W, Z or whatever)? But … How do we know that X causes Y, but not the other way around? How do we know that it is X that causes Y, but not something else (W, Z or whatever)? We need a valid research design: a method with which we can test the causal hypothesis. We need a valid research design: a method with which we can test the causal hypothesis.

5. 528 April 2009 @ Crawford School Conditions for causality Co-variation (or Correlation): Two phenomena (i.e., variables) tend to be related. If one of the two variables changes, another variable changes. [Necessary condition for causality.] Co-variation (or Correlation): Two phenomena (i.e., variables) tend to be related. If one of the two variables changes, another variable changes. [Necessary condition for causality.] Time-order: The presumed cause (an independent variable) happened before the presumed consequence (a dependent variable). Time-order: The presumed cause (an independent variable) happened before the presumed consequence (a dependent variable). Non-spuriousness: The co-variation between independent and dependent variables is not caused by other factors. Non-spuriousness: The co-variation between independent and dependent variables is not caused by other factors.

6. 628 April 2009 @ Crawford School Research designs for causal inference True experimental design True experimental design Almost always quantitative Almost always quantitative Quasi experimental design Quasi experimental design Often quantitative Often quantitative “natural experiment” in Shively “natural experiment” in Shively Correlational design Correlational design Both qualitative (intentional selection of observations) and quantitative (correlation and regression analysis) Both qualitative (intentional selection of observations) and quantitative (correlation and regression analysis) “natural experiment without pre-measurement” in Shively “natural experiment without pre-measurement” in Shively

7. 728 April 2009 @ Crawford School True experiment – 1 Physical, biological and medical sciences, as well as some social sciences (e.g., psychology), have traditionally used true experiments. It is becoming popular in other social science disciplines. Physical, biological and medical sciences, as well as some social sciences (e.g., psychology), have traditionally used true experiments. It is becoming popular in other social science disciplines. It is very important to understand how an experiment is set up, because the logic involved is highly relevant to all types of research design for causal thinking. It is very important to understand how an experiment is set up, because the logic involved is highly relevant to all types of research design for causal thinking.

8. 828 April 2009 @ Crawford School True experiment – 2 The classic (simplest) experimental design The classic (simplest) experimental design 1. Select a sample of your experimental study. 2. Randomly divide the subjects (e.g., students) into two groups: the “experimental (or treatment) group” and the “control group.” 3. Give a “stimulus (or treatment)” only to the experimental group. 4. Measure the dependent variable and compare the difference in the dependent variable between the groups.

9. 928 April 2009 @ Crawford School “Random Assignment” Note: In this course, “matching” means something different (= the intentional selection of observations).

10. 1028 April 2009 @ Crawford School Types of true experiments Laboratory experiment Laboratory experiment Small-scale experiment in artificial setting Small-scale experiment in artificial setting The effects of negative TV campaigns on voter’s preference of presidential candidates and on voter turnout (Ansolabehere and Iyenger 1995). The effects of negative TV campaigns on voter’s preference of presidential candidates and on voter turnout (Ansolabehere and Iyenger 1995). Field experiment Field experiment Large-scale experiment in “real” setting Large-scale experiment in “real” setting The effects of personal canvassing, telephone calls, and direct mail on voter turnout (Gerber and Green 2000). The effects of personal canvassing, telephone calls, and direct mail on voter turnout (Gerber and Green 2000).

11. 1128 April 2009 @ Crawford School Merits Well-structured experiments can control other factors almost perfectly, because the attributes of experimental and control groups (observable and unobservable) are, on average, the same. Well-structured experiments can control other factors almost perfectly, because the attributes of experimental and control groups (observable and unobservable) are, on average, the same. They also satisfy the “time-order” condition of causality. They also satisfy the “time-order” condition of causality. Thus, “only randomization [i.e., a true experiment with random assignment] provides a clear enough causal interpretation to settle issues of social-scientific research conclusively” (Shively, p. 87). Thus, “only randomization [i.e., a true experiment with random assignment] provides a clear enough causal interpretation to settle issues of social-scientific research conclusively” (Shively, p. 87).

12. 1228 April 2009 @ Crawford School Limitations In both laboratory and field experiments In both laboratory and field experiments Experimental research is often limited to investigations of political and social communications. Experimental research is often limited to investigations of political and social communications. It may be ethically inappropriate to conduct experiments with human beings, societies and politics as subjects. It may be ethically inappropriate to conduct experiments with human beings, societies and politics as subjects. In laboratory experiments In laboratory experiments Samples in an experiment may not represent the population. Samples in an experiment may not represent the population. Experiments are conducted in an artificial setting. Experiments are conducted in an artificial setting.

13. 1328 April 2009 @ Crawford School Quasi experiment The basic quasi experimental design The basic quasi experimental design 1. Measure the dependent variable of the subjects. 2. Wait until some subjects are exposed to the independent variable, or observe that the values of the independent variable change among the subjects. (Note: The subjects are not assigned to groups: no random assignment.) 3. Measure the dependent variable again, and compare the difference in the dependent variable among the subjects.

14. 1428 April 2009 @ Crawford School

15. 1528 April 2009 @ Crawford School Other examples “Panel” studies – the same set of people are surveyed multiple times. “Panel” studies – the same set of people are surveyed multiple times. “Before-and-after” studies in policy-oriented research. “Before-and-after” studies in policy-oriented research. e.g., the effects of increasing speed limits on the number of traffic fatalities. (Not all states in the US increased the speed limits.) e.g., the effects of increasing speed limits on the number of traffic fatalities. (Not all states in the US increased the speed limits.) Note: There should be a “control group” in these research. Note: There should be a “control group” in these research.

16. 1628 April 2009 @ Crawford School Merits and Limitations They satisfy the “time-order” condition – The value of a dependent variable changes after the value of a key independent variable is observed (or changed). They satisfy the “time-order” condition – The value of a dependent variable changes after the value of a key independent variable is observed (or changed). Important: They can control “subject-specific, time- invariant” variables, but not others. Important: They can control “subject-specific, time- invariant” variables, but not others. You should carefully examine what other factors might affect the causal relationship and try to control them (if possible). You should carefully examine what other factors might affect the causal relationship and try to control them (if possible).

17. 1728 April 2009 @ Crawford School Correlational design The basic correlational design The basic correlational design 1. Select a sample of your study. 2. Measure the dependent and independent variables. (Note: The values of the independent variables must vary among the subjects.) 3. If the dependent variable differs among the subjects, ascribe this to the effect of the independent variable.

18. 1828 April 2009 @ Crawford School

19. 1928 April 2009 @ Crawford School Other examples Studies examining how survey respondents with different attributes vote differently. Studies examining how survey respondents with different attributes vote differently. Studies examining how countries with different economic conditions experience different environmental issues and conflicts. Studies examining how countries with different economic conditions experience different environmental issues and conflicts. Studies examining how Indonesian local governments with different social conditions affect their budget performance. Studies examining how Indonesian local governments with different social conditions affect their budget performance.

20. 2028 April 2009 @ Crawford School Merits and Limitations The “time-order” condition is not always satisfied. We need “auxiliary” information (e.g., common sense, prior knowledge, etc.) to judge the direction of causality. The “time-order” condition is not always satisfied. We need “auxiliary” information (e.g., common sense, prior knowledge, etc.) to judge the direction of causality. A correlational design with just one independent variable fails to control other factors. A correlational design with just one independent variable fails to control other factors. You should carefully examine what other factors might affect the causal relationship and try to control them. You should carefully examine what other factors might affect the causal relationship and try to control them.

21. 2128 April 2009 @ Crawford School Design without a control group A typical design without a control group A typical design without a control group 1. Measure a certain phenomenon or behavior (A) you want to explain. 2. Observe a certain phenomenon, behavior or other exogenous shocks (B) that you think as a “cause”. 3. Measure the phenomenon or behavior (A’) you want to explain again. If the phenomenon or behavior has changed from A to A’, then ascribe this change to the occurrence of B.

22. 2228 April 2009 @ Crawford School An Example Taiwan presidential election in 2000. Taiwan presidential election in 2000. The pre-election level of support for each candidate. The pre-election level of support for each candidate. China’s military actions near Taiwan, which are intended to affect the election result. China’s military actions near Taiwan, which are intended to affect the election result. The results of the presidential election The results of the presidential election The Democratic Progressive Party’s Chen Shui-bian won The Democratic Progressive Party’s Chen Shui-bian won The ruling Kuomingtang’s Lien Chan lost. The ruling Kuomingtang’s Lien Chan lost. Many argue that China’s attempts were counterproductive and helped Chen to win. Really? Many argue that China’s attempts were counterproductive and helped Chen to win. Really?

23. 2328 April 2009 @ Crawford School Control group Important: A control “group” and a control “variable” are different. If there is no control group, your independent variable does not vary. Important: A control “group” and a control “variable” are different. If there is no control group, your independent variable does not vary. A control group = a portion of your observations (the subjects of your study) that are not exposed to your key independent variable. A control group = a portion of your observations (the subjects of your study) that are not exposed to your key independent variable. A control variable = another factor explaining the dependent variable. A control variable = another factor explaining the dependent variable.

24. 2428 April 2009 @ Crawford School Examples An example of true experiments: An example of true experiments: Taking an introductory Australian Government course (X) increases political interest (Y).  “Students who do not take the course.” Taking an introductory Australian Government course (X) increases political interest (Y).  “Students who do not take the course.” An example of natural experiments: An example of natural experiments: Watching a presidential debate (X) increases intensity of support (Y).  “Students who do not watch the debate.” Watching a presidential debate (X) increases intensity of support (Y).  “Students who do not watch the debate.” An example of correlational designs: An example of correlational designs: Voter turnout (Y) is lower in urban areas (X).  “Rural areas.” Voter turnout (Y) is lower in urban areas (X).  “Rural areas.”

25. 2528 April 2009 @ Crawford School Limitations Research design without a control group is very common in historical descriptions of political, social and economic events. This method may be useful to understand historical processes and details, but it is not recommended for causal analysis. Why? Because without a control group you cannot say that a variation in Y is caused by a variation in X (because there is no variation in X). Research design without a control group is very common in historical descriptions of political, social and economic events. This method may be useful to understand historical processes and details, but it is not recommended for causal analysis. Why? Because without a control group you cannot say that a variation in Y is caused by a variation in X (because there is no variation in X). They may, however, help us identify some new measures, theories, and/or puzzles. They may, however, help us identify some new measures, theories, and/or puzzles.

26. 2628 April 2009 @ Crawford School Summary Random Assignment Pre- measurement Control Group True experimental design YesYes/NoYes Quasi experimental design NoYesYes Correlational design NoNoYes Design without a control group NoYesNo

27. 2728 April 2009 @ Crawford School Next week Intentional selection of observations Intentional selection of observations How to choose observations? How to choose observations? How to avoid problematic causal inference? How to avoid problematic causal inference? Some controversies Some controversies Observations vs. cases Observations vs. cases Objectives of qualitative research Objectives of qualitative research