5 May Crawford School 1 Comparative Case Studies – 1 Semester 1, 2009 POGO8096/8196: Research Methods Crawford School of Economics and Government

25 May Crawford School This week Intentional selection of observations Intentional selection of observations How to choose observations? How to choose observations? How to avoid problematic causal inference? How to avoid problematic causal inference? Some controversies Some controversies Observations vs. cases Observations vs. cases Objectives of qualitative research Objectives of qualitative research

35 May Crawford School Causal thinking: Review A hypothesis: X causes Y A hypothesis: X causes Y X = a key independent variable X = a key independent variable Y = a dependent variable Y = a dependent variable To claim that X causes Y, … To claim that X causes Y, … The effects of the “other” variables (Z) should be controlled. The effects of the “other” variables (Z) should be controlled. X and Z should be causally prior to Y. X and Z should be causally prior to Y. (And, of course, X and Y must be correlated.) (And, of course, X and Y must be correlated.)

45 May Crawford School True experimental design True experimental design An ideal design but applicability is limited. An ideal design but applicability is limited. Quasi-experimental/correlational designs Quasi-experimental/correlational designs Both designs are commonly used not only in quantitative studies but also in qualitative studies. Both designs are commonly used not only in quantitative studies but also in qualitative studies. Design without a control group Design without a control group = a design without a variation in X. = a design without a variation in X. Research designs: Review

55 May Crawford School Controlling other factors Then, how to control other factors in studies other than true experiments. Then, how to control other factors in studies other than true experiments. Quantitative researchers collect a large number of observations with more than one independent variable and, then, apply a technique of “multivariate statistical analysis.” Quantitative researchers collect a large number of observations with more than one independent variable and, then, apply a technique of “multivariate statistical analysis.” Qualitative researchers conduct “intentional selection of observations” – data collection and analysis proceeding together. Qualitative researchers conduct “intentional selection of observations” – data collection and analysis proceeding together.

65 May Crawford School Intentional selection of observations The “best” intentional selection of observations: The “best” intentional selection of observations: 1. Choose observations so that other factors (Z) are as similar as possible – to avoid “omitted variable bias”. 2. Choose X and Z that are causally prior to Y – to avoid “simultaneous bias”. 3. You need to ensure variation in the key independent variable (X) without regard to the dependent variable (Y) – to avoid “selection bias”.

75 May Crawford School An example Y = The level of democracy in an African nation X = American “aid to democracy” in Africa X Z1Z1Z1Z1 Z2Z2Z2Z2 Design 1 Country A High Aid Country B Low Aid Design 2 Country A High Aid High Edu. Peaceful Country B Low Aid High Edu. Peaceful

85 May Crawford School Data collection stages Ideally, two stages of data collection. Ideally, two stages of data collection. 1. Given prior knowledge (i.e., implications from theories, previous empirical findings, common sense, your own experiences, etc), collect information about independent variables and decide appropriate observations for hypothesis testing. 2. Collect information about your dependent variable and compare observations.

95 May Crawford School Problems – 1 The difference between observations may be due to some significant “omitted variables.” The difference between observations may be due to some significant “omitted variables.” Why is Country B not getting aid from the US, if it has as favorable conditions as Country A? Why is Country B not getting aid from the US, if it has as favorable conditions as Country A? It is difficult, if not impossible, to find a proper set of observations. It is difficult, if not impossible, to find a proper set of observations. Can we find two African countries that are similar in many aspects but only one of them receiving aid from the US? Can we find two African countries that are similar in many aspects but only one of them receiving aid from the US?

105 May Crawford School Problems – 2 The control by intentional selection of observations can improve the likelihood of obtaining valid causal inferences, but we should acknowledge its limitations. The control by intentional selection of observations can improve the likelihood of obtaining valid causal inferences, but we should acknowledge its limitations. Practically, you can only control a few variables in qualitative comparative studies. Practically, you can only control a few variables in qualitative comparative studies. It is thus very important to think which of many potential “third factors” must be controlled. It is thus very important to think which of many potential “third factors” must be controlled.

115 May Crawford School What to control? – 1 What should be controlled? What should be controlled? Without thinking causal theories/hypotheses and reviewing extant empirical findings, you cannot decide which independent variables should be included or excluded in your analysis. Without thinking causal theories/hypotheses and reviewing extant empirical findings, you cannot decide which independent variables should be included or excluded in your analysis. Logical thinking (about causality) and careful review of existing studies are very important! Logical thinking (about causality) and careful review of existing studies are very important!

125 May Crawford School What to control? – 2 More specifically, More specifically, You should control Z if you expect that Z is correlated with BOTH X and Y. You should control Z if you expect that Z is correlated with BOTH X and Y. If you expect that Z is correlated with X (or Y) but not with Y (or X), omitting Z does not cause omitted variable bias, while including Z causes inefficiency. If you expect that Z is correlated with X (or Y) but not with Y (or X), omitting Z does not cause omitted variable bias, while including Z causes inefficiency. You should NOT control Z that happened after X and Y. You should NOT control Z that happened after X and Y.

135 May Crawford School “Selection bias” Another potential problem very common in qualitative comparative studies is a “selection bias” problem. Another potential problem very common in qualitative comparative studies is a “selection bias” problem. A typical selection bias problem: A typical selection bias problem: We know what we want to see as the outcome of the research (i.e., the confirmation of a favorable hypothesis). Then, intentionally or unintentionally, we select observations on the basis of combinations of X and Y that support the desirable conclusion. We know what we want to see as the outcome of the research (i.e., the confirmation of a favorable hypothesis). Then, intentionally or unintentionally, we select observations on the basis of combinations of X and Y that support the desirable conclusion.

145 May Crawford School Rules of selection – 1 How to avoid selection bias? How to avoid selection bias? General rules General rules The best way to avoid selection bias is to use a “random sampling” of (many) observations, but it is not always applicable. The best way to avoid selection bias is to use a “random sampling” of (many) observations, but it is not always applicable. If intentional selection is required, we should attempt to get these observations which are pivotal in deciding the question of interest, not those which merely support our position. If intentional selection is required, we should attempt to get these observations which are pivotal in deciding the question of interest, not those which merely support our position.

155 May Crawford School Rules of selection – 2 Specific rules Specific rules If there is no variation in the key independent variable, you cannot test your hypothesis (i.e., a research design without a control group.) If there is no variation in the key independent variable, you cannot test your hypothesis (i.e., a research design without a control group.) Selection of observations based on the values of the dependent variable causes biased inference. Selection of observations based on the values of the dependent variable causes biased inference. Selection of observations based on the values of the independent variable/s causes no biased inference. Selection of observations based on the values of the independent variable/s causes no biased inference.

165 May Crawford School Bad examples Inkeles and Rossi (1956) compared a number of industrialized nations and argued that industrialization leads to the particular prestige hierarchy of occupations. Inkeles and Rossi (1956) compared a number of industrialized nations and argued that industrialization leads to the particular prestige hierarchy of occupations. A research claiming that authoritarian rule leads to high economic growth rates, based on authoritarian observations with high growth rates and non-authoritarian observations with low growth rates. A research claiming that authoritarian rule leads to high economic growth rates, based on authoritarian observations with high growth rates and non-authoritarian observations with low growth rates.

175 May Crawford School Potential pitfalls: Summary You fail to control the effects of other significantly relevant variables.  “omitted variable bias” (i.e., problem associated with selection of variables) You do not select appropriate observations.  “selection bias” (i.e., problem associated with selection of observations) Your key independent variable is partly explained by your dependent variable.  “simultaneous bias” (an advanced topic not covered in this course).

185 May Crawford School Remark These problems are not specific to qualitative comparative studies. These problems are not specific to qualitative comparative studies. In quantitative/statistical studies, particularly ones with the random sampling of observations, the degree of selection bias is small. In quantitative/statistical studies, particularly ones with the random sampling of observations, the degree of selection bias is small. Except in true-experimental studies, omitted variable bias is always present to a greater or lesser degree. Except in true-experimental studies, omitted variable bias is always present to a greater or lesser degree. In correlational studies, particularly ones with survey data, simultaneous bias is common. In correlational studies, particularly ones with survey data, simultaneous bias is common.

195 May Crawford School Controversies Is a single-case study valid in making causal inference? Is a single-case study valid in making causal inference? Do we need to make causal inference in qualitative research? Do we need to make causal inference in qualitative research?

205 May Crawford School Observations vs. cases An observation An observation … is defined as an observed answer (to a question), behavior, or phenomenon for one unit for one dependent variable. It includes information on the values of the independent variables. … is defined as an observed answer (to a question), behavior, or phenomenon for one unit for one dependent variable. It includes information on the values of the independent variables. = the unit of analysis = the unit of analysis A case A case … can refer to a single unit, on which many variables are measured (= an observation). … can refer to a single unit, on which many variables are measured (= an observation). … can also refer even to a large domain for analysis. … can also refer even to a large domain for analysis.

215 May Crawford School Example – 1 Individuals Q1 (DV) Q2 (IV-1) Q3 (IV-2) Mr. A YesHighAgree Mr. B NoMediumDisagree Mrs. C YesLowDisagree Mrs. D YesHighAgree 4 observations and 3 variables Observations = individuals

225 May Crawford School Example – 2 A comparative study of modern wars. A comparative study of modern wars. Observations = modern wars Observations = modern wars World War II is one of the “observations.” World War II is one of the “observations.” The number of battle deaths may be a dependent variable. The number of battle deaths may be a dependent variable. A comparative study of battles during WW II A comparative study of battles during WW II Observations = battles during the WW II. Observations = battles during the WW II. World War II is a “case” that includes a large number of observations. World War II is a “case” that includes a large number of observations. The number of battle deaths may be a dependent variable. The number of battle deaths may be a dependent variable.

235 May Crawford School A single case study? A single case study? A single case study? A single “case” study is valid as long as there are more than one observation given that case. A single “case” study is valid as long as there are more than one observation given that case. A single “observation” study is totally invalid to make causal inference. (No variation in X!) A single “observation” study is totally invalid to make causal inference. (No variation in X!) What to choose? What to choose? You should choose “appropriate” observations for your comparative study. You should choose “appropriate” observations for your comparative study. Even in a single case study, you can increase the number of observations. Even in a single case study, you can increase the number of observations.

245 May Crawford School Qualitative research Manheim, Rich, Willnat (p ) Manheim, Rich, Willnat (p ) Qualitative researchers are less likely to be interested in testing a pre-formed hypothesis. Qualitative researchers are less likely to be interested in testing a pre-formed hypothesis. Qualitative researchers are usually unconcerned with “holding constant” some factors in order to make causal inference. Qualitative researchers are usually unconcerned with “holding constant” some factors in order to make causal inference. Qualitative researchers are often far less concerned with observing “representative” cases. Qualitative researchers are often far less concerned with observing “representative” cases.

255 May Crawford School Objectives – 1 Is it impossible/meaningless to make causal inference in qualitative research? Is it impossible/meaningless to make causal inference in qualitative research? No. As long as you have a hypothesis that you want to test, you should design an effective comparative research to make causal inference. No. As long as you have a hypothesis that you want to test, you should design an effective comparative research to make causal inference. Do we always have to make causal inference? Do we always have to make causal inference? No. There are multiple objectives to collect qualitative data. No. There are multiple objectives to collect qualitative data.

265 May Crawford School Objectives – 2 Often, a question can be better answered by “mixing” various research methods and collecting various types of empirical information. Often, a question can be better answered by “mixing” various research methods and collecting various types of empirical information. Preliminary (qualitative) data collection that helps you to formulate your hypothesis. Preliminary (qualitative) data collection that helps you to formulate your hypothesis. Qualitative data collection (even a single-observation study) to understand why statistical analysis shows the correlation between X and Y. Qualitative data collection (even a single-observation study) to understand why statistical analysis shows the correlation between X and Y. Understanding process, meaning, mechanism of social and human behaviors and phenomenon. Understanding process, meaning, mechanism of social and human behaviors and phenomenon.

275 May Crawford School Next session More on case studies (by Prof. Paul t’Hart) More on case studies (by Prof. Paul t’Hart) Characteristics of case studies Characteristics of case studies When to undertake case study research? When to undertake case study research? Challenges in case study research Challenges in case study research Sensible description and analysis in a case study Sensible description and analysis in a case study Bad case study research: traps to avoid Bad case study research: traps to avoid Good case study research: tips Good case study research: tips