3 March Crawford School 1 Empirical Research Design – 2 Semester 1, 2009 POGO8096/8196: Research Methods Crawford School of Economics and Government

23 March Crawford School This week Research topics and questions Research topics and questions How to choose a research topic? How to choose a research topic? Normative, empirical, and analytical questions Normative, empirical, and analytical questions What constitutes a “good” research question? What constitutes a “good” research question? Theories, hypotheses and operational definitions Theories, hypotheses and operational definitions What is a (causal and empirical) theory? What is a (causal and empirical) theory? What constitutes a “good” theory? What constitutes a “good” theory? What are hypotheses and operational definitions? What are hypotheses and operational definitions?

33 March Crawford School Choosing a topic – 1 Picking a research topic is the critical decision in doing research but is difficult to teach how to pick it. Picking a research topic is the critical decision in doing research but is difficult to teach how to pick it. This task requires not only that you are familiar with as broad a range of existing studies and issues as possible, but that you also have some idea of where an existing body of research (or an existing argument in policy debate) is weakest. This task requires not only that you are familiar with as broad a range of existing studies and issues as possible, but that you also have some idea of where an existing body of research (or an existing argument in policy debate) is weakest.

43 March Crawford School Choosing a topic – 2 Critically read and critically think. Critically read and critically think. You should identify what the bottom-line argument (i.e., X causes Y) is in each article, book or policy debate. You should identify what the bottom-line argument (i.e., X causes Y) is in each article, book or policy debate. Can you think of any “alternative” argument? Can you think of any “alternative” argument? Do you think the argument is applicable to other data, which the author does not use (e.g., data from Asian countries)? Do you think the argument is applicable to other data, which the author does not use (e.g., data from Asian countries)? Any puzzle? Any puzzle? One often effective way of looking at choosing a topic is to work on something in our lives, which puzzles us. One often effective way of looking at choosing a topic is to work on something in our lives, which puzzles us.

53 March Crawford School Examples from my research Why do Japanese village elections often record very high voter turnout (e.g., over 95%) even without a compulsory voting system? Why do Japanese village elections often record very high voter turnout (e.g., over 95%) even without a compulsory voting system? Do policy debates between parties promote active civic participation in an election (as often advocated by the media)? Do policy debates between parties promote active civic participation in an election (as often advocated by the media)? Does natural-resource abundance truly hinder democratization (as argued by Ross and others)? Does natural-resource abundance truly hinder democratization (as argued by Ross and others)?

63 March Crawford School Remarks Should we produce a “great work”? Should we produce a “great work”? There exists what is called “publication bias.” Intuitive and no-surprise findings are often not published. There exists what is called “publication bias.” Intuitive and no-surprise findings are often not published. “The contribution of even a minor article is greater than that of the ‘great work’ that stays forever in a desk drawer or within the confines of a computer” (KKV, p. 9). “The contribution of even a minor article is greater than that of the ‘great work’ that stays forever in a desk drawer or within the confines of a computer” (KKV, p. 9).

73 March Crawford School Q & A When you read existing studies, understand … When you read existing studies, understand … What is the author’s research question? What is the author’s research question? What is the author’s answer to the question. What is the author’s answer to the question. When you conduct own research, explain … When you conduct own research, explain … What is your research question? What is your research question? What is your answer to the question? What is your answer to the question? Is the author’s/your question, … Is the author’s/your question, … empirical, normative, or analytical? empirical, normative, or analytical?

83 March Crawford School Normative questions A normative question is a question examining what is desirable and undesirable. The answers depend on the value judgments of the individual who answers the question (i.e., of a researcher). A normative question is a question examining what is desirable and undesirable. The answers depend on the value judgments of the individual who answers the question (i.e., of a researcher). Typical in normative philosophy or in policy debates Typical in normative philosophy or in policy debates Examples Examples Is democracy the best form of government? Is democracy the best form of government? Should the Reserve Bank cut the interest rate? Should the Reserve Bank cut the interest rate?

93 March Crawford School Analytical questions An analytical question is a question examining the validity of a proposition, which is completely dependent on a set of assumptions and definitions. An analytical question is a question examining the validity of a proposition, which is completely dependent on a set of assumptions and definitions. Typical in formal theory (and mathematics) Typical in formal theory (and mathematics) An example An example Does the system of simple majority rule voting always give an unambiguous social choice between alternatives? Does the system of simple majority rule voting always give an unambiguous social choice between alternatives?

103 March Crawford School Empirical questions An empirical question is a question examining what is true or untrue in the real world. The question can be supported or rejected by observations and/or experience. An empirical question is a question examining what is true or untrue in the real world. The question can be supported or rejected by observations and/or experience. Typical in theory-oriented or policy-oriented empirical research Typical in theory-oriented or policy-oriented empirical research Examples Examples Does economic development cause democratization? Does economic development cause democratization? Did the tax reform flatten income distribution? Did the tax reform flatten income distribution?

113 March Crawford School Notes Your empirical question should be, in principal, answerable without “your” value judgment and with observable information (e.g., government statistics, findings from a survey, findings from an interview, your participant observations, other data sources, etc.). Your empirical question should be, in principal, answerable without “your” value judgment and with observable information (e.g., government statistics, findings from a survey, findings from an interview, your participant observations, other data sources, etc.). Note that you never know what is true or untrue in the future: Your question should not be about something in the future. Note that you never know what is true or untrue in the future: Your question should not be about something in the future.

123 March Crawford School Should we always ask empirical questions? Many important topics in social sciences are concerned with normative questions about what ought to be in societies, economies and politics. But most social scientists today are concerned with empirical questions. This does not mean that social scientists think normative issues as unimportant. Many important topics in social sciences are concerned with normative questions about what ought to be in societies, economies and politics. But most social scientists today are concerned with empirical questions. This does not mean that social scientists think normative issues as unimportant. We may be able to deal with normative issues indirectly by examining empirical questions. How? We may be able to deal with normative issues indirectly by examining empirical questions. How?

133 March Crawford School Reformulating questions Method 1: change the frame of reference Method 1: change the frame of reference Is Kevin Rudd doing a good job? Is Kevin Rudd doing a good job? Does the public think that John Howard is doing a good job? Does the public think that John Howard is doing a good job? Method 2: test the underlying argument Method 2: test the underlying argument Should the Research Bank cut the interest rate to boost economy? Should the Research Bank cut the interest rate to boost economy? Did an increase in the interest rate (under similar conditions) boost economy? Did an increase in the interest rate (under similar conditions) boost economy?

143 March Crawford School Good empirical questions Clarity: It must be clear and reasonably specific. Clarity: It must be clear and reasonably specific. Testability: It must be a question that can be investigated given available empirical resources. Testability: It must be a question that can be investigated given available empirical resources. Theoretical significance and/or practical relevance: It must be important theoretically and/or practically. Theoretical significance and/or practical relevance: It must be important theoretically and/or practically. Originality: It should have at least some degree of originality. Originality: It should have at least some degree of originality.

153 March Crawford School Examples: good or bad? What is today’s political and economic relationship between China and ASEAN? What is today’s political and economic relationship between China and ASEAN? What fraction of people in North Korea support the current Kim Chong-il regime? What fraction of people in North Korea support the current Kim Chong-il regime? Should Australia’s state upper houses of parliament be abolished? Should Australia’s state upper houses of parliament be abolished? Why did a mayor of City A in NZ resign? Why did a mayor of City A in NZ resign? Is the China and Taiwan reunification possible? Is the China and Taiwan reunification possible?

163 March Crawford School Remarks If you can formulate a “good” question (i.e., a clear, testable, important, and original question), then a half of your research is done. The rest of your research is collecting pieces of information and presenting your answer. If you can formulate a “good” question (i.e., a clear, testable, important, and original question), then a half of your research is done. The rest of your research is collecting pieces of information and presenting your answer. Formulating a “good” question is, however, not easy. Formulating a “good” question is, however, not easy.

173 March Crawford School An empirical theory A (causal and empirical) theory: “If X happens, Y will follow.” A (causal and empirical) theory: “If X happens, Y will follow.” Dependent variable (Y) = some phenomenon that is to be explained. Dependent variable (Y) = some phenomenon that is to be explained. Independent variable (X) = one or more factors that are thought to affect the dependent variable. Independent variable (X) = one or more factors that are thought to affect the dependent variable. No variable is innately either independent or dependent. No variable is innately either independent or dependent. Discussion: Why do we need such a theory? Discussion: Why do we need such a theory?

183 March Crawford School Examples – political science If a country uses the single-member district system, the effective number of political parties is expected to shrink to two. If a country uses the single-member district system, the effective number of political parties is expected to shrink to two. If a country increases its military expenditure, it is expected to go to war. If a country increases its military expenditure, it is expected to go to war. Countries with higher per capita incomes are more likely to be democracies. Democracies do not tend to initiate wars. * An example showing that no variable is innately dependent or independent. Countries with higher per capita incomes are more likely to be democracies. Democracies do not tend to initiate wars. * An example showing that no variable is innately dependent or independent.

193 March Crawford School Good theory – 1 “Simplicity” “Simplicity” A theory should give us as simple as a handle on the universe as possible. A theory should give us as simple as a handle on the universe as possible. Note that theories are often stated in a “conditional” manner: Y is caused by X, when all the other variables are controlled. Note that theories are often stated in a “conditional” manner: Y is caused by X, when all the other variables are controlled. Independent variables = “key” causal variable/s (often denoted as X) + other “control” variables (often denoted as Z). Independent variables = “key” causal variable/s (often denoted as X) + other “control” variables (often denoted as Z).

203 March Crawford School Good theory – 2 “Predictive accuracy” “Predictive accuracy” Given available data, a theory should make accurate prediction (i.e., X does, indeed, cause Y). Given available data, a theory should make accurate prediction (i.e., X does, indeed, cause Y). Note: “Prediction” does not necessarily mean that you have to predict something in the future. If a set of independent variables, to a considerable extent, explains the dependent variable (i.e., events, actions, opinions measured in the past), your theory has high predictive accuracy. Note: “Prediction” does not necessarily mean that you have to predict something in the future. If a set of independent variables, to a considerable extent, explains the dependent variable (i.e., events, actions, opinions measured in the past), your theory has high predictive accuracy.

213 March Crawford School Good theory – 3 “Importance” “Importance” A theory should be applicable broadly and generally to a variety of situations and countries. A theory should be applicable broadly and generally to a variety of situations and countries. A specific theory should also make contributions to a variety of preexisting theories that are affected by the new theory. For example, a theory on electoral change would be more valuable if it could have implications of other broader areas of theories, e.g., democratic theory. A specific theory should also make contributions to a variety of preexisting theories that are affected by the new theory. For example, a theory on electoral change would be more valuable if it could have implications of other broader areas of theories, e.g., democratic theory.

223 March Crawford School Elegance If a theory meets the three criteria, the theory can be described as elegant. If a theory meets the three criteria, the theory can be described as elegant. An example: Making Democracy Work An example: Making Democracy Work What explains the effectiveness of governmental institutions? What explains the effectiveness of governmental institutions? The root determinant of effective government is civic community – participation by people in non- profit, non-governmental social activities. The root determinant of effective government is civic community – participation by people in non- profit, non-governmental social activities. The theory is simple. It is important, and it makes accurate prediction. The theory is simple. It is important, and it makes accurate prediction.

233 March Crawford School Beautiful surprise A good theory gives us a “beautiful surprise.” A good theory gives us a “beautiful surprise.” A piece of research, which goes against our expectations or makes us rethink our world, gives us a special kind of pleasure. A piece of research, which goes against our expectations or makes us rethink our world, gives us a special kind of pleasure. Examples Examples Tougher border controls increase the number of illegal immigrants. Tougher border controls increase the number of illegal immigrants. Participation by people in community activities enhance the effectiveness of local government administration. Participation by people in community activities enhance the effectiveness of local government administration.

243 March Crawford School Theories and hypotheses A theory is a set of generalizations about a topic (often a set of generalized causal relationships between phenomena: “X causes Y”). It is broad and general. A theory is a set of generalizations about a topic (often a set of generalized causal relationships between phenomena: “X causes Y”). It is broad and general. A hypothesis is an empirical statement derived from a theory (or an observable implication of a theory). It is focused and specific. A hypothesis is an empirical statement derived from a theory (or an observable implication of a theory). It is focused and specific.

253 March Crawford School Variables and operational definitions A variable is an empirical property that can take on two or more values. A variable is an empirical property that can take on two or more values. A theory is a statement about general (and often conceptual) variables. A theory is a statement about general (and often conceptual) variables. A hypothesis is a statement about specific variables. A hypothesis is a statement about specific variables. An operational definition is a set of directions as to how the variables in a hypothesis are to be observed and measured. An operational definition is a set of directions as to how the variables in a hypothesis are to be observed and measured.

263 March Crawford School Next session – 31 March Levels of research and unit of analysis Levels of research and unit of analysis Problems of measurement: Problems of measurement: The basic problem of measurement The basic problem of measurement Reliability and validity Reliability and validity Precision in measurement and measures Precision in measurement and measures Quantitative vs. qualitative measures Quantitative vs. qualitative measures * Guest lectures on 10, 17 and 24 March.