1. Dr. Johnson. www.researchdemystified.org1 Other Research Approaches Research Method for Public Administrators Dr. Gail Johnson

2. Dr. Johnson. www.researchdemystified.org2 Research Complexity Not all research approaches fit into the Xs and Os framework Some approaches are really analytic techniques (e.g. cost-benefit analysis) Some approaches are data collection tools (e.g. survey research)

3. Dr. Johnson. www.researchdemystified.org3 Key Points about Other Approaches Each approach has unique characteristics and are useful in specific situations Each has strengths and limitations Each has fundamental requirements Researchers often uses a mix of approaches

4. Dr. Johnson. www.researchdemystified.org4 Commonly Used Research Approaches Secondary Data Analysis Evaluation Synthesis Content Analysis Survey Research Case Studies Cost-Benefit Analysis

5. Dr. Johnson. www.researchdemystified.org5 Secondary Analysis of Data In the planning phase, researchers should determine whether there is data they can use It is more efficient to use data collected by others if it meets the research requirements and is of high quality. For example: That National Longitudinal Survey of Youth collected data about drug use.

6. Dr. Johnson. www.researchdemystified.org6 Secondary Analysis of Data Large research studies gather data that is specifically related to the research questions and often gather other data that might be useful. The Framingham study focused on factors related to heart disease, tracking 5,000 people from 1983-2003. Other researchers used that data to look at why people are happy. Researchers Conclusion: happiness is contagious!

7. Dr. Johnson. www.researchdemystified.org7 Things to Consider Does the database contain the data needed to answer your research question? If the question is about drug use among high school seniors, then a study looking at drug use among adults will probably be inappropriate.

8. Dr. Johnson. www.researchdemystified.org8 Things to Consider How did they collect the data and does it meet research standards? How did they measure their key variables? Did they use documented procedures for selecting people, developing data collection instruments, and pre-testing?

9. Dr. Johnson. www.researchdemystified.org9 Things to Consider What quality control procedures were used to ensure the accuracy of the data in the database? What is the error rate? This means the difference between the data collected and the data in the database.

10. Dr. Johnson. www.researchdemystified.org10 Things to Consider How hard will it be to gain access to the data? Will the researchers provide all the documentation, including the data dictionary that specifies every variable and how every value for ever variable is coded? For example, are Men coded 1 and Women coded 2 or is it Women coded 1 and Men coded 2?

11. Dr. Johnson. www.researchdemystified.org11 Evaluation Synthesis Useful when there are many studies that have been done on a specific topic Nutrition programs for the poor to reduce infant mortality Early learning programs for disadvantaged children Impact of race on death penalty sentencing

12. Dr. Johnson. www.researchdemystified.org12 Evaluation Synthesis An expanded version of using available data Cheaper than original data collection Creates a larger data base, increasing the potential for drawing conclusions While individual studies often have limited ability to draw conclusions, if the findings of many small studies form a pattern, conclusions have greater credibility

13. Dr. Johnson. www.researchdemystified.org13 How to Do an Evaluation Synthesis Collect all studies that appear to be relevant Establish criteria to rate the quality of each of the studies Only those meeting the criteria are included Summarize the key findings and impact measures Look for similarities, especially among the highest quality studies

14. Dr. Johnson. www.researchdemystified.org14 Content Analysis Used to measure the amount and/or message of communication It can be used to analyze content of speeches, newspaper stories, emails, greeting cards, television, movies, blogs, memos, diaries or journals Subjects covered in MPA textbooks, amount of violence in Saturday morning cartoons, or images of political women portrayed in news journals

15. Dr. Johnson. www.researchdemystified.org15 Content Analysis In the narrowest definition, it converts qualitative data into quantitative data: It can count the number of stories about the Iraq war on major national network news shows to determine whether it changed over time It can count the number of times presidential candidates spoke about specific policy issues

16. Dr. Johnson. www.researchdemystified.org16 Content Analysis But it is often used to summarize qualitative data What were the major themes that emerged from the focus groups with teenaged mothers on welfare? Is there a discernable political bias in the editorials of the major newspapers (NY times, Washington post, Boston globe, LA times)?

17. Dr. Johnson. www.researchdemystified.org17 Content Analysis: Measurement Challenges Conceptual and operational definitions How would conceptually define “political bias”? Do you want to look at support for Republican or Democrat candidates or liberal or conservative policy positions?

18. Dr. Johnson. www.researchdemystified.org18 Content Analysis: Measurement Challenges How would you operationalize “liberal” and “conservative”—and are they they only political ideologies you want to include? What about libertarians, for example? What about fiscal conservatives who are liberal on social policy?

19. Dr. Johnson. www.researchdemystified.org19 Content Analysis: Measurement Challenges Conceptual and operational definitions How would you measure “media images” of women governors? How would you operationalize “strong” or”weak”, or “masculine” or “feminine”?

20. Dr. Johnson. www.researchdemystified.org20 Content Analysis: Reliability Challenges Coding Reliability: Will everyone code the same images in exactly the same way? Requires clear and very detailed decision rules Requires all team members are trained

21. Dr. Johnson. www.researchdemystified.org21 Assuring Reliability Jargon term: Inter-rater reliability Each team member will code a small sample of the same material Their coding will be compared and if there are differences, the team will figure out why there were differences and fix the procedures. Everyone should be coding the same material in the same way.

22. Dr. Johnson. www.researchdemystified.org22 Survey Research A data collection approach (discussed later under data collection) is used when the researchers want to determine the views of many people Typically, the exact same questions are asked with limited response options Think of a multiple choice exams (will cover more in data collection) The structure limits bias of interpretation and is easier to analyze

23. Dr. Johnson. www.researchdemystified.org23 Survey Research Open-ended questions—like fill-in-the blank exams—are possible but: Can be burdensome to complete if too many open-ended questions are asked Hard to analyze Bias is a threat.

24. Dr. Johnson. www.researchdemystified.org24 Examples of Survey Research Opinion polls to a cross-sectional sample of people in your town Employee attitude survey of everyone at your place of work Customer survey on the back of a postcard to all who visit the Post Office during the month of February

25. Dr. Johnson. www.researchdemystified.org25 Survey Research In-person Can be costly to do many in-person interviews But useful when people are not literate For example: it might be the best way to find out about health care access in rural villages in poor countries

26. Dr. Johnson. www.researchdemystified.org26 Survey Research Mail Requires mail address and participants who can read Cost of printing and postage Requires follow-up to get people to complete them

27. Dr. Johnson. www.researchdemystified.org27 Survey Research Phone. Useful in locations where most people have phones. Works well when the questions are simple (like who are you going to vote for in the next election) and can be done in less than 15 minutes. Random digit dialing is used to get a random sample; Even those with unlisted phones are called. Challenges of cell phones: pollsters are now including cell phones. But you have to call many, many numbers before you find someone who is willing to take the survey.

28. Dr. Johnson. www.researchdemystified.org28 Survey Research Email or Web-based Limited to those with access to this technology Requires email addresses Web-based limited to those with access Can work well in work setting or in associations where everyone has access Web-based applications often create the database, saving time and money

29. Dr. Johnson. www.researchdemystified.org29 Focus Groups Another data collection approach that is sometimes presented as a research design approach. (discussed in more detail under data collection) Small groups (6-12 people), trained facilitator moderates the discussion, few broad open-ended questions, food, pay to participate.

30. Dr. Johnson. www.researchdemystified.org30 Case Studies Useful in answering descriptive and normative questions Used in describing “best practices” Can focus on one or several people, groups, communities, organizations, processes, cities, etc. When it is not possible to gather data from every person, community, program or organization— Case Studies provide a way to say, “for example.”

31. Dr. Johnson. www.researchdemystified.org31 Case Studies Case studies are useful when the researchers want to gain an in-depth understanding of a situation, of how a complex process works, or lessons learned about implementing an innovative program It helps to make a problem or situation concrete: can tell stories of the impact of a program on a few people’s lives It can capture complexity and variation in a situation

32. Dr. Johnson. www.researchdemystified.org32 Choosing Case Studies How many? No fixed rules It will depend on the situation and available resources More is better to capture variation but case studies are always about taking one or a handful of examples from a much larger universe

33. Dr. Johnson. www.researchdemystified.org33 Choosing Case Studies Basis for selection Convenience: easy for researchers to access Particular aspects: best, typical, worst, or mixed cases Diversity: mix of locations, people, types of organizations or perspectives Random: removing researcher bias from decision

34. Dr. Johnson. www.researchdemystified.org34 Choosing Case Studies Researchers have to explain their rationale for whatever basis they use to select cases Selection should have “Face validity”: the choices should make sense to the average, reasonable person

35. Dr. Johnson. www.researchdemystified.org35 How to do a Case Study Case studies can employ a wide variety of methods to gather data and often use several in combination Triangulation: using different methods to gather information from various perspectives increases the validity of the results Can use available data, surveys, interviews, observations, analysis of files and records, content analysis of memos, cost-benefit analyses

36. Dr. Johnson. www.researchdemystified.org36 Case Studies: Are They Credible? Case studies are criticized because they typically have a small scope. This is true. But that is also true of classic experiments. You have to beyond “smallness”—and look at what they did and how they did it. Researchers should state the limitations of their work and conclusions should stay within the boundaries of their limitations.

37. Dr. Johnson. www.researchdemystified.org37 Case Studies: Do They Make Sense? Case studies are criticized because they are seen as inherently qualitative. Case studies may include quantitative data and may in fact be entirely quantitative. While case studies may be qualitative, there is nothing wrong with doing qualitative research as long as conclusions do not go beyond the limitations of the design.

38. Dr. Johnson. www.researchdemystified.org38 Case Studies in Public Administration Case studies make sense in public administration where it is difficult to do national studies because of insufficient resources (time, money, staff) Visiting six IRS offices may be far more doable than visiting 100

39. Dr. Johnson. www.researchdemystified.org39 Case Studies in Public Administration Like any research, quality is built into the design and procedures used: Relevant research questions Clear and appropriate measures Data collection strategies than minimize error and bias Honest about limitations Conclusions stay within those limitations

40. Dr. Johnson. www.researchdemystified.org40 Cost-Benefit Analysis Answers Normative Questions: Do the benefits justify the costs? Do the costs exceed the benefits? Assumes that the costs and the benefits can be meaningfully measured in $$Dollars$$

41. Dr. Johnson. www.researchdemystified.org41 Cost-benefit Analysis Is it cost beneficial to get a college degree? What are the costs? Tuition, books, room and board Opportunity cost: The money forgone from working fulltime in order to go to college The earnings if that money had been invested

42. Dr. Johnson. www.researchdemystified.org42 Cost-benefit Analysis Is it cost beneficial to get a college degree? What are the benefits? Difference in income between job with college degree and job without a college degree Calculated over 20 years Need to take inflation into account (remember to use constant dollars)

43. Dr. Johnson. www.researchdemystified.org43 Cost-benefit Analysis Is it cost beneficial to get a college degree? What are benefits that cannot be measured in dollars? Career opportunities: some jobs are only available for those with college degrees, even if they do not pay that much Increased self-esteem Transformative experience: exposure to new ideas Sustains a vibrant democracy

44. Dr. Johnson. www.researchdemystified.org44 Cost-benefit: Public Sector Policy analysis: what are 5 different approaches to these issues 1. Reduce recidivism for drug users 2. Prevent unwed teenaged parenthood 3. Provide health care to all citizens 4. Prevent financial meltdowns like we saw in the banks and financial industries in Fall 2008 What are assumptions are you making between cause and effect in each scenario?

45. Dr. Johnson. www.researchdemystified.org45 Cost-benefit: Public Sector Policy analysis: Pick one policy issue and consider how you might think about calculating the costs and benefits for each one. Note: one policy choice is always to do nothing Assuming there was that was the clear winner in the cost-benefits contest, are dollars the only consideration in the political environment?

46. Dr. Johnson. www.researchdemystified.org46 Cost-benefit: Public Sector Values, beliefs, political ideology will all affect the debate But each side are likely to present their best guess of costs and benefits. And their estimates are likely to differ

47. Dr. Johnson. www.researchdemystified.org47 Other Costs Social Costs and Externalities Costs paid by the larger society Take a look at the externalities of the material’s economy: http://www.storyofstuff.com / Of course—you might want to question where some of the numbers come from but it does point out that we tend not to take externalities (meaning someone else pays a price) into account.

48. Dr. Johnson. www.researchdemystified.org48 Other Costs Externalities: costs paid by someone other than those receiving the benefit Those producing or using goods and services do not pay the full costs E.g. Corporations pollute but the taxpayers pay for the cleanup City A benefits from having manufacturing company but City B down river pay for the price: either higher health costs or costs for cleanup of toxics

49. Dr. Johnson. www.researchdemystified.org49 Other Benefits Short term benefits are easier to measure than longer-term benefits Direct benefits are easier to measure than indirect benefits Direct: savings from health promotion program because less spent on health treatments Indirect: savings from loss of productivity

50. Dr. Johnson. www.researchdemystified.org50 Other Benefits Intangible benefits: avoidance of pain and grief of infant deaths It is harder to attach a dollar value: if it is your child, the life of your child is priceless

51. Dr. Johnson. www.researchdemystified.org51 Cost-Benefit Assumptions Implied balance between costs and benefits But: who gains is not necessarily the same as who pays! Key Question: Do these estimates and assumptions affect the results in a particular way?

52. Dr. Johnson. www.researchdemystified.org52 Uses of Cost-benefit Analysis While not as concrete as often portrayed, it is a useful framework for considering policy options. Can provide a deeper understanding of the many connections between policy options, their potential costs and benefits, externalities, and equity considerations

53. Dr. Johnson. www.researchdemystified.org53 Choosing a Research Approach The choice is guided by the situation If data is available, then using available data makes sense If you want to gather people’s opinion, then a survey makes sense If you want an in-depth understanding of a program or community, then a case study makes sense

54. Dr. Johnson. www.researchdemystified.org54 Choosing a Research Approach The choice is guided by the situation If many studies have already been done, then an evaluation synthesis makes sense If you want to make sense of communications, then content analysis makes sense Of you want to determine which approach is more likely to be cost effective, than cost benefit analysis makes sense

55. Dr. Johnson. www.researchdemystified.org55 Choosing a Research Appoach Research approaches can be combined: A survey might be used to gather data for a correlation design using statistical controls A case study might include a content analysis of news clippings, available data and interviews with key community leaders

56. Dr. Johnson. www.researchdemystified.org56 Key Takeaway Lessons If the researchers have posed a cause-effect question, then they should use a design in the Xs and Os framework The classic experiment is best for impact questions because it is necessary to eliminate all other rival explanations Be cautious if they have asked a cause-effect question or implied a cause-effect relationship: Is the research design strong enough to support the conclusions?

57. Dr. Johnson. www.researchdemystified.org57 Creative Commons This powerpoint is meant to be used and shared with attribution Please provide feedback If you make changes, please share freely and send me a copy of changes: Johnsong62@gmail.com Visit www.creativecommons.org for more informationwww.creativecommons.org