Presentation is loading. Please wait.

Presentation is loading. Please wait.

Policy Evaluation Primer. Action Research Cycle Stage One: Problem Identification: Acknowledge the need for change. Can be an existing, even historical,

Similar presentations


Presentation on theme: "Policy Evaluation Primer. Action Research Cycle Stage One: Problem Identification: Acknowledge the need for change. Can be an existing, even historical,"— Presentation transcript:

1 Policy Evaluation Primer

2 Action Research Cycle

3 Stage One: Problem Identification: Acknowledge the need for change. Can be an existing, even historical, problem, or an emerging issue. Stage One: Problem Identification: Acknowledge the need for change. Can be an existing, even historical, problem, or an emerging issue. Stage Two: Evaluation: Develop and carry out methods for evaluating the breadth and depth of the issue and how it informs practice. Stage Two: Evaluation: Develop and carry out methods for evaluating the breadth and depth of the issue and how it informs practice. Stage Three: Recommendations: Based on the evaluation, provide specific recommendations for change and/or continued evaluation. Stage Three: Recommendations: Based on the evaluation, provide specific recommendations for change and/or continued evaluation.

4 Stage Four: Application/Practice: Work with the powers that be to take action and institutionalize the recommendations. Stage Four: Application/Practice: Work with the powers that be to take action and institutionalize the recommendations. Stage Five: Reflection: With changes in place, reflect on ways in which new practices affect the work environment. Concurrently, reflect on what you, as an individual researcher, and/or the team learned from the process of the research. Stage Five: Reflection: With changes in place, reflect on ways in which new practices affect the work environment. Concurrently, reflect on what you, as an individual researcher, and/or the team learned from the process of the research. Stage Six: Consideration of New Questions: Acknowledge and dialogue about new questions that have emerged from the changes. Have the changes worked? Are there any shortcomings? Stage Six: Consideration of New Questions: Acknowledge and dialogue about new questions that have emerged from the changes. Have the changes worked? Are there any shortcomings?

5 Problem identification: What is the problem or issue you wish to examine? What is the problem or issue you wish to examine? Is the problem an ill-structured or structured one? Is the problem an ill-structured or structured one? Initial situation clearly defined; one desirable goal; clear dominant means to achieve that goalInitial situation clearly defined; one desirable goal; clear dominant means to achieve that goal Many ways to describe problem, complex goal state, values matter, no one clear best goal- state, etc.Many ways to describe problem, complex goal state, values matter, no one clear best goal- state, etc. Structuring ill-structured problems complicated, but necessaryStructuring ill-structured problems complicated, but necessary

6 Problem identification: Who is this a problem or issue for? Who is this a problem or issue for? Who might be impacted by an eventual resolution of the problem or issue? (e.g. who are the stakeholders?) Who might be impacted by an eventual resolution of the problem or issue? (e.g. who are the stakeholders?) What is your unit of analysis? What is your unit of analysis?

7 Problem Formulation Problem Formulation “Some ways to formulate complex social problems surely are more useful than others in a given situation, but there is never a single, clearly correct problem definition on which analysis can converge. Nor is there any purely analytic way of specifying which problems, once formulated, ought to be higher than others on the political agenda.” Lindblom and Woodhouse P.22 “Some ways to formulate complex social problems surely are more useful than others in a given situation, but there is never a single, clearly correct problem definition on which analysis can converge. Nor is there any purely analytic way of specifying which problems, once formulated, ought to be higher than others on the political agenda.” Lindblom and Woodhouse P.22

8 How do we evaluate policies? What’s the goal? What’s the goal? To develop improved policiesTo develop improved policies Academic/’wonk’ process Academic/’wonk’ process Political process Political process To critique policies we disagree withTo critique policies we disagree with To find support for policies we agree withTo find support for policies we agree with To know what position to declare on a specific policyTo know what position to declare on a specific policy

9 Who’s the audience? Who’s the audience? Are you responding to a request from a politician/decision maker?Are you responding to a request from a politician/decision maker? Are you working to develop and refine general policies (e.g. academics)?Are you working to develop and refine general policies (e.g. academics)? Are you working to promote specific policies?Are you working to promote specific policies?

10 “If professional analysis deserves to be seen as no more than an input to political interaction and judgment, never a substitute for it, then inquiry and judgment by ordinary people remain at the heart of the policy- making process.” “If professional analysis deserves to be seen as no more than an input to political interaction and judgment, never a substitute for it, then inquiry and judgment by ordinary people remain at the heart of the policy- making process.” Lindblom and Woodhouse P.22

11 Evaluation requires analysis To increase our understanding To increase our understanding To solve problems To solve problems

12 Analysis A search for understanding: why and how? A search for understanding: why and how? Not description Not description We break a problem down into component parts to simplify analysis We break a problem down into component parts to simplify analysis How does a car work? How does a car work? Take it apart and figure out how each part works?Take it apart and figure out how each part works? Not enough– need to figure out the interactions between the partsNot enough– need to figure out the interactions between the parts Same is true for policiesSame is true for policies

13 Evaluation in Complex vs. Simple Problems

14 Traditional Scientific Method Objective is to understand Value-free puzzle solving exercise Careful observations and measurements Form falsifiable hypothesis Repeated experiments to test hypothesis If hypothesis is falsified, form new hypothesis When hypothesis can’t be falsified, it becomes a theory Only expert opinions matter Replicability, analytical rigor and peer review determine quality

15 Ideology Continued belief in theory even when contradicted by empirical evidence Continued belief in theory even when contradicted by empirical evidence Refusal to test theories empirically Refusal to test theories empirically “The terms of the debate can be grounded wholly in how the world ought to be, not in how it is.” Charles Murray “The terms of the debate can be grounded wholly in how the world ought to be, not in how it is.” Charles Murray

16 Public Policy Problems System is in a constant state of rapid change, and may be beyond our comprehension. Complexity, non-linearity, chaos, evolution and unexpected surprises Not value-neutral: Multiple legitimate perspectives Ethical values, not objective facts Uncertain facts Lack of baseline data Different sets of facts Different interpretation of facts

17 Public Policy Problems High stakes Urgent decisions Ill-structured problems Poorly defined problems Poorly defined solutions: usually many possibilities Difficulty of experimentation sample size of one ethical issues constant change scaling

18

19 Causality and complex systems Feedback loops Feedback loops Positive feedback loop Positive feedback loop Land prices increase, speculative demand increases, increased demand increase land prices…Land prices increase, speculative demand increases, increased demand increase land prices… Greenspan says housing bubble exists, people stop speculating, prices fall, people sell, prices fall more…Greenspan says housing bubble exists, people stop speculating, prices fall, people sell, prices fall more… Negative feedback loops Negative feedback loops Milk prices increase, farmers expand production, increased supply drives price back to equilibrium.Milk prices increase, farmers expand production, increased supply drives price back to equilibrium.

20

21

22 Ways of Stating Casual Relations Religious Attendance and Divorce Rates Religious attendance ____________ reduced divorce. Religious attendance ____________ reduced divorce.“causes” “leads to” “is related to” “influences” “is associated with” “produces” “results in” “reduces the likelihood of…”

23 Post Normal Science Facts are uncertain Facts are uncertain stakes are high stakes are high decisions are urgent decisions are urgent Values matter Values matter

24 Transdisciplinary In the university there are disciplines, in the real world there are problems In the university there are disciplines, in the real world there are problems Communication and understanding across disciplines is essential Communication and understanding across disciplines is essential Communication with sectors outside of academia is essential Communication with sectors outside of academia is essential

25 Identifying and Gathering Data Define and structure your problem. Put together outline of what you need to know. Gaps are often first information you need to collect. Define and structure your problem. Put together outline of what you need to know. Gaps are often first information you need to collect. Make list of data sources Make list of data sources Best to rely on existing dataBest to rely on existing data How much information do you need? How much information do you need?

26

27 How much information? Too much or too little information Too much or too little information Time and Cost Time and Cost “Most policy decisions are made using no more than perfunctory analysis, because decisions cannot wait until ‘all the facts are in.’” Lindblom and Woodhouse p.20“Most policy decisions are made using no more than perfunctory analysis, because decisions cannot wait until ‘all the facts are in.’” Lindblom and Woodhouse p.20 Optimal ignorance Optimal ignorance

28 Distinguishing “good” information from “bad” Source: peer reviewed, grey literature, Internet, government statistics, etc., wikipedia, etc. Source: peer reviewed, grey literature, Internet, government statistics, etc., wikipedia, etc. Triangulation Triangulation

29

30 Questions to ask: Data collected independently? Data collected independently? ExamplesExamples How were the data collected? How were the data collected? Why were the data collected? Why were the data collected? When were the data collected? When were the data collected? Who collected the data? Who collected the data? Are only selected data being reported? Are only selected data being reported?

31 Quantitative vs. Qualitative Obsession with quantification and ‘academic purity’ Obsession with quantification and ‘academic purity’ Numbers and symbols Numbers and symbols Interpretation of the facts Interpretation of the facts

32 Literature review: Goals To demonstrate a familiarity with a body of knowledge and establish credibility. To demonstrate a familiarity with a body of knowledge and establish credibility. To show a path of prior research and how current analysis is linked to it. To show a path of prior research and how current analysis is linked to it. To integrate and summarize what is known in an area. To integrate and summarize what is known in an area. To learn from others and stimulate new ideas. To learn from others and stimulate new ideas. Neuman W.L. 2000, p446

33 Interviews Standardized questionnaires Standardized questionnaires ‘Elite’ or specialized interviewing ‘Elite’ or specialized interviewing Stakeholder interviews Stakeholder interviews

34 Communication Your communication must have a specific purpose. You must fully understand this purpose, and know how to articulate it to others; Your communication must have a specific purpose. You must fully understand this purpose, and know how to articulate it to others; You must communicate to the appropriate audience, in a format suitable to that audience and to the situation; You must communicate to the appropriate audience, in a format suitable to that audience and to the situation; Your content must be well organized, concise and to the point; Your content must be well organized, concise and to the point; You must capture and hold the attention of your audience; You must capture and hold the attention of your audience; Your graphic material must be suitable for the audience, clear and easy to understand, and help to convey your message; Your graphic material must be suitable for the audience, clear and easy to understand, and help to convey your message; You must be credible, and your information must be credible; and You must be credible, and your information must be credible; and You must get critical evaluation of polished draft documents or practice presentations You must get critical evaluation of polished draft documents or practice presentations


Download ppt "Policy Evaluation Primer. Action Research Cycle Stage One: Problem Identification: Acknowledge the need for change. Can be an existing, even historical,"

Similar presentations


Ads by Google