Presentation on theme: "INTRODUCTION TO POLICY ANALYSIS Text:"— Presentation transcript:
1 INTRODUCTION TO POLICY ANALYSIS Text: Patton and Sawicki, Basic Methods of Policy Analysis and Planning, 2nd edition.
2 THE NATURE OF PUBLIC PROBLEMS In the public sector, problems:are fuzzy and ill-defined;have political as well as purely technical aspects;often lack a good cause-effect knowledge base;
3 may be solved only by producing new problems; often involve tradeoffs between cost and effectiveness;may be hard to measure adequacy of results;may be hard to measure fairness of results.
4 CLASSICAL RATIONAL PROBLEM -SOLVING MODEL In theory, problems can be approached using a rational, comprehensive problem solving model. The demands of this model are:1) Define the problem2) Determine important social values3) Identify all alternatives4) Assess all alternatives5) Select optimal alternative6) implement optimal alternative
5 LIMITATIONS IN THE PUBLIC SECTOR Theoretical Model1) Define the problem2) Determine important social values3) Identify all alternatives4) Assess all alternatives on all values5) Select optimal alternative6) implement optimal alternativePublic Sector Limitation1) Problems are interlined2) No agreement on social values3) Limited time, knowledge4) Limited resources, lack of predictability5) Pressure to select the first good solution6) Short time horizon to produce results
6 -Traditional research is concerned with broad, theoretical, complex questions. It uses explicit scientific steps and invariant procedures.Policy analysis, on the other hand, is practical, situational and flexible.- It addresses local problems and focuses on making decisions.- It is more craft or art than science.
7 Traditional ResearchSeeks "truth"Explicit steps and procedures Addresses broad questionsFocus on complexitySciencePolicy AnalysisIs practicalFlexible, situational Addresses local problems Focus on decision-makingCraft
8 CASE STUDIES IN POLICY ANALYSIS - Problems in the public sector are multi-faceted and difficult to pin down.- As if that was not bad enough, the knowledge domain of public policy is ill-structured.- This means that there is no "one best way" to solve all problems.- Giving policy analysis only one methodology is like giving a home owner only a hammer to solve all household problems.
9 - A new approach is needed to learning in this area. This approach is offered by case studies.- Case studies link problems to a reality;- They offer the opportunity for the application of policy analysis techniques in a concrete context.
10 The way information is remembered and use is linked to the way it is learned. - Case studies provide cues to the types of techniques that are needed to approach a solution to the problem. - These cues help policy analysts learn multiple approaches to learning and to problem solving.
11 Use of case studies will help to: 1) Recognize situations where analysis is appropriate and productive;2) Become competent in the application of different approaches and methods;3) Learn how to communicate the results of policy analysis.
12 TIPS FOR PRACTICAL POLICY ANALYSIS 1) Quickly identify the central decision criterion of the problem (What is the most important factor in buying a car? Taking a new job?)2) Identify what types of public sector actions can be taken(Taxing, spending, sanctions, incentives, moral suasion, education?)3) Avoid the "one best way" approach (Have many tools in the tool box, not just one)
13 4) Learn how to deal with uncertainty (Admit it, estimate its possible effects) 5) Say it with numbers (Charts, graphs, tables, maps, etc.)6) Make the analysis simple and transparent (Provide details in a technical appendix)7) Check and re-check the facts (Use multiple sources of facts, triangulation)8) Learn to anticipate the objections of opponents (Improves the ultimate product)
14 9) Give analysis, not decisions (Distinguish between analysis and advocacy) 10) Push the boundaries of the envelope (Expand the problem definition; introduce novel solutions)11) Policy analysis is never 100% complete, rational, and correct (How much time, money, and personnel is available to do the job?)
15 2. ALTERNATIVE POLICY ANALYSIS MODELS 1) Quadea. Policy formulationb. Search for alternativesc. Forecast the futured. Model the impacts of the alternativee. Evaluate, compare, and rank the alternatives
16 2) MacRae and Wildea. Define the problemb. Determine criteriac. Generate alternativesd. Choose course of actione. Evaluate policy after implementation
17 3) Stokey and Zeckhauser a. Determine the underlying problemb. Determine the objectivesc. Generate alternativesd. Predict consequences of each alternativee. Determine criteria for measuring achievementsf. Choose course of action
18 4) Urban Institutea. Define the problemb. Identify objectivesc. Select criteriad. Specify the cliente. Calculate the cost of each alternativef. Assess the effectiveness of each alternativeg. Present the findings
19 5) Weiner and Vininga. Problem analysisa.1. Understand the problema.2. Choose goals and constraintsa.3. Choose method of solutionb. Solution analysisb.1. Choose evaluation criteriab.2. Specify alternativesb.3. Assess alternativesb.4. Recommend solution
21 7) Patton and Sawickia. Verify, define and detail the problemb. Establish evaluation criteriac. Identify alternative policiesd. Assess alternative policiese. Display and distinguish among alternativesf. Implement, monitor, and evaluate the policy
22 SIX STEP POLICY ANALYSIS 1) Verify, define and detail the problem2) Establish evaluation criteria3) Identify alternative policies4) Assess alternative policies5) Display and distinguish among alternatives6) Implement, monitor, and evaluate the policy
23 VERIFY, DEFINE AND DETAIL THE PROBLEM State the problem meaningfully:Determine the magnitude and extent of the problemContinually redefine the problem in light of what is possibleEliminate irrelevant materialQuestion the accepted thinking about the problemQuestion initial formulations of the problemSay it with dataLocate similar policy analysesLocate relevant sources of data
24 Eliminate ambiguityClarify objectivesResolve conflicting goalsFocus on the central, critical factorsIs it important?Is it unusual?Can it be solved?Identify who is concerned, and why?What power do concerned parties have?Make a quick estimate of resources required to deal with the problem
25 2) ESTABLISH EVALUATION CRITERIA What are the important policy goals, and how will they be measured?Identify criteria central to the problem and relevant to the stakeholdersClarify goals, values and objectivesIdentify desirable and undesirable outcomesIs there a rank order of importance among the criteria? What will be the rules for comparing alternatives?
26 Administrative EaseCosts and benefitsEffectivenessEquityLegalityPolitical acceptability
27 3) IDENTIFY ALTERNATIVE POLICIES Consider a wide range of optionsConsider the status quo, or no-action alternativeConsult with expertsBrainstorming, Delphi, Scenario writingRedefine the problem if necessary
28 4) ASSESS ALTERNATIVE POLICIES Select appropriate methods and apply them correctlyEstimate expected outcomes, effects, and impacts of each policy alternativeDo the predicted outcomes meet the desired goals?Can some alternatives be quickly discarded?Continue in-depth analysis of alternatives that make the first cut
29 5) DISPLAY AND DISTINGUISH AMONG ALTERNATIVES Choose a format for displayShow strengths and weaknesses of each alternativeDescribe the best and worst case scenario for each alternativeUse matrices, reports, lists, charts, scenarios, arguments
30 6) IMPLEMENT, MONITOR, AND EVALUATE THE POLICY Draw up a plan for implementationDesign monitoring systemSuggest design for policy evaluationWas the policy properly implemented?Did the policy have the intended effect(s)?
31 ROLE OF THE POLICY ANALYST Policy analysis is a systematic evaluation of the technical and political implications of alternatives proposed to solve public problems. Policy analysis refers to both the process of assessing policies or programs, and the product of that analysis.A policy analyst:uses qualitative and quantitative data;uses a variety of approaches to the problem;applies appropriate methods correctly.
32 Who does policy analysis? Is public policy analysis a calling?A vocation?A service?A guild?A cult?The role of the policy analyst is to:Produce arguments for debates about public policyProduce evidence for decisions about public policyAct as internal organizational consultantsAct as external policy consultantsHandle both technical and people aspects of policy analysis
33 All policy represents the distribution of power and resources All policy represents the distribution of power and resources. These policies are an expression of values. Values and beliefs are often used as short-cuts to decision-making.What code of ethics should the policy analyst adopt?What about the professional values of obligation, responsibility, discretion, and citizenship?What about published professional codes of ethics, such as ASPA, ICMA, AICP, NASW, NSPE, etc.?
34 The policy analyst has responsibilities, to the client, the customer, the self, the profession, the public interest, fairness, equity, law, justice, efficiency, effectiveness, and the practice itself.Who is to define what is good?Whose values or goals should be pursued?What is the right thing to do?Who or what is ultimately to be served?Should the analyst try first and foremost to do good, or to do no harm?Should the analyst give neutral advice, or normative advocacy?Should the analyst be supportive or adversarial?
35 Bias is inevitable in policy analysis Bias is inevitable in policy analysis. To mitigate the effects of bias, the analyst can:identify all underlying assumptionskeep accurate recordsuse multiple sources of informationuse replicable methods and modelsidentify the client's goals and valuesidentify the formal and informal actors and institutionsaddress relevant professional and ethical considerations
36 3. CROSS-CUTTING METHODS SELECTING TECHNIQUESSelecting the appropriate techniques to use in policy analysis depends on a variety of factors:what the client wants to knowthe time availableknowledge of the decision criteriacomplexity of the issueavailable data
37 Some techniques commonly used in various stages of policy analysis include: 1. Verifying, Defining, and Detailing the ProblemBack-of-the-envelope calculationsQuick decision analysisCreation of valid operational definitionsPolitical analysisIssue paper/first cut analysis
44 7. Cross-Cutting Methods Identifying and gathering dataLibrary search methodsInterviewing for policy dataQuick surveysBasic data analysisAssessing information qualityCommunicating the analysis
45 CROSS-CUTTING METHODS Cross-cutting methods are techniques of policy analysis that can be used at nearly any stage in the analysis.They are useful tools for the policy analyst to know how to use. They include:Identifying and gathering dataLibrary search methodsInterviewing for policy dataQuick surveysAssessing information qualityBasic data analysisCommunicating the analysis
46 IDENTIFYING AND GATHERING DATA Policy analysts need to know how to search for existing information, such as:academic journal articlesarchivescensus recordshearingslegislative historynews media reportspast policy analysespublic agency reportspublic records
47 People are also good sources of information, including advocacy groupsexpertsissue networkspersonal contactsprofessional colleagues
48 Even personal observation can be a source of data Even personal observation can be a source of data. Personal observation can furnish data on usage patterns, compliance patterns, insights into the problem, anecdotes, and innovative suggestions. However, observation is time consuming and may suffer from problems with accuracy, bias, limited samples, and difficult to quantify data. Observational methods include "sidewalk surveys," mechanical counting devices, measures of erosion, satellite images, etc.
49 Other sources of information include: federal agencieslibrarieslocal agenciesnon-profit agenciesprivate organizationsresearch institutesstate agenciesthink tanksuniversities
50 Policy analysts should seek information from multiple sources ("triangulation"), especially on controversial data. Problems with sources of data include:outdated statisticsirrelevant datamisleading datapoor quality databiased data
51 Looking for documents that may be helpful in doing the policy analysis is important. But three questions that must be asked are:1) do such documents exist?2) can they be obtained in a reasonable time?3) when is additional searching no longer worthwhile?
52 LIBRARY SEARCH METHODS Libraries are excellent sources of policy-related information. To make the most of library resources, follow these strategies:1) look up basic policy-related terms and definitions in encyclopedias, dictionaries, or a subject-related thesaurus; each policy issue area has its own terms and jargon2) develop a list of search terms for searching computerized bibliographic data bases, electronic guides to library holdings, and Internet access;
53 3) identify key journals in the field and skim their table of contents for the past 1-2 years; 4) check guides to current periodicals, newspapers, news magazines, trade journals, and guides to the literature5) check annual reviews in the policy subject area; conference proceedings on the subject; government hearings on the subject, etc.
54 The federal government offers a wide variety of sources: Congressional DirectoryGovernment YearbooksGuide to Federal StatisticsInternational StatisticsPopulation ReportsStatistical Abstract of the U.S.U.S. CensusU.S. Government Printing Office cataloguesU.S. Government Manual
55 Many sources of legal information have bearing on policy issues: Adjudication and case lawAgency regulationsCode of Federal RegulationsFederal and State statutesFederal RegisterLegal PeriodicalsMunicipal ordinancesSupreme Court decisions
56 INTERVIEWING FOR POLICY DATA Interviewing is typically conducted with either mass, elite, intensive, or focus group methodologies. Interviewing is typically used:to gather historical background, context, and evolution of the policyto gather basic facts about the problemto assess political attitudes and resources of major playersto gather ideas about the future, trends, and forecaststo generate additional contacts and materials (snowball technique)
57 Elite (specialist) interviewing is most typically used when: it is a short-term policy projectit is on a new topicthere is a lack of existing literatureinformants are reluctant to put information into writingno quantitative data are availableit is not feasible to use hired interviewers
58 To set up interviews, the policy analyst usually: arranges appointments in advancemakes formal or informal requests (letterhead, telephone)sends a reminder letter and follows up with a phone callgives the name of a mutual friend or influential person as a referencecollects background information prior to the interviewwill conduct a telephone interview if a face-to-face interview is not possible
59 When conducting the actual interview, it is usually accepted behavior to: ask before using any recording devicepromise anonymity and/or confidentiality of informationtake notes during the interviewkeep to the allotted timethank the person for the interviewsend a follow-up letter
60 QUICK SURVEYS Surveys can be conducted by mail, in person, or by telephone. Survey methodology is described in many standard research texts. Cross-sectional interviews are conducted at one point in time across a wide sample of the population. Longitudinal interviews are conducted repeatedly over many time intervals (months, years, decades) with the same individuals. A comparison of the advantages and disadvantages of the most typical surveys is displayed below.
61 May not be representative Least staff time required Type of SurveyResponse RateSample ConcernsStaffingOther concernsMail15%May not be representativeLeast staff time requiredResponse rate improves with giftsTelephone50%Limited to those with telephonesModerate staff time requiredShort and simple questionsIn-Person75%May be needed for less-educatedMost intensive; most supervisorsCan cover complex issues
62 ASSESSING INFORMATION QUALITY When collecting information and data for policy analysis, the analyst must assess the quality of the information and data collected.Document Analysis:When was the document generated?What was the original purpose of the document?Is there an obvious bias in the document?What is the pattern of word usage?Does the document omit important information?Are there errors in the document?
63 Assessing Interviews: Was the information plausible?Was the information consistent?Does the information diverge from accepted facts?Did the respondent report direct experience?Did the respondent have ulterior motives?Did the respondent operate under some constraints?Was the respondent candid?Did the respondent acknowledge areas of ignorance?Was the respondent self-critical?
64 Data quality:Are multiple sources of information consistent?Were data collected independently, from separate sources?Is the data original or re-organized?Do the data pertain to a particular geographic locale?Were the data collection methods systematic?For what purpose were the data originally collected?How old are the data? Were they affected by timing?Was there bias or special motivation in the collection of the data?
65 BASIC DATA ANALYSIS Data are not generally useful in their raw form. Instead, they must be analyzed. Data are most often analyzed using descriptive and/or inferential statistics. Descriptive statistics search for patterns in the data and look for relationships to gain insight into the problem. Inferential statistics attempt to estimate a characteristic of a population from data gathered from a sample.
66 Descriptive univariate statistics look for patterns in the data Descriptive univariate statistics look for patterns in the data. They are best presented in graphical form, using frequency distributions, cumulative distributions, bar charts, histograms, pie charts, and frequency polygons. Statistics include the mean, median, and mode, as well as the range, standard deviation, and variance.
67 Descriptive bivariate statistics look for relationships in the data Descriptive bivariate statistics look for relationships in the data. They are best presented in tables, plots, scattergrams, and time series graphs. Measures of association include Lambda, Gamma, and Pearson's r.
68 Inferential statistics make probabilistic statements (or inferences) about a whole population based on the results obtained from a partial sample. Measures of statistical significance are used to estimate whether two groups differ from one another, or whether there is a chance that a relationship observed in a sample also exists in a population. These measures include Chi Square, Z-scores, t-tests, and F-tests.
69 COMMUNICATING THE ANALYSIS Written Communication:Work from an outline--keep separate folders for each section of the analysisWork from goals and deadlines--generate a complete draft and then fill in the holesGet help--with editing of rough drafts; revise for clarity; incorporate new ideasInclude a Table of Contents--sections include Executive Summary; Problem Definition; Decision Criteria;Alternatives; Comparison of Alternatives; Conclusions and Recommendations;
70 Use graphics--charts, graphs, flow charts, tables, maps, pictures, diagrams, drawings, etc. Use geographic information systems (GIS)--to generate maps of data distributionsSimplicity--use the active voice for verbsAccuracy--verify facts; triangulate; check all calculationsDocumentation--note all formulas used and assumptions madeFairness--use references and give credit to your sources of dataNeatness--use good grammar, spelling, punctuation, syntax, etc.
71 Oral PresentationsKnow your audienceKeep it short and simpleUse visual aids and handoutsAllow time for questions, comments and criticisms
72 4. VERIFYING, DEFINING, AND DETAILING THE PROBLEM PROBLEM DEFINITIONThe first thing the policy analyst must do is to ask:1) Does a problem exist?2) Can anything be done about it?3) Does the client have the power?If the answers are no, then there is no point in doing a policy analysis.
73 Pitfalls in public policy problem definition: 1) accepting the client's definition of the problem2) looking only for the simple and obvious3) thinking that any and all problems need a public solution4) confusing the need for short- versus long-term solutions5) confusing the values of individuals versus collectivities
74 Don't Need Public Policies Do Need Public PoliciesIndividual problemsSocial problemsWidespread problemsSerious problemsRelative problemsAbsolute problems
75 DEVELOPING PROBLEM STATEMENTS In developing problem statements:1) think about the problem2) delineate the boundaries of the problem3) develop a fact base4) list goals and objectives for policy solutions5) identify the policy envelope (key players)6) develop preliminary costs and benefits7) review the problem statement
76 BACK-OF-THE-ENVELOPE CALCULATIONS One of the first things a policy analyst will do is to try to get a handle on the possible dimensions of the problem and potential solutions. The analyst may ask,1) How many people are we talking about?2) What is the likely cost per unit of service?3) How much of the target population can we serve?4) How much do we have available to spend?5) Will more staff be needed?6) Will this impact the budget/tax rate?7) What are the trends in this area?8) What will happen if we do nothing?
77 For example, try to estimate these parameters if half the children in the state are not receiving the required immunizations before beginning school. Start with the number of children in the state up to age 5.Which immunizations are required?How much does each one cost?How many children could realistically be reached?How much do we have available to spend?Could we get more from the Federal government?Will more state staff be needed, or can this be handled by the private/non-profit sector?Will this impact the budget/tax rate?What are the trends in this area--is the problem increasing or decreasing over time?What will happen if we do nothing?
78 The information for doing back-of-the-envelope calculations can come from 1) reference works2) experts3) past studies or quick research4) informed guesses, extrapolation, rules of thumb, estimation, parallel reasoning, triangulation, etc.
79 QUICK DECISION ANALYSIS Quick decision analysis is a variation on the technique of making decision trees. Decision trees are ways of diagramming a problem, when the problem has more than one solution. It is a tool to help policy analysts see the logical alternatives to a problem.
80 POLITICAL ANALYSIS Policy analysts recognize that politics is important at all stages of the policy process, including policy analysis. There are a number of ways to communicate about potential political influences or factors that may impinge on the policy analysis. These techniques attempt to allow political factors to be treated like any other important considerations in policy analysis.
81 The analyst may draw up a list of issues involved in defining the problem, and identify a number of potential political actors who have taken positions on those issues. A table can display the likely support or opposition of each group to each issue. For example, what are the issues involved in raising the age at which teens can get a driver's licence to 18? Which groups are likely to support (+) or oppose (-) problem definitions that focus on these issues?
82 GroupsLower teen auto accident, death & injury ratesHardship for teens who work or commute to schoolLower insurance rates for family carsM.A.D.D.+?Parents/Voters-Insurance Lobby
83 FIRST CUT POLICY ANALYSIS An issue paper is a study that is conducted in preparation of making a decision on whether or not to do a policy analysis. It describes the problem, the attendant issues, the political groups involved, and concludes whether or not a policy analysis will be feasible.
84 A first cut policy analysis concentrates on identifying preliminary recommendations. It is a mini-policy analysis, conducted in a short period of time, using simple techniques. It forms the basis for a much more in-depth, complex, and thorough full-fledged policy analysis.
85 Researched analysis refers to a more traditional research project, perhaps conducting a pilot study of several policy alternatives to generate concrete data on which to base recommendations.However, policy analysts rarely have the luxury of the time and resources needed, nor do they often work for someone who is far enough removed from the problem to resist pressures for a quick solution.
86 5. ESTABLISHING ANALYSIS CRITERIA WHAT ARE CRITERIA Every time a policy problem is identified, some statement of goals is adopted. The goals are what the adopted policy alternative should accomplish. Goals are broad, formal, long-term problem-solving achievements that are desired. An example might be to make sure that all rivers are safe, clean, and usable.
87 Goals are translated into objectives Goals are translated into objectives. Objectives are more concrete statements about desired end states, with time tables, target populations, and resource limits. An objective might be to make the Colorado River safe for swimming and fishing.
88 Criteria are the measurable dimensions of objectives Criteria are the measurable dimensions of objectives. Criteria are used to compare how close different proposed policy alternatives will come to meeting the goals of solving the problem. Criteria set the rules to follow in analyzing and comparing different proposed policy alternatives (solutions).
89 Sample criteria for improving river water quality might be: effectiveness--how much of an improvement in water quality will this alternative produce?cost--how much will it cost to improve the quality of the river using this alternative?technical--do we have the equipment and know-how to use this alternative?political--is this alternative politically acceptable?
90 Measures are the actual measurements that will be taken of each proposed policy alternative. For example, measures such as the following might be employed:effectiveness--how many milligrams of pollutants per liter of water will this alternative clean up?cost--how many dollars will be required to implement this alternative?technical--is the necessary equipment for this alternative available and are people trained to use it?political--what percentage of the voting-age population will favor this alternative in a statewide poll?
91 One difficulty in specifying criteria and measures is that many problem statements have vague, fuzzy, or even conflicting goals.This is often necessary in order to get consensus on taking some action about the problem.But this complicates the selection of criteria.
92 If dirty rivers are a problem, and the goal is to have clean rivers, what is the most important considerations in choosing between different ways of cleaning up the rivers?Is it cost?Is it effectiveness?Is it equity?What do we mean by "clean"?It is impossible to get rivers 100% clean.Do we use Federal, State, or local standards on admissible levels of toxicity?How will we measure the level of cleanliness that different policy alternatives are likely to produce?
93 RELIABILITY AND VALIDITY The criteria and their measures must be unambiguous. They should be relatively straightforward and simple to measure. Their application should produce uniform results, no matter who does the measuring of different alternatives. And repeated measurements of the same alternative should produce the same results, again, no matter who does the measuring.
94 Criteria and measures should be appropriate to the unit of analysis. That is, if the goal of a proposed policy alternative is to change the investment strategies of cities, the unit of measurement is cities, not individuals.Be sure to specify whether the unit of measurement is households or families, census tracts or neighborhoods, school children or school districts, etc.
95 ECONOMIC CRITERIAMost policy analysis involves at least one economic criterion. These include impacts on the economy, on expected public sector revenues, on government spending, etc.The most common economic criteria are costs. These may include:borrowing costs--the costs of borrowing fundsdecreases in net worth--decreases in assets and/or liabilitiesdirect costs--directly attributable to the policy alternative
96 indirect costs--additional impacts not included in the goals intangible costs--costs that cannot be counted or quantifiedmonetarizable costs--can be expressed in dollarsone-time fixed costs--new capital expenditures, equipment, training, etc.operations and maintenance costs--ongoing costs of the alternativeopportunity costs--other things that could have been done with the same resources insteadtangible costs--can be counted and quantified
97 Costs need to be counted. One cannot assume that the money was going to be spent anyway.Costs should be identified as completely as possible, eliminating unpleasant surprises down the road.Another type of cost criterion that is often employed is marginal cost.That is, if some good or service is already being produced, how much more will it cost to produce one additional unit of output?
98 The types of costs that are considered in marginal analysis are: fixed costs--these do not vary in the short run, no matter how many units are producedvariable costs--these vary directly with the volume of output of goods or servicesaverage costs--the total of units of output divided by the total costs of outputmarginal costs--the costs of producing one additional unit of outputsunk costs--these are costs that can be ignored as they have already been spent in the past
99 Another type of economic criterion is benefits. Benefits are the opposite of costs.Benefits are ways in which the policy actors will be better off.Benefits can be measured in many of the same ways as costs, including:direct benefits--directly attributable to the policy alternativeincreases in net worth--increases in assets and/or liabilitiesIndirect benefits--additional benefits not included in the goals
100 interest earned--interest that will accrue or be paid intangible benefits--benefits that cannot be counted or quantifiedmonetarizable benefits--can be expressed in dollarsone-time benefits--one-time reduction in the problemongoing benefits--continuing decreases in the problemtangible benefits--can be counted and quantified
101 Benefits are often more difficult to quantify than costs Benefits are often more difficult to quantify than costs. One alternative is to use "shadow prices," or the value of the benefits in a perfectly competitive market, for example, free recreation facilities, wilderness areas, parks, etc.
102 EQUITY CRITERIAEfficiency and effectiveness are technical and economic questions, but equity is a public question.Equity asks about the social allocation of burdens and benefits.Equity asks the questions of "who pays?" and "who benefits?"
103 A proposed policy alternative may impact equity if it will change the distribution of burdens and benefits in society. There is no universally approved optimal or right answer for how benefits and burdens should be distributed in society.That is a continuing area of contention, and essentially a political decision.
104 Horizontal equity asks whether burdens and benefits are being shifted among groups in society which are relatively equal. Vertical equity asks whether burdens and benefits are being shifted among groups in society which are relatively unequal. Inter-generational equity asks whether burdens or benefits are being shifted from one time period to another, whether younger generations will have to pay more and receive less than older ones, or vice versa.
105 Groups are often identified on the basis of: residenceincomecitizenshiprace or ethnicitysexagefamily statushome ownershipeducational statusveteran statuscriminal recordsubstance abusehealth
106 Problems in assessing equity include: how should the population be sub-divided?how should groups be defined?should historical criteria, the status quo, or desired states be used?what is a burden?what is a benefit?what is a degree of need?what is an ability to pay?
107 TECHNICAL CRITERIA Effectiveness is often used as a criterion by which to judge policy proposals.Effectiveness is the extent to which the proposed policy will attain the goals set forth in the problem statement. For example, if the goal is to decrease the current teenage driver accident rate,how much will each policy alternative decrease the rate below current levels?
108 Another technical criterion is technical feasibility. This asks whether the technology exists or is readily available to implement a proposed alternative.For example, one proposed policy alternative may be to install in all cars a breath analyzing device that would not let a car start if the driver has been drinking.However, this technology is not widely or cheaply available.
109 Other technical criteria may question whether the measurement of criteria can be conducted at the desired level of reliability and validity.For example, are there tests that can adequately measure whether students in bilingual education programs have the same level of literacy as students in non-bilingual education programs?
110 POLITICAL CRITERIA Many times the client for the policy analysis will hold a political office. In that case, the policy analyst must often include political criteria in the assessment of proposed policy alternatives. Political viability asks whether or to what extent a proposed policy alternative will be acceptable to relevant powerful groups, decision makers, legislators, administrators, citizens, neighborhoods, unions, or others.
111 Other ways of assessing political viability include: acceptability--is the proposed alternative acceptable to policy makers, policy targets, the general public, voters, etc.?appropriateness--is the proposed alternative appropriate to the values of the community, society, the legislature, etc.?Legal--is the proposed alternative legal under current law, or will statutes have to be amended or enacted?responsive--will the proposed alternative meet the real or perceived needs of the target group, the public, etc.?
112 ADMINISTRATIVE CRITERIA Many public policies are implemented by public agencies. Therefore, administrative operability or administrative ease are often used as criteria for judging proposed public policies.Questions that may be addressed include:authority--does the agency have the authority to implement the proposed policy?
113 commitment--does the proposed policy have the commitment of top managers, field staff, and support staff?capacity--does the agency have the resources to implement the proposed policy, in terms of staff, skills, money, training, expertise, etc.?support--are the facilities, equipment, and other support available for the proposed policy?
114 6. IDENTIFYING ALTERNATIVES GENERATING ALTERNATIVESBefore alternatives can be generated,1) the problem must be correctly identified, and2) relevant criteria for judging alternatives must be specified
115 At first, the policy analyst can generate a large number of alternatives, but later reduce them to a manageable size (between four and seven). Consider alternatives like the status quo, but also radically different. Consider what may be possible under different circumstances.
116 Some criteria that are often used in judging the suitability of alternatives include: 1) cost--can we afford it; will it be cost-effective?2) reliability--does it have proven success, or is it subject to failures?3) stability--will it still work if conditions change?4) invulnerability--will it work if one of its component parts fails?5) flexibility--can it accomplish more than one thing?6) riskiness--is there a high chance of all or nothing?
117 7) communicability--is it easy to understand? 8) merit--does it address the problem?9) simplicity--is it easy to implement?10) compatibility--is it congruent with existing norms and procedures?11) reversibility--can we return to our prior state if it fails?12) robustness--can it succeed in different future states?
118 SOURCES OF ALTERNATIVES 1) The status quo or no action alternativeThis means that current efforts will continue at the same level. It is important to consider how effective any different alternative will be at changing the status quo.A baseline analysis:identifies clear trade-offs with the present;clarifies project objectives;underlines whether there is a need for action or not;provides linkages to existing efforts;identifies problems likely to emerge;confirms that no optimum solution exists.
119 2) experiences of others with similar problems, from reported research findings, experts, laws, public opinion polls, new technology, etc.3) re-define the problem from others' points of view, including opponents of any change4) consider the ideal, then apply political, economic, and other constraints5) start from generic, to modified, to custom-made alternatives6) Quick Surveys by telephone, fax, or , of peers, old MPA classmates, people in the policy issue network, public meetings or hearings, content analysis of editorials, letters to the editor, etc.
120 7) Literature review of professional and academic journals, government reports, collected proceedings from conferences, on-line services (lexis-nexus, first search, article first etc.).8) Case studies of real world experiences: why was the alternative adopted, what were the circumstances, what other alternatives were considered and discarded, how did it eventually work out, what modifications were made after implementation.
121 9) Passive collection and classification: keep a folder for collecting interesting policy solutions on a regular basis, even if no problem exists at the moment, from clients, superiors, advocates, media, interest groups, etc. Then refer to the folder in emergencies.10) Develop Typologies: identify all the types of persons likely to be affected by any policy alternative, and what the probable reaction of each group would be to each type of alternative suggested; then develop alternatives that can overcome the objections of most of the groups.
122 11) Use analogies: 'new' problems are really just like other 'old' problems. Personal Analogy: pretend to be someone affected by this problem, identify with the problem to see what types of policy alternatives suggest themselves;Direct Analogy--look at solutions to other problems to see if they can be applied to this one;Symbolic Analogy--imagine the most aesthetically satisfying solutions rather than merely technologically sound ones;Fantasy Analogy--image the ideal solution, and try to preserve as much of it as possible when working backwards through real world constraints.
123 12) Brainstorming--can be oral, written, or electronic 12) Brainstorming--can be oral, written, or electronic. Brainstorming has two phases, first a pure idea-generation phase, where no judgments are made about any ideas; and second, an evaluation and ranking phase, to help arrive at concrete solutions.13) Feasible Manipulation--takes existing policy activities and develops alternatives based on limited, moderate, or wide manipulation of the range of possible activities.
124 14) Modify existing solutions: Magnify--do more, more often, larger, longer, exaggerate, add new components, new resourcesMinify--do less, less often, smaller, shorter, omit, remove, split apart, under use, fewer resourcesSubstitute--switch components, apply in different order, use different materials, try a different location or different sponsor
125 Combine--blend approaches, combine units, combine purposes, combine sponsors Re-arrange--reverse, invert, change sequence, speed up, slow down, randomizeLocation--use single or multiple locations, node versus scattered, temporary versus permanentTiming--accelerate, lag, stagger, run concurrently, shorter span, longer span, time sharingFinance--provide, purchase, tax, user fee, subsidy, co-pay, deductible, voucher, contract out
126 Organization--centralized versus decentralized, mandated versus voluntary, regulated, prohibited, enforced, inform, imploreDecision Sites--individual, unit, organization, elected, appointed, advisory, bindingInfluence Points--users, providers, intermediaries, beneficiaries, payersRisk Management--guarantees, insurance, remedial correction
127 PITFALLS1) Too much reliance on past experience 2) Failure to capture ideas and insights (listen, write them down, record them) 3) Too early closure on problem definition 4) Sets a policy preference too soon before all the alternatives are known 5) Criticizing new ideas as they are offered 6) Some alternatives are ruled out too early on 7) Failure to re-consider discarded alternatives if conditions change
128 7. ASSESSING POLICY ALTERNATIVES Which policy alternative should be adopted? In this step in the policy analysis process, the policy analyst takes each of the proposed policy alternatives and, one by one, applies each of the decision criteria to each alternative.
129 For example, say we have specified that we will be using the criteria of efficiency, cost, political acceptability, and equity to make our decision. We have defined what we mean by each of these and how they will be measured.For example, efficiency is defined as the amount of reduction in the teenage driving accident rate, and is measured by the Department of Motor Vehicles as the number of accident involving teen drivers divided by the total number of teens in the state.
130 We must then look at each proposed policy alternative, one at a time, and ask, what would be the efficiency of this alternative?What would be the cost of this alternative?What would be the political acceptability of this alternative? And how will this alternative affect equity?We then repeat this process for every alternative, including the no-action alternative.How do we know what the efficiency of each alternative will be?That involves forecasting.
131 FORECASTINGThe criteria that will be important in assessing proposed policy alternatives determine what needs to be forecast. For example, if the goal of a proposed policy alternative is to lower the teenage driving fatality rate, then what needs to be forecast is the teen driving fatality rate, first under the assumption that no action is taken, and the under the assumption that the policy alternative being considered is implemented.
132 There are a variety of methods used to make forecasts There are a variety of methods used to make forecasts. Forecasting methods range from simple stereotyping to complex statistical formulas.Intuition may use techniques such as Delphi, scenario writing, or feasibility assessment. However, it requires that the participants be quite knowledgeable, and it needs to be checked for logical consistency.
133 Theoretical models identify important variables and specify the nature of the linkages among them. Then the model is used to predict outcomes when one or more of the variables are changed. Models are built from information, experience, expert advice, etc.Constructing a model helps to get to the key elements of the situation, and focus on the most important concerns. It identifies the key factors and the relationships among them which will likely be impacted by any proposed policy alternative. It demonstrates the likely consequences of either the no action alternative, or any other rival alternative.
134 Models may be expressed in words, in physical dimensions (e. g Models may be expressed in words, in physical dimensions (e.g., architectural models), or in numerical form.Extrapolation uses the past to predict the future, assuming there are stable patterns. For example, if the population of Arizona has been growing at 50% every ten years, then a graph showing past growth can be extended into coming years to predict future growth.
135 Extrapolation is useful for conducting a baseline analysis, showing what is expected if the status quo or no action alternative is adopted. It is relatively simple and cheap and can be accurate in many circumstances. Data can be either raw numbers or a computed rate of change.
136 Extrapolation requires precise definitions of criteria and measures, and accurate measurement. It is most often used when there are linear patterns in the data. Extrapolation is less useful in the case of new problems, new issues, or new policy areas, where there is little or no past data.
137 The most commonly used form of regression is linear regression, and the most common type of linear regression is called ordinary least squares regression. Linear regression uses the values from an existing data set consisting of measurements of the values of two variables, X and Y, to develop a model that is useful for predicting the value of the dependent variable, Y for given values of X.
138 Elements of a Regression Equation The regression equation is written as Y = a+bX + eY= Dependent variable (Y), what is being predicted or explaineda=Alpha, a constant; equals the value of Y when the value of X=0b=Beta, the coefficient of X; the slope of the regression line; how much Y changes for each one-unit change in X.X= Independent variable (X), what is predicting or explaining the value of Ye=The error term; the error in predicting the value of Y, given the value of X (it is not displayed in most regression equations).
139 For example, say we know what the population has been in past years for a given area. We can compute the values of the components of the regression equation, and, and use them to predict what the area's population will be in future years.
141 If the data are not linear, that is, if on a graph the line that best shows the relationship between the two variables is not a straight line, then simple linear regression cannot be used to extrapolate into the future. Instead, the data must be converted, for example, to logarithms, or a different sort of regression must be used.
142 ECONOMIC ANALYSISOne of the most widely used economic analysis tools is to look at the long term costs and the long term benefits of a proposed policy alternative. From there, the policy analyst can calculate either the Net Present Value (NPV), the Cost-Benefit Ratio, or the Internal Rate of Return (IRR) of each alternative.
143 To calculate the long term costs and long term benefits of a proposed policy alternative, the policy analyst must assemble estimates of the initial or implementation year costs and benefits of the alternative, and the subsequent costs and values for each additional year the project will be in effect.Say that the city wants to upgrade its data processing function. It asks for bids showing costs to be submitted by different vendors. It also estimates the savings that will occur from each bid (for example, from a reduction in personnel).
144 For the first bid, the policy analyst estimates the following YEAR:12345Costs$15,000$1,223Benefits$4,000
145 DiscountingThe next step is for the policy analyst to decide on a discount rate. The discount rate assumes that money spent in the future will not cost as much as money spent today.Similarly, money gained in the future will not be worth as much as money gained today. This is based on the human preference for wanting to put off costs (or payments) as long as possible, and wanting to receive benefits (or pay) as soon as possible.
146 The discount rate is usually obtained from economists, from agency policy, or from the nature of the project being considered (i.e., whether a large infrastructure project, a revenue-bond based project, or a general obligation bond based project). Another source is the discount rate charged by the Federal Bank, or the interest rate paid on government bonds.
147 At times, the choice of which discount rate to use has been highly politicized. Because many government projects have high initial costs but a long stream of benefits, a low discount rate will make a project look more favorable, and a high discount rate will make a project look less favorable.
148 The same discount rate is generally applied to both the project costs and the project benefits. If inflation is going to be factored in, it should be applied to both the costs and benefits separately, before the discount factor is applied.To calculate the discounted costs, multiply each year's costs by that year's discount factor (the discount rate factor be obtained from a table of discount rates):
152 Net Present ValueThe Net Present Value is the value of the project if all the costs were paid today and all the benefits were gained today. To find NPV, subtract discounted costs from discounted benefits: Discounted Benefits $17,807 - Discounted Costs $16,087 = $1,720The Net Present Value of each policy alternative must be calculated separately, and then it can be compared to the NPV of each other policy alternative, to find the one with the highest NPV.
153 Cost-Benefit RatiosThe costs and the benefits of any policy alternative can be compared in a number of ways. Cost-benefit ratios are obtained by dividing discounted benefits by discounted costs:Discounted benefits =$17,807Discounted costs =$16,087Benefit/Cost ratio =1.1
154 Note that the highest benefit-cost ratio may not have the highest NPV Note that the highest benefit-cost ratio may not have the highest NPV. These are two different types of analysis. The most efficient projects have the highest benefits-to-costs ratio, but many policy analysts prefer to maximize NPV. In any case, NPV should be a positive number, and the benefit-cost ratio should be greater than 1.0
155 Internal Rate of Return The internal rate of return is an expression of the discount rate at which discounted benefits would equal discounted costs. For the example above, at an 8% discount rate, discounted benefits would equal $15,971 and discounted costs would also equal $15,971.
156 If the calculated IRR is greater than the discount rate being used for the project, then that is an indication that the project should be carried out. Generally, IRR is not comparable to either NPV or the benefits-to-costs ratio. The IRR from one project, however, can be directly compared to the IRR from an alternative project.
157 Sensitivity AnalysisOften there is no clearly superior potential policy alternative, but several that seem equally acceptable. One alternative may be better on the criterion of efficiency, while another is better on costs, and a third on political acceptability.A policy analyst will usually try to see how sensitive the analysis is to changes in assumptions. Things that the policy analyst will test include:1) the length of the project (how long will benefits continue)2) the discount rate3) the value placed on various quantities (costs, benefits, probabilities, etc.)
158 For example, a city wants to replace old garbage trucks with newer models, and it assumes the new trucks will last 20 years. What if the benefits only last 10 years? Or if the annual maintenance costs are 50% higher than what was budgeted?Or does a project still have a positive NPV if the discount rate is raised from 4% to 6%? Is the IRR still greater than the discount rate? Is the benefit-to-cost ratio still greater than 1.0 ?
159 In another example, the city assumes that building a new parking garage will raise an additional $2,000 per parking space per year in sales taxes, as well as the revenues from parking. What if only $1,000 is raised?Or say a university wants to get more students to park on campus instead of on nearby neighborhood streets. It thinks that if it reduces the parking permit fee by $10, then 25% more students will buy one and park on campus. What if only 5% more students buy one?
160 A city has vacant land that it can sell, lease, or keep A city has vacant land that it can sell, lease, or keep. The city wants to sell. It assumes that someone will buy the land and develop it, increasing the city's property tax revenues. But what if the office building remains mostly vacant? Would this change the city's decision on selling? What are the probabilities of the different outcomes? What if the probabilities for the favored outcome decrease?
161 Another type of sensitivity analysis is to identify the break-even point. This can vary according to:the length of the project (how many years are needed to break even?)the discount rate (how low before benefits equal costs?)the value of other quantities (e.g., amount of extra parking permits sold?)
162 Contingency analysis identifies what will happen if one of the basic assumptions about the project is altered. For example, what if there are large cost over-runs? What if people do not behave as predicted (e.g., buy more parking permits?)
163 A fortiori analysis examines the likelihood that any one factor will take on a value that makes the project infeasible. For example, what if the project takes two years to complete instead of one? What if interest rates rise dramatically? What if new regulations are adopted that make the project technically impossible?
164 To perform sensitivity analysis, 1) list all relevant considerations;2) establish the range of values that each variable can take, from low to high;3) holding all other values constant, vary the value of one variable at a time;4) test sensitive values to find the break-even, contingency, and a fortiori points.
165 RISK ANALYSISSome decision-makers are risk averse. They want to minimize any possible losses, rather than to pursue the (riskier) maximum possible gains. They will want to go for the sure thing (the alternative with the highest probability--in this case, do nothing--especially if limits their possible losses (for this alternative, the worst case scenario is to break even at 0).
166 Another way to begin the appreciate the different possible outcomes of different policy alternatives is to use quick decision analysis. This is a way to visually represent a small number of alternatives and their consequences.
167 Quick decision analysis identifies key issues, and helps the policy analyst to decide what information is necessary to assess each possible alternative. It helps to structure thinking about the probability or likelihood that certain outcomes will occur.It also helps the policy analysts or decision-makers to reveal their attitudes about risk and uncertainty. And it alerts the policy analyst to the possible political ramifications of predicted outcomes.
168 The steps in constructing a quick decision analysis are: 1) identify the dimensions of the analysis (problem, alternatives, outcomes)2) construct a diagram3) forecast the likely outcome for each alternative4) assess how likely each outcome is in terms of probability5) calculate the expected value of each alternative
169 For example, say a city wants to know if it should offer tax abatement to encourage economic development. The two alternatives are, simply, to do nothing (not offer the abasement), and to do something (offer the abatement).Each possible policy alternative has two possible outcomes: new economic development occurs, or new economic development does not occur.
170 Likely OutcomesDo NothingOffer AbatementDevelopment OccursNo new developmentDevelopment OccursNo new developmentChange in Property Tax Revenues+$100 m+$900 mCost of offering the abatement-$600 m-$200 mCost of additional city services-$25 m-$100 mNet +$75 m+$200 mProbability of this outcomep=0.3p=0.7p=0.6p=0.4Expected Value of this alternative0.3 x $75 m = $22.5 m0.7 x $0 m = $0 m0.6 x $200 m = $120 m0.4 x-$200 m = -$80 m+$22.5 m+$40 m
171 However, it is important to question quick decision analysis. -What studies were used to estimate outcomes and probabilities?-Were discount rate applied?-What time frame was considered?-What were the opportunity costs (how could the money be spent elsewhere?)-How sensitive are these figures to changes in the economy?-At what probability would the expected value of the two alternatives be equal?
172 If there is a great deal of uncertainty about the analysis, there are a number of strategies: 1) delay until more is known2) map out all uncertainties and the information that is needed3) collect more data to reduce uncertainty4) estimate a wide range of possible values for those which are uncertain5) develop alternatives under a wide range of possible conditions
173 6) build in more flexibility 7) build in more backup8) compromise to an acceptable alternative, even if it is not the optimal one9) choose a strategy that minimizes the maximum possible losses10) conduct in-depth research to provide the information needed
174 POLITICAL ANALYSISOften one criteria for assessing proposed policy alternatives is political acceptability to the client. A political feasibility analysis can help the policy analyst identify the important elements to be considered for each proposed policy.1) Actors--people, groups, and organizations2) Beliefs and motivations--which are negotiable, and which are non-negotiable?3) Resources--power, influence, money, staff, public opinion, etc.
175 4) Effectiveness--leadership, ability to use resources effectively 5) Sites--agendas, windows of opportunity, sequencing of decisions, etc.A political feasibility analysis takes each proposed policy alternative and examines how well it will hold up in the current political reality. Which actors will favor or oppose it, and why (beliefs and motivations)? What resources do they have, and how effective will they be at supporting or opposing the policy? Where is the debate on the policy to occur, and which actors or groups will be most powerful there? Does any group have veto power?
176 IMPLEMENTATION ANALYSIS Even after a policy is adopted, there still may be resistance to its implementation. In conducting an implementation analysis, the policy analyst looks at factors that will make the alternative easier or more difficult to implement, such as:1) are there few or many actors required to implement this alternative?2) will there be one or multiple implementation settings?
177 3) will there be a single or multiple sets of instructions? 4) what is the degree of consensus around this alternative?5) what magnitude of change will be required?6) how much of the political conflict from the adoption stage will be displaced into the implementation stage?7) can game theory be used to model the possible outcomes?8) are the necessary resources present, such as administrative will, competence, budget, skills, authority, personnel, etc.
178 8. DECISION RULES and DISPLAYING ALTERNATIVES Policy assessment techniques do not determine which policy should be adopted. Policy analysis presents the benefits and drawbacks of each alternative, but in addition one or more decision rules are needed in order to determine which policy is the "best."
179 There are many problems in trying to determine which policy to adopt. 1) Many problems in the public sector have multiple facets. Policies are designed with multiple goals or objectives. There may be no dominant objective, or several objectives may be in conflict.2) there are multiple criteria to take into account--technical, economic, political, and administrative--but who decides which is the most important?3) not all important considerations can be converted into comparable units, such as dollar values.4) which is the proper criterion to use, greatest net present value? greatest internal rate of return? largest benefit-cost ratio?
180 5) there is often a lack of agreement beforehand on decision rules, or which rules to apply 6) even if each decision criterion is optimized separately, there may still be a sub-optimal choice at the end (a camel is a horse designed by a committee). The policy analyst is often faced with trying to present multiple policy alternatives which have been assessed in terms of multiple decision criteria. There are various methods which can be used to display this information in a way that facilitates decision-making.
181 PAIRED COMPARISONS If there are 5 different ways of collecting the city's garbage, then the analyst can compare method 1 with method 2 and determine which is superior. The better of the two is then compared with method 3, and again one of them is determined to be superior. The winner of each contest is then compared with another of the remaining alternatives until all have been evaluated and the winner of the last contest is the overall winner.
182 This method presents a simple, step-by-step comparison that is relatively easy to follow, if somewhat tedious. The problem with this method is that the outcome may be influenced by the order in which the alternatives were considered.
183 SATISFICING The analyst presents all the alternatives that meet the minimum threshold levels on all criteria. The minimum threshold is then increased on each criterion, and those alternatives which do not meet the new levels are dropped. This process continues until only one alternative is left.
184 For example, say there are 6 alternatives to solve the parking problems on campus and three criteria: net revenues, number of permits sold, and student satisfaction. The minimum levels are $50,000 in revenues, 10% increase in permits sold, and at least 50% students satisfaction. Two alternatives are dropped because they do not meet one or more of these minimum levels.
185 The levels are then raised to $75,000 in revenues, 20% increase in parking, and 65% student satisfaction. Another two alternatives drop out. Finally, raising the minimums to $100,000 in revenues, 25% increase in parking, and 75% student satisfaction eliminates one alternative and the only remaining alternative that meets all these minimums is the winner.
186 This method assures that the minimum "needs" for the policy will be met, and, in addition, offers the prospect of meeting higher than minimum needs ("desires"). The problem is that it changes the definition of acceptable after the analysis has been completed.
187 GRADING METHOD The consequences of each alternative on each criterion are considered. A grade of "Pass" or "Fail" is assigned to each alternative on each criterion. Only those alternatives which "Pass" on all criteria are retained; those which have any "Fails" are rejected. The retained criteria are then compared further. This is comparable to the satisficing method discussed above.
188 LEXICOGRAPHIC ORDERING The analyst lists all those alternatives that ranked most highly on the one most important criterion. These alternatives are all considered to be roughly equal. The alternatives are then compared on the second most important criterion. Those alternatives which rank most highly are retained, and they are then compared on the third most important criterion. This process is repeated until the alternative that is most highly ranked on all the criteria is found.
189 For example, say there are 5 alternatives for economic development for the downtown area, and there are three criteria: increased revenues, increased jobs, and citizen satisfaction. Three alternatives rank highly on the most important criterion of increased jobs. These three are then compared on the second criterion, citizen satisfaction. Only two are highly ranked. These last two are compared on the third criterion of increased revenues, and the most highly ranked is the winner.
190 This is a rather straightforward process of comparison This is a rather straightforward process of comparison. However, it assumes there is agreement on which is the first most important criterion, the second most important, and so on. For situations with multiple objectives (criteria) or multiple decision makers, this may be difficult.
191 NON-DOMINATED ALTERNATIVES All alternatives are measured on all criteria, and their rank order on each criterion is displayed. Every alternative that is ranked the most highly on any criterion is retained; any alternative that is not ranked #1 on at least one criterion is discarded. The result is only "non-dominated" alternatives, that is, alternatives that are clearly superior on at least one criterion. These alternatives can then be compared further by another method.
193 In this example, only Alternatives B, C, and D were ranked #1 in at least one category. Alternative B is not dominated by any other alternative on Criterion 2; Alternative C is not dominated by any other alternative on Criterion 3; and Alternative D is not dominated by any other alternative on Criterion 1. Alternatives B, C, and D must now be considered further.
194 Say there are five designs for a new community building, and the designs are ranked in terms of their suitability for athletics, as well as their suitability for arts and crafts. Only one design will be ranked #1 in each category. These two designs are the only non-dominated alternatives. Since they are each ranked #1 on one category, and they are each ranked #2 in the other category, they are considered equal.
195 A third criterion may be added as a "tie-breaker A third criterion may be added as a "tie-breaker." For example, the two alternatives might be ranked in terms of their suitability for meetings. The higher ranking of the two will be the winner.
196 This method is straightforward and relatively easy to follow This method is straightforward and relatively easy to follow. However, it brings in additional criteria (tie-breakers) after the analysis has been completed and some alternatives have been eliminated.
197 EQUIVALENT ALTERNATIVES When at least one of the criteria can be measured in quantitative units, for example, dollars, this method can be used to compare two alternatives. It involves converting other units of measurement to dollars as well and then comparing the two alternatives again.
198 For example, you have been offered two possible jobs For example, you have been offered two possible jobs. You assess these jobs in terms of five criteria: salary (measured in dollars), climate (measured in days of sunshine per year), commute time (measured in minutes), nature of job (interesting or uninteresting), and potential for advancement (measured as good or poor).
200 Since the two jobs are both measured on at least one criterion--salary--in a quantitative form (dollars), the next step is to convert the value of the other criteria to dollars as well. For example, how much of the $42,000 salary would you be willing to give up to have more days of sunshine per year? Say that the extra 40 days of sunshine per year are worth $1,600 (you would be willing to take Job B at $1,600 less in pay if it had 40 more days of sunshine). Show these calculations in a revised chart.
202 Now, how much of the remaining salary of Job B would you be willing to give up to have a shorter commute? Say that cutting commute time by 10 minutes per day is worth $1,000 (you would be willing to take Job B at $1,000 less in pay if it had 10 minutes less of commuting). Show these calculations in a revised chart.
204 Now, how much of the remaining salary of Job B would you be willing to give up to have a more interesting job? Say that it is worth $1,400 (you would be willing to take Job B at $1,400 less in pay if it was more interesting). Show these calculations in a revised chart.
206 Now, how much of the remaining salary of Job B would you be willing to give up to have greater potential for advancement? Say that it is worth $2,500 (you would be willing to take Job B at $2,500 less if there was greater advancement potential). Show these calculations in a revised chart.
208 Now the two jobs are "equalized" on all criteria except the quantitative on (salary). Since the two jobs are equal except for salary, then the job with the higher salary (Job A) is the winner.
209 This method is rather complex and has many steps This method is rather complex and has many steps. It is also very subjective and best suited to situations where there are only individual decision makers rather than groups. It does help the decision maker to clarify their personal preferences, however, and may provide useful insight into the most important facets of the problem.
210 WEIGHTED DECISION CRITERIA When there are multiple decision criteria, a ranking or weighting system can be developed to reflect the relative importance of each criterion in the decision making process. These are sometimes referred to as "importance weights."
211 Say that the state wants to adopt a plan to give grants to the elderly to defray their utility expenses. The decision criteria are number of elderly reached, speed at delivering the grants, and controls on fraud. Each alternative is measured on each criterion and a raw score is assigned.
213 RAW SCORESElderly ReachedSpeed of DeliveryFraud CurbsAlternative A543Alternative BAlternative CAlternative D2
214 The decision criteria have been weighted by the decision makers, so that the number of elderly reached is worth 50% of the decision, the speed at delivering grants is worth 30%, and controls on fraud is worth 20%.
215 The raw score for each alternative on each criterion is then multiplied by the importance weight for each criterion. The result is a weighted score. The weighted scores for each alternative on all the criteria are then added for the total weighted score.
216 WEIGHTED SCORESElderly Reached (50%)Speed of Delivery (30%)Fraud Curbs (20%)Total ScoreAlternative A5 x .5 = 2.54 x .3 = 1.23 x .2 = 0.64.3Alternative B4 x .5 = 2.04 x .2 = 0.84.0Alternative C3 x .3 = 0.93.5Alternative D2 x .5 = 1.05 x .2 = 1.02.9
217 The alternative which scores the highest on the number of elderly reached (Alternative A) is the alternative with the highest total score. An alternative which has very good controls on fraud, but does not do as well in the areas of reaching the elderly or speed, has the lowest total score.
218 The advantages of this method are that arriving at the superior alternative is straightforward and relatively easy. Rather than considering alternatives one at a time, or considering criteria one at a time, this method considers all alternatives on all criteria simultaneously. The disadvantages are that many criteria are not measurable in quantitative form and there may be disagreement over the importance weights of the criteria.
219 GROLLER SCORECARD The matrix (or "scorecard") is a way of displaying the impacts of each alternative in terms of each criterion in "natural" units of measure, such as reduction in fatality rate, political acceptability, administrative ease, effectiveness, impact on need for additional landfills, etc. The scorecard indicates the extent to which each alternative attains the objectives specified by each criterion.
220 This approach the viewer(s) to grasp a complex analysis by summarizing it in a chart. It can help a group to come to a decision without the need to impose quantitative measurement or assign importance weights to criteria.
221 REDUCE PERMIT PRICESTRONGER PARKING ENFORCEMENTEDUCATIONAL CAMPAIGNCOST TO THE UNIVERSITYMore students buy permits; no net loss or gainHigher costs in enforcementModerate costs for materialsSOLVES PARKING PROBLEMCuts illegal parking by 25%Cuts illegal parking by 40%Cuts illegal parking by 10%ACCEPTABILITY TO STUDENTSHighly acceptableHighly unacceptableModerately acceptableACCEPTABILITY TO COMMUNITYUnacceptableBest optionWorst option
222 9. IMPLEMENTATION, MONITORING, & EVALUATION IMPLEMENTATION ANALYSISThe full policy process is often described by the following steps:1) problem definition2) alternative generation3) analysis of alternatives4) policy adoption5) policy implementation6) policy evaluation
223 While this course has focused on the first three steps, the last three steps are equally important. A thorough policy analysis will include some consideration of policy implementation, monitoring, and evaluation.
224 The policy analyst can sketch out an implementation plan for the most highly ranked alternative(s) that considers: 1) relevant actors and their interests 2) required resources and who might provide them 3) facilitators and barriers likely to be encountered 4) reasonable time frame
225 Implementation analysis might involve writing a "best-case" scenario and a "worst-case" scenario for each policy alternative, as well as the "most likely" outcome. The idea is to think systematically through the implementation process, identify potential problems, and develop actions that can be taken to either avert catastrophes or reduce losses.
226 POLICY MONITORINGPolicy maintenance refers to keeping the policy or program going after it is adopted. Policy monitoring refers to the process of detecting how the policy is doing.
227 To monitor a policy, some data about the policy must be obtained To monitor a policy, some data about the policy must be obtained. A good implementation plan will suggest some ways in which ongoing data about the policy can be generated in the regular course of policy maintenance, for example, from records, documents, feedback from program clients, diary entries of staff, ratings by peers, tests, observation, and physical evidence.
228 POLICY EVALUATIONPolicy evaluation is the last step in the policy process. It may ask deep and wide-ranging questions, such as:1) was the problem correctly identified, or was the correct problem identified?2) were any important aspects overlooked?3) were any important data left out of the analysis? did this influence the analysis?4) were recommendations properly implemented?5) is the policy having the desired effect?6) are there any needs for modification, change, or re-design? what should be done differently next time?
229 When policies fail to have the intended effect, it is usually due to one of two types of failure: theory failure, or program failure.A theory failure occurs when the policy was implemented as intended, but failed to have the desired effect. This may occur when, for example, a school adopts school uniforms to curb violence in the school, but the violence remains at the same level.
230 The policy was implemented (uniforms were adopted) but the expected change did not occur. The theory that violence occurs due to style of dress is wrong. There must be some other cause of school violence, which would require a different policy to address.
231 An implementation failure occurs when the policy is not implemented as intended. For example, the school may adopt a uniform policy, but the majority of the students ignore it. The level of violence in the school does not change. We still do not know whether adopting school uniforms would lower the level of violence in the schools; we only know that uniforms were not adopted.
232 FORMATIVE EVALUATIONIf adequate monitoring processes are in effect, it should be fairly easy to detect whether a policy has been implemented as intended. This type of policy monitoring has been referred to as formative evaluation. Formative evaluation documents and analyzes how a policy is implemented, with the objective of making improvements as the implementation process unfolds.
233 SUMMATIVE EVALUATIONSummative evaluation is conducted after a program has been fully implemented. It looks at whether the program is meeting its objectives, and why or why not.
234 Evaluations may be unpopular for many reasons: 1) the program is controversial;2) there are strong political interests in seeing it succeed or fail;3) there are difficulties in measuring program accomplishments;4) those involved may be uncooperative;5) program effects may be influenced by outside developments.
235 To decide whether an evaluation will be helpful, the answer to the following questions should be "yes":1) will the evaluation be accepted by politicians, administrators, and/or participants?2) has an evaluator been involved from the beginning?3) are there measurable objectives?4) are data available?5) are multiple evaluation methods plausible?6) has the program remained stable over time?7) can program staff become involved in the evaluation?8) will the findings be made widely available?
236 EVALUATION DESIGNPolicy evaluation applies accepted social science research methods to public programs. The same research designs used in laboratory experiments are not always practicable in the field, but the same principles can guide the planning and execution of policy evaluation.
237 Before-and-After Evaluation: a policy is evaluated for the changes it has produced since its implementation; the situation is controlled to exclude other possible influences on the outcome.With-and-Without Evaluation: a policy is evaluated for producing changes in the target population, compared to another population without the policy.
238 After-Only Evaluation: the extent to which the policy goals were achieved, compared to the state of affairs before the policy was implemented; but the situation is not controlled to exclude other possible influences on the outcome.Time-Series Evaluation: the changes produced by the policy, tracked over a long time period.