Presentation on theme: "PPA 691 – Seminar in Public Policy Analysis Lecture 9 – Monitoring Policy Outcomes."— Presentation transcript:
PPA 691 – Seminar in Public Policy Analysis Lecture 9 – Monitoring Policy Outcomes
Introduction The consequences of policy actions are never fully known in advance and, for this reason, it is essential to monitor policy actions after they have occurred. In fact, policy recommendations may be viewed as hypotheses about the relationship between policy actions and policy outcomes: if action A is taken at time t 1, outcome O will result at time t 2.
Monitoring in Policy Analysis Monitoring is the policy-analytic procedure used to produce information about the causes and consequences of public policies. Monitoring, since it permits analysts to describe relationships between policy-program operations and their outcomes, is the primary source of knowledge about policy implementation. Monitoring is primarily concerned about with establishing factual premises about public policy.
Monitoring in Policy Analysis Monitoring performs at least four major functions: –Compliance: monitoring helps determine whether the actions of program administrators, staff, and other stakeholders are in compliance with standards and procedures imposed by legislatures, regulatory agencies, and professional bodies. –Auditing: monitoring helps determine whether resources and services intended for certain target groups and beneficiaries have actually reached them.
Monitoring in Policy Analysis Monitoring performs at least four major functions: –Accounting: monitoring produces information that is helpful in accounting for social and economic changes that follow the implementation of broad sets of public policies and programs over time. –Explanation: monitoring also yields information that helps to explain why the outcomes of public polices and programs differ.
Monitoring in Policy Analysis Sources of information. –Information must be: Relevant. –Macronegative versus micropositive. Reliable. –Observations precise and dependable. Valid. –Information about policy outcomes should measure what we think it is measuring.
Monitoring in Policy Analysis Sources of information. –Information on policy outcomes. Historical Statistics of the United StatesCurrent Opinion Statistical Abstract of the United StatesUnited States Census of Population by States County and City Data BookCongressional District Data Book Census Use StudyNational Opinion Research Center General Social Survey Social and Economic Characteristics of StudentsSurvey Research Center National Election Studies Educational Attainment in the United StatesNational Clearinghouse for Mental Health Information Current Population ReportsNational Clearinghouse for Drug Abuse Information The Social and Economic Status of the Black Population in the United States National Criminal Justice Reference Service Female Family HeadsChild Abuse and Neglect Clearinghouse Project Monthly Labor ReviewNational Clearinghouse on Revenue Sharing Handbook of Labor StatisticsSocial Indicators Congressional QuarterlyState Economic and Social Indicators Law Digest
Monitoring in Policy Analysis Sources of information. –When information is not available from existing sources, monitoring may be carried out by some combination of questionnaires, interviewing, field observation, and the use of agency records.
Monitoring in Policy Analysis Types of policy outcomes. –Consequences: Policy outputs – the goods, services, or resources received by target groups and beneficiaries. Policy impacts – actual changes in behavior and attitudes that result from policy outputs. –Populations: Target groups – individuals, communities, or organizations on whom policies and programs are designed to have an effect. Beneficiaries – groups for whom the effects of policies are beneficial or valuable.
Monitoring in Policy Analysis Types of policy actions. Policy actions have two major purposes. –Regulation – actions designed to ensure compliance with certain standards or procedures. –Allocation – actions that require inputs of money, time, personnel, and equipment. –Regulative and allocative actions may have consequences that are distributive or redistributive.
Monitoring in Policy Analysis Types of policy actions. –Policy inputs – the resources – time, money, personnel, equipment, and supplies – used to produce outputs and impacts. –Policy processes – the administrative, organizational, and political activities and attitudes that shape the transformation of policy inputs into policy outputs and impacts.
Monitoring in Policy Analysis ISSUE AREA POLICY ACTIONSPOLICY OUTCOMES InputsProcessesOutputsImpacts Criminal Justice.Dollar expenditures for salaries, equipment, maintenance. Illegal arrests as percentages of total arrests. Criminals arrested per 100,000 known crimes. Criminals convicted per 100,000 known crimes. Municipal Services. Dollar expenditures for sanitation workers and equipment. Morale among workers. Total residences served. Cleanliness of streets. Social welfare.Number of social workers. Rapport with welfare recipients. Welfare cases per social worker. Standard of living of dependent children.
Monitoring in Policy Analysis Definitions and indicators. –Variables versus constants. –Definitions. Constitutive definitions – gives meaning to words used to describe variables by using synonymous words. –Provide no concrete rules or guidelines for actually monitoring changes. Operational definitions – gives meaning to a variable by specifying the operations necessary to experience and measure it. Indicators – directly observable characteristics.
Approaches to Monitoring APPROACH TYPES OF CONTROL TYPE OF INFORMATION REQUIRED Social systems accountingQuantitative Available and/or new information Social experimentation Direct manipulations and quantitative New information Social auditing Quantitative and/or qualitative New information Research and practice synthesis Quantitative and/or qualitative Available information
Approaches to Monitoring
Social Systems Accounting An approach and set of methods that permit analysts to monitor changes in objective and subjective social conditions over time.
Social Systems Accounting The major analytic element of social systems accounting is the social indicator. –Statistics that measure social conditions and changes therein over time for various segments of a population. By social conditions, we mean both the external (social and physical) and the internal (subjective and perceptional) contexts of human existence in a given society.
Social systems accounting AREAINDICATOR Health and illnessPersons in state mental hospitals Public safetyPersons afraid to walk alone at night EducationHigh school graduates aged 25 and older EmploymentLabor force participation by women IncomePercent of population below the poverty line HousingHouseholds living in substandard units Leisure and recreationAverage annual paid vacation in manufacturing PopulationActual and projected population Government and politicsQuality of public administration Social values and attitudesOverall life satisfaction and alienation Social mobilityChange from father’s occupation Physical environmentAir pollution index Science and technologyScientific discoveries Representative social indicators.
Social Systems Accounting Must assume that the reasons for indicator change are related to policy actions. This is not always the case. Advantages. –Identify areas of insufficient information. –When indicators provide reliable information, possible to modify policies.
Social Systems Accounting Limitations. –Choice of indicators influenced by values of analysts. –Social indicators frequently do not cover manipulable policy actions. –Most social indicators are based available data about objective (rather than subjective) social conditions. –Social indicators provide little information about how inputs are transformed into outcomes.
Social Systems Accounting
Social Experimentation Use of social indicators often leads to random innovation rather than systematic change. Social experimentation is the process of systematically manipulating policy actions in way that permits more or less precise answers to questions about the sources of change in policy outcomes.
Social Experimentation Characteristics and procedures. –Direct control of experimental treatment. –Comparison (control) groups. –Random assignment. Goal: internal validity – the capacity of experiments and quasi-experiments to make valid causal inferences about the effects of actions on outcomes.
Social Experimentation Threats to internal validity. –History. –Maturation. –Instability. –Instrumentation and testing. –Mortality. –Selection. –Regression artifacts.
Social Experimentation Social experimentation is weakest in the area of external validity or generalizability. Neglects policy processes.
Seattle-Denver Income Maintenance Experiment A social experiment is a field test of one or more social programs — or, to use the phraseology of the natural sciences, a test of one or more "treatments." A social experiment is a field test in the sense that families or individuals are actually enrolled in a pilot social program offering some type of special benefit or service.
Seattle-Denver Income Maintenance Experiment It is "experimental" in the sense that families or individuals are enrolled in each of the tested programs on the basis of a random assignment process, for example, the flip of a coin.
Seattle-Denver Income Maintenance Experiment To draw conclusions about the effects of the treatment, it is necessary to collect information about the people who are enrolled in each experimental program and about those who receive no special treatment (called the control group), and then to compare them on the basis of the collected information.
Seattle-Denver Income Maintenance Experiment The primary focus of SIME/DIME — the effect of income maintenance on work effort or labor supply — was a good subject for experimentation. An increase in unearned income or a decrease in the effective wage rate were hypothesized to lead to decreases in labor supply.
Seattle-Denver Income Maintenance Experiment In the case of SIME/DIME, the experiment was in fact designed to test the effect of two different kinds of social programs on participant work effort. The two policies were a variety of cash transfer or negative income tax programs and several combinations of job counseling and education or training subsidy programs.
Seattle-Denver Income Maintenance Experiment Any observed effects of the experiment was interpreted as the differential effects of the experimental treatment compared to existing government programs. Thus any observed experimental-control differences in outcomes must be interpreted as estimates of the effect of replacing the early 1970s status quo with the experimental programs.
Seattle-Denver Income Maintenance Experiment Table 4. Labor Supply Response of Husbands: A. Overall NIT response (Percentage difference in annual hours worked) Year Sample Group year sample year sample Total Sample ——
Seattle-Denver Income Maintenance Experiment Table 5. Labor Supply Response of Wives: A. Overall NIT response (Percentage difference in annual hours worked) Year Sample Group year sample year sample Total Sample ——
Seattle-Denver Income Maintenance Experiment Table 6. Labor Supply Response of Female Heads: A. Overall NIT response (Percentage difference in annual hours worked) Year Sample Group year sample year sample Total Sample ——
Social Auditing One of the limitations of social systems accounting and social experimentation is that both approaches neglect or oversimplify policy processes.
Social Auditing Social auditing explicitly monitors relations among inputs, processes, outputs, and impacts in an effort to trace policy inputs from disbursement to final recipient. Social auditing helps to determine whether ineffective outcomes are the result of inadequate inputs or processes that divert resources or services from intended target groups or beneficiaries.
Social Auditing Processes monitored of two main types: –Resource diversion (from target or beneficiary groups). –Resource transformation (changes in meaning of policy actions from administrator to recipient).
Social Audit Steps The three phases of a social audit Phase 1: Design and data collection –Clarify the strategic focus –Design instruments, pilot test –Collect information from households and key informants in a panel of representative communities
Social Audit Steps Phase 2: Evidence-based dialogue and analysis –Link household data with information from public services –Analyze findings in a way that points to action –Take findings back to the communities for their views about how to improve the situation –Bring community members into discussion of evidence with service providers/planners.
Social Audit Steps Phase 3: Socialization of evidence for public accountability –Work-shopping –Communication strategy –Evidence-based training of planners and service providers –Media training –Partnerships with civil society
Research and practice synthesis. An approach to monitoring that involves the systematic compilation, comparison, and assessment of the results of past efforts to implement public policies. Two primary sources of available information. –Case studies of policy formulation and implementation. –Research reports that address relations among policy actions and outcomes.
Research and practice synthesis. Two methods. –Case survey method – a set of procedures used to identify and analyze factors that account for variations in the adoption and implementation of policies. Requires case coding scheme, a set of categories that capture key aspects of policy inputs, processes, outputs, and impacts.
Research and practice synthesis Two methods. –Research survey method – a set of procedures used to compare and appraise results of past research on policy actions and outcomes. Yields empirical generalizations about sources of variation in policy outcomes, summary assessments of the confidence researchers have in these generalizations, and policy alternatives or action guidelines that are implied in these generalizations. Requires the construction of a format for extracting information about research reports.
Research and practice synthesis Advantages. –Comparatively efficient way to compile and appraise an increasingly large body of cases and research reports on policy implementation. –The case survey method is one among several ways to uncover different dimensions of policy processes that affect policy outcomes. –The case survey method is a good way to examine both objective and subjective conditions.
Research and practice synthesis Limitations. –All related to reliability and validity of information.
Research and Practice Synthesis: Example David H. Greenberg; Charles Michalopoulos; Philip K. Robins. Do Experimental and Nonexperimental Evaluations Give Different Answers about the Effectiveness of Government- Funded Training Programs?
Research and Practice Synthesis: Example Abstract This paper uses meta-analysis to investigate whether random assignment (or experimental) evaluations of voluntary government-funded training programs for the disadvantaged have produced different conclusions than nonexperimental evaluations. Information includes several hundred estimates from 31 evaluations of 15 programs that operated between 1964 and 1998.
Research and Practice Synthesis: Example The results suggest that experimental and nonexperimental evaluations yield similar conclusions about the effectiveness of training programs, but that estimates of average effects for youth and possibly men might have been larger in experimental studies.
Research and Practice Synthesis: Example The results also suggest that variation among nonexprimental estimates of program effects is similar to variation among experimental estimates for men and youth, but not for women (for whom it seems to be larger), although small sample sizes make the estimated differences somewhat imprecise for all three groups.
Techniques for Monitoring APPROACH GRAPHIC DISPLAYS TABULAR DISPLAYS INDEX NUMBERS INTER- RUPTED TIME- SERIES ANALYSIS CONTROL- SERIES ANALYSIS REGRES- SION DIS- CONTI- NUITY ANALYSIS Social systems accountingXXXXXO Social auditingxxxxxo Social experimen- tationxxxxxx Research and practice synthesisxxoooo