1. PPA 503 – The Public Policy-Making Process
Lecture 9a - Evaluation

2. Evaluating Public Programs
Program evaluation is a way of bringing to public decision-makers the available knowledge about a problem, about the relative effectiveness of past and current strategies for addressing or reducing that problem, and about the observed effectiveness of particular programs.

3. Administrative Purposes for Evaluation
Policy formulation – to assess or justify the need for a new program and to design it optimally on the basis of past experience.
Information on the problem addressed by the program: how big is it? What is its frequency and direction? How is it changing?
Information on the results of past programs that dealt with the problem: were those programs feasible? Were they successful? What difficulties did they encounter?
Information allowing the selection of one program over another: what are the comparative costs and benefits? What kinds of growth records were experienced?

4. Administrative Purposes for Evaluation
Policy execution – to ensure that a program is implemented in the most cost-effective and technically competent way.
Information on program implementation: how operational is the program? How similar is it across sites? Does it conform to the policies and expectations formulated? How much does it cost? How do stakeholders feel about it? Are there delivery problems or error, fraud, and abuse?

5. Administrative Purposes for Evaluation
Policy execution – to ensure that a program is implemented in the most cost-effective and technically competent way.
Information on program management: what degree of control exists over expenditures? What are the qualifications and credentials of the personnel? What is the allocation of resources? How is program information used in decision making?
Ongoing information on the current state of the problem or threat addressed in the program: is the problem growing? Is it diminishing? Is it diminishing enough so that the program is no longer needed? Is it changing in terms of its significant characteristics.

6. Administrative Purposes for Evaluation
Accountability in public decision making – to determine the effectiveness of an operating program and the need for its continuation, modification, or termination.
Information on program outcomes or effects: what happened as a result of program implementation?
Information on the degree to which the program made or is making a difference: what change in the problem or threat has occurred that can be directly attributed to the program?
Information on the unexpected (and expected) effects of the program.

7. Functions and Roles of Evaluation Sponsors
Executive branch (federal, state, local).
Program managers (cost-effectiveness).
Agency heads and top policy makers (need, effectiveness).
Central budget or policy authorities (effectiveness, need).

8. Functions and Roles of Evaluation Sponsors
Legislative branch:
Congressional and legislative policy and evaluation offices (all aspects).
Legislative authorization, appropriations, and budget committees (program funding and refunding).
Oversight committees (all aspects).
Regardless of sponsor, evaluators should clearly specify the objectives and limitations of each evaluation.

9. Functions and Roles of Evaluation Sponsors
As a general rule, public administrators should expect their work on program effectiveness and feasibility to be of more general use than their work on implementation, which will be of most use to program managers and agency heads.
Information needs will be larger for large programs than small, new programs over old.

10. Evaluation Approaches
Front-end analysis – evaluative work conducted before a decision to move ahead with a program.
Evaluability assessment – reasonableness of assumptions and objectives, comparison of objectives to program activities, feasibility of full-scale evaluation.

11. Evaluation Approaches
Process evaluation – describe and analyze the processes of implemented program activities – management strategies, operations, costs, interactions, etc.
Effectiveness or impact evaluation – how well has a program been working? Are the changes the result of the program?

12. Evaluation Approaches
Program and problem monitoring – continuous rather than snapshot – inform on problem characteristics or track program or problem progress in several areas.
Metaevaluation or evaluation synthesis – reanalyzes findings from several analyses to determine what has been learned.

13. Evaluation Approaches

14. Introduction to Evaluation Procedures
Program evaluation is the use of social research methods to systematically investigate the effectiveness of social intervention programs.
Draws on techniques and concepts of social science disciplines
Intended to be used for improving programs and informing social action aimed at ameliorating social problems.

15. Introduction to Evaluation Procedures
Modern evaluation research grew from pioneering efforts in the 1930s and burgeoned in the post-war years as new methodologies were developed.
The social policy and public administration movements have contributed to the professionalization of the field and to the sophistication of the consumers of evaluation research.

16. Introduction to Evaluation Procedures
The need for program evaluation is undiminished in the 2000s and may even be expected to grow.
Contemporary concern over the allocation of scarce resources makes it more essential than ever to evaluate the effectiveness of social interventions.

17. Introduction to Evaluation Procedures
Evaluation must be tailored to the political and organizational context of the program to be evaluated.

18. Introduction to Evaluation Procedures
The assessment of one or more program domains:
The need for the program
The design of the program
The program implementation and service delivery
The program impact or outcomes
Program efficiency
Accurate description of program performance and assessment against relevant standards or criteria.

19. Introduction to Evaluation Procedures
Program evaluation presents many challenges to the evaluator
Changes in circumstances and activities during an evaluation.
Appropriate balance between science and pragmatism.
Diversity of perspectives and approaches.

20. Introduction to Evaluation Procedures
Most evaluators are trained as social scientists or social researchers.
Complex evaluations may require specialized staffs.
Basic knowledge is good for researchers and consumers.

21. Tailoring evaluations
Every evaluation must be tailored to the circumstances of the program to yield credible and useful answers to specific questions while still allowing practical implementation.

22. Tailoring evaluations
Influences on evaluation plans include the purpose of the evaluation.
Provide feedback for program improvement to program managers and sponsors.
Establish accountability to decision-makers with responsibility to ensure that the program is effective.
Contribute to knowledge about some form of social intervention.

23. Tailoring evaluations
Influences also include the nature of program structure and circumstances.
The program must be responsive to:
How new or open to change the program is.
The degree of consensus or conflict among stakeholders about the nature and mission of the program.
The values and concepts inherent in the program rationale and design.
The way in which the program is organized and administered.

24. Tailoring evaluations
Evaluation planning must also accommodate limitations on resources.
Resources include:
Funding;
Time for completion;
Pertinent technical expertise;
Program and stakeholder cooperation;
Access to important records and program material.
Balance between what is desirable and what is feasible.

25. Tailoring evaluations
The evaluation design can be structured around three issues.
The questions the evaluation is to answer;
The methods and procedures to be used to answer these questions;
The nature of the evaluator-stakeholder interactions during the course of the evaluation.

26. Tailoring evaluations
Deciding on the appropriate relationship between the evaluator and the evaluation sponsor, as well as other major stakeholders, is an often neglected, but critical aspect of an evaluation plan.
Independent is often expected
But participatory or collaborative may enhance stakeholders skills or political influence.

27. Tailoring evaluations
Evaluation questions and methods fall into five categories:
Need for services;
Program conceptualization and design;
Program implementation;
Program outcomes; and
Program efficiency.

28. Tailoring evaluations
Evaluation terms corresponding to these categories include needs assessment, process evaluation, and impact assessment.
Much of evaluation planning consists of identifying the evaluation approach corresponding to the type of questions to be answered and tailoring specifics to the program situation.

29. Identifying issues and formulating questions
A critical phase in evaluation planning is the identification and formulation of the questions that the evaluation will address.
These questions focus the evaluation on the areas of program performance most at issue for key stakeholders and guide the design so that it will provide meaningful information about program performance.

30. Identifying issues and formulating questions
Good evaluation questions must identify clear, observable dimensions of program performance that are relevant to the program’s goals and represent domains in which the program can realistically be expected to have accomplishments.

31. Identifying issues and formulating questions
What most distinguishes evaluation questions, however, is that they involve criteria by which the identified dimensions of program performance can be judged.
If the formulation of the evaluation questions can include performance standards on which key stakeholders agree, evaluation planning will be easier and the potential for disagreement with the results reduced.

32. Identifying issues and formulating questions
To ensure that matters of greatest significance are covered in the evaluation design, the evaluation questions are best formulated through interaction and negotiation with the evaluation sponsors and other stakeholders representative of significant groups or distinctly positioned in relation to program decision-making.

33. Identifying issues and formulating questions
Although stakeholder input is critical, the evaluator must be prepared to identify program issues that warrant inquiry.
Evaluator should conduct a somewhat independent analysis of the assumptions and expectations on which the program is based.

34. Identifying issues and formulating questions
Make the program theory explicit.
Program Theory
Program’s Organizational Plan
Service Utilization Plan
Impact Theory

35. Identifying issues and formulating questions

36. Identifying issues and formulating questions
Program theory describes the assumptions inherent in a program.
Encompasses impact theory, which links program actions to intended outcomes; and
Process theory, which describes a program’s organizational plan and scheme for ensuring utilization of its services by the target population.

37. Identifying issues and formulating questions
When these procedures have generated a full set of evaluation questions, evaluator must organize them into related clusters.
Draw on stakeholder input and professional judgment to set priorities.
With the priority evaluation questions determined, evaluator is ready to design the part of the evaluation devoted to answering them.

38. Meeting the Need for Evaluation
Three basic questions
Can the results of the evaluation influence decisions about the program?
Can the evaluation be done in time to be useful?
Is the program significant enough to merit evaluation?

39. Choices Facing Evaluators
Evaluation design
What are the evaluation questions?
What comparisons are needed?
What measurements are needed?
How will the resulting information be used?
What “breakouts” (disaggregations of data) are needed, such as by facility or type of client?

40. Choices Facing Evaluators
Data Collection
What are the primary data sources?
How should data be collected?
Is sampling required? Where and how?
How large a sample is needed?
How will data quality be ensured?

41. Choices Facing Evaluators
Data Analysis
What analytical techniques are available (given the data)?
Which analytical tools will be most appropriate?
In what format will the data be most useful?
Getting Evaluation Information Used
How should evaluation findings be packaged for different audiences?
Should specific recommendations accompany evaluation reports to encourage action?
What mechanisms can be used to check on implementation of recommendations?