Process Evaluation: Considerations and Strategies CHSC 433 Module 4/Chapter 8 L. Michele Issel UIC School of Public Health

Making Widgets Count not only the widgets, but who gets the widgets and what goes into making the widgets.

Definitions l Systematic examination of coverage and delivery l Measuring inputs into the program l Finding out if the program has all its parts, if the parts are functional and operational

Program Components Discrete interventions, or groups of interventions, of the overall program that are designed to independently or synergistically effect recipients. Objectives per component

Types of Implementation Evaluations l Effort: quantity of input l Monitoring: use of MIS information l Process: internal dynamics, strengths and weaknesses l Component: assess distinct program parts l Treatment: what was supposed to have an effect

Purpose of Process Evaluation l Assurance and accountability l Understanding of outcomes l Mid-course corrections l Program replicability

From Perspective of: l Evaluator l Funders (Accountability) l Management for the program

Stakeholders and Expectations l Focus on the explicit Objectives Program descriptions l Uncover the implicit Review program theory Review objectives Role play possible outcomes

Program Theory Components

Organizational Plan How to garner, configure, and deploy resources, organize program activities so that the intended service is developed and maintained

Service Utilization Plan How the intended target population receives the intended amount of the intended intervention through interaction with the program’s service delivery system

Rigor in Process Evaluation l Appropriateness of the method l Sampling strategy l Validity and Reliability l Timing of data collection

Key Decision: 1 How much effort to expend ~ what data are needed to accurately describe the program. Choose based on: Expected across site variation Making report credible Literature about the intervention

Key Decision: 2 What to look for ~ what program features are most critical, valuable to describe. Choose based on: What is most often cited in program proposal The budget What may be related to program failure

Go Back to Objectives l Process objectives per program component: How much Of what Will be done By who By when

Components and Objectives

Possible Foci of Process Evaluation l Place: Site, Program l People Practitioner/provider Recipient/participant l Processes Activities Processes Structure l Policy

Levels of Analysis l Individuals Program participants Program providers l Programs As a whole l Geographic locations Regions and state

Types of Questions ? l What was done and by whom? l How well was it done and how much was done? l What contributed to success/failure? l How much of what resources were used? l Is there program drift?

Sources of Program Variability l Staff preferences and interest l Materials availability and appropriateness l Participants expectations, receptivity, etc l Site physical environment and organizational support

Roots of Program Failure

Causes of Program Failure l Non-Program No participants No program done l Wrong intervention Not appropriate for the problem l Unstandardized intervention Across site, within program variations

Program Failure cont l Mis-management of program operations l Wrong recipients l Barriers to the program l Program components unevenly delivered, monitored

Data Sources from Program l Resources used l Participant provided data Quality Match with process evaluation l Existing records Sampling of records Validity and reliability issues

Data Sources from Evaluator l Surveys and Interview of Participants l Observation of Interactions l Survey and Interview staff

Evaluating Structural Inputs l Organizational structure Supervision of staff Place in organizational hierarchy l Facilities, equipment l Human resources Leadership Training

Measures of Delivery l Measures of program delivery l Measures of coverage l Measures of effectiveness

Measures of Implementation l Measures of Volume (Outputs): Number of services provided l Measures of Workflow: Client time Staff work time

Targets, Recipients, and Coverage

Measures of Coverage Undercoverage= # recipients in need /# in need Overcoverage= # recipients not in need /# recipients Coverage efficiency= (under - over) x 100

Measures of Effectiveness Effectiveness Index = % reached per program standard per program component Program Effectiveness Index = Sum of Effectiveness Indexes/# program components

Bias in Participation l due to self-selection l results in under or overcoverage l may be related to recruitment l can be identified with good data collection (monitoring)

Measures of Efficiency l ratio of input per output l productivity per staff, per cost, per hour l cost per participant, per intervention l etc...

Evaluating Costs l Payments by agency l Payments by secondary funders l Payments by participants versus charges!

Monitoring and CQI l Similar types of data presentation Control charts Fishbone diagrams Flow charts Gantt charts etc. l Overlapping purposes

Reaching Conclusions l Compare data to objectives l Compare data to needs assessment data l Compare data to other sites or other programs

Worksheet Exercise l For each program objective: What is the focus and level of the process evaluation What data sources needed Who collects data

References Rossi, Freeman & Lipsey (1999). Evaluation: A systematic approach. Sage Publications Patton (1997). Utilization focused evaluation. Sage Publications. King, Morris, Fitz-Gibbon (1987). How to assess program implementation. Sage Publications. Weiss (1972). Evaluation Research. Prentice Hall