Presentation on theme: "Process Evaluation. Objectives: Overview of process evaluation Steps for developing a process evaluation plan Measures."— Presentation transcript:
Objectives: Overview of process evaluation Steps for developing a process evaluation plan Measures
Process Evaluation - Why Emphasis has been on outcome evaluation to determine program success Some see the bottom line as everything Cant ignore bottom line, but good ideas do not always yield good results Hard evidence of program impact is grounded in how an intervention is implemented and delivered
Process Evaluation - Why A programs lack of success could be related to problems with program design and/or delivery, and failure to reach target population Process evaluation has the potential to ensure successful program implementation as well as increase understanding of outcome results (Babbie & Mouton, 2001; Pawson & Tilley, 2004)
Process Evaluation Formative evaluation: use of evaluation to improve program during the development phase Process evaluation: evaluates actual delivery of services vs. those intended Implementation: evaluation of all of the activities focused on the actual operation of a program Quality assurance: systematic process of checking to see whether a product or service being developed is meeting specified requirements
Process Evaluation - What What was the program intended to be? Standards of Care? Best practices? Reach? (Specification of program components, review of historical program data) What is delivered in reality? (Methods for measuring program functioning) Identification of gaps between the intended and the actual program delivery and reach
Process Evaluation - What The use of empirical data to assess the delivery of programs…...verifies what the program is, whether or not it is delivered as intended to the targeted recipients and in the intended dosage (Scheirer, p.40, in Handbook of Practical Program Evaluation, Wholey et al., 1994)
Process Evaluation - Why 1.Provides information on quality of services, and whether program was implemented as planned 2.Assesses whether or not program is reaching the intended recipients, and their level of satisfaction 3.Increases knowledge about program components contributing to outcomes – extent to which theyre working or not working 4.Increases understanding about successful implementation of programs in complex organizational and community contexts
Process Evaluation - Why 5.Identifies and addresses program implementation problems in order to improve the program 6.Links program processes to program outcomes 7.Identifies organizational, program, and individual factors facilitating or impeding program implementation
Process Evaluation - Who Funders Agency and program staff Community stakeholders Program participants
Process Evaluation - How Successful programs based on set of standards and objectives: Competent staff Good management Adequate facilities Sufficient funding Producing sufficient outputs to meet client needs Output level Output quality Reaching the intended target population
Process Evaluation – How Six Steps (Steckler & Linnan, 2002) 1.Full description of program: Specification of purpose, underlying theory, objectives, strategies, expected impacts and outcomes. (conveyed in program logic model) 2.Describe all elements consistent with optimum delivery of program – strategies, activities, staffing. This may include identification of factors (internal and external) contributing to program implementation and delivery problems
Process Evaluation Questions - How 3.Develop list of potential process evaluation questions The most useful questions reflect: A diversity of stakeholder perspectives Key components of your program Priority information needs Available resources to answer your key evaluation questions
Process Evaluation Questions - How Process evaluation questions address the Who What When How many in relation to program inputs, activities, outputs, and target population – the left hand side of your program logic model
Process Evaluation - Categories Steckler and Linnan (2002) recommend a minimum of four elements or categories of process evaluation: Program activities delivered Program activities received Reach Fidelity They also recommend documenting recruitment procedures and describing the context of the intervention.
Process Evaluation Questions - How Coverage – extent of target population participation in program Activity, service, or agency level System level Service Delivery – extent to which program components are being delivered as planned Activity, service, or agency level System level
Questions about coverage – At the activity, service, or agency level Has the activity, service, or agency served the intended clients? What were the demographic and clinical characteristics of clients? What proportion of clients completed treatment, how many sessions were completed, and at what point did they drop- out? What were the characteristics of those who dropped out?
Questions about coverage – At the system level How many treatment programs exist for this target population within the region? How many clients are seen by each program in a year? Are there differences in the types of clients seen at each program?
Questions about service delivery – At the activity, service, or agency levels By what route did clients enter treatment? What services were actually delivered to clients in treatment and is this what was intended? What was the average length of stay or the average number of appointments kept?
Questions about service delivery – At the system level Are different treatment programs aware of one another? Do different treatment programs refer clients to one another? What is the relationship between general medical services and specialized treatment programs?
Process Evaluation Questions - How Steps to identifying questions with staff and stakeholders: Brainstorm questions relating to program, using logic model to guide process Sort questions from brainstorming session into relevant categories
Process Evaluation: Six Steps 4.Determine methods for process evaluation Qualitative and quantitative data sources Qualitative methods: interviews, focus groups, logs, case studies, open-ended survey questions, content analysis of videotaped sessions Quantitative methods: surveys, direct observation, checklists, attendance logs, document review (Baranowski & Stables, 2000; Devaney & Rossi, 1997; McGraw et al, 2000)
Process Evaluation: Six Steps 5.Consider program resources and program characteristics and context Team must consider resources needed to answer questions from step 3 using methods from step 4 Considerations: time for planning, pilot testing instruments, any training needed, data collection, entry, analysis, and reporting Staff time, client burden, disruption to the intervention
Process Evaluation: Six Steps 6.Finalize the process evaluation plan Set priorities and ensure feasibility of plan Determine priorities Important to staff and stakeholders Address important program needs Reflect key elements of program logic model Determine feasibility Can be answered with available resources and within timeframe
Process Evaluation: Methods Qualitative Focus groups Key informant interviews Direct observation Quantitative Chart or record reviews Activity logs Demographics Qualitative and Quantitative In-person interviews (clients, staff, management)
Process Evaluation: Selection of Measures How meaningful is the measure? How feasible is the measure? How useful is the measure?
Process Evaluation: Selection of Measures Questions involving quality Budget process – e.g., review of minutes Quality of services – standard client records Questions re: program operations Checklist of intended programming/services Questions on why a program was developed Planning documents, meeting minutes, funding proposals, needs assessments
Process Evaluation - Methods Program coverage or reach Compare actual participation to intended participation Provides information about the level of program participation by the target group Requires clear definition of the intended target group, monitoring of key target group characteristics, and rate of participation
Process Evaluation – Methods to Evaluate Coverage Develop clear definition of target population Identify key characteristics of target population for monitoring purposes Collect data on identified characteristics Analyze data to determine if actual population served is the intended population Analyze data to determine if persons served by program meet eligibility criteria Using characteristics data, determine which subgroups are over/under represented Analyze characteristics of program drop-outs
Process Evaluation: Methods to Evaluate Delivery Program records: Specify evaluation questions re: program delivery Identify sources of data to answer each question Assess adequacy of current sources of data Develop a data collection plan – sources of data, who will collect data, etc. Develop an analysis plan (e.g., tables, graphs, figures)
Process Evaluation: Methods Client satisfaction or perception of care Client surveys can assess overall satisfaction with services as well as information regarding content of program and manner in which it was delivered Established measures: Client Satisfaction Questionnaire (CSQ-8) Youth Services Survey for Families Consumer-Oriented Mental Health Survey Kansas Family Satisfaction Survey
Process Evaluation: Selection of Measures Standardized measures: Example: The National Inventory of Mental Health Quality Measures (developed by the Center for Quality Assessment and Improvement in Mental Health (CQAIMH): Future launch of the Centres Measures Database
Process Evaluation – Tips for Data Collection Create a schedule for collecting process data and stick to it Divide tasks & assign individuals to each task Maximize existing opportunities for data collection Only identify questions of interest/value and keep the evaluation focused Periodically review your instruments to ensure evaluation approach is still useful
Next Steps March 27: webinar on writing the final report March 31: Award term ends April 30: Final report due May 1 to 14: Evaluation readiness assessment for the Evaluation Implementation Grants (EIG) June 2: Deadline for EIG grant applications June webinar: The road to data collection
For more information Susan Kasprzak / Ext Tanya Witteveen / Ext. 3483