2Process Evaluation Objectives: Overview of ‘process evaluation’ Steps for developing a process evaluation planMeasuresAs you know, the framework that all of you have been working on over the last many months has been focused on outcomes as well as process; in other words, measuring client outcomes as well as the way the program is being delivered and whether the program is reaching the intended population. In many ways, this webinar could be perceived as coming too late; most of you will have already gone through the task of identifying process evaluation questions and struggles of how you’re going to objectively measure your own internal program.The emphasis of the Evaluation Capacity Building Grant is on building capacity, so I think the value in this and other webinars is on building knowledge for both enhancing and making modifications to your current framework as well as applying these new skills to further evaluation within your agencies.Over this webinar, I’m planning to identify the various steps that you need to consider in developing a process evaluation plan. Some, of course, will overlap with developing your outcome evaluation plan; however, some are distinctive and need specific time and attention. Lastly, we will address the issue of how to measure process related variables.
3Process Evaluation - Why Emphasis has been on outcome evaluation to determine program successSome see the ‘bottom line’ as everythingCan’t ignore ‘bottom line’, but “good ideas do not always yield good results”‘Hard’ evidence of program impact is grounded in how an intervention is implemented and deliveredA number of years ago, evaluation was all about numbers – how many clients were seen, how many staff hours were spent with clients, etc. From there, evaluation took an about face to the point where emphasis has been solely on the measurement of client outcomes – more specifically, behavioural changes – as a way of determining program success.The mandated use of CAFAS and BCFPI are clear examples of this. I think it’s important that we begin to see that the way that a program is implemented and delivered can have significant impact on client outcomes, and without taking that into consideration, the results of an evaluation can be very misleading. Now I’m not suggesting that counting the number of client participants, and number of clients on the waiting list, or measuring behavioural change over the term of an intervention are not important.Clearly they are, and in fact, the methods for data collection that you are already using should be incorporated into an evaluation plan. What I’m suggesting is that your evaluation plan be more robust and comprehensive so that by the end of the evaluation, the results of the data collection will provide a more reliable picture of the program and more fully inform program decisions.
4Process Evaluation - Why A program’s lack of success could be related to problems with program design and/or delivery, and failure to reach target populationProcess evaluation has the potential to ensure successful program implementation as well as increase understanding of outcome results (Babbie & Mouton, 2001; Pawson & Tilley, 2004)Let’s say, for example, that your evaluation focuses only on outcome measures, and your results suggest a deficit in care. The problem with this, however, is that the results do not provide clues as to the causes of the deficit. Process measures, on the other hand, which assess the component parts of the actual treatment provided, can be used to identify sub-optimal practice and provide direction for improvement of program activities.Process evaluation also adds a qualitative dimension to the statistics being gathered. You might learn, for example, how many counseling sessions of what duration were received by participants in a drug treatment program; or how many new cases were opened. A process evaluation would expand these indicators by providing information on particular modalities of treatment. In addition to furnishing richer data on some of the more quantitatively elusive aspects of program operations, a process assessment can also capture important information about a program's social, cultural, and political context. Since, for example, local cultures play a key role in shaping the direction and results of new program policies and approaches, understanding the local environment may be valuable to understanding how a program works.Field observation, reviews of case reports and interviews of program personnel, agency staff, key personnel in related agencies, and community and political leaders, are all important sources of descriptive and historical data on the program and the environment in which it operates. These data serve a number of important purposes:First and foremost, understanding exactly how a program operates (and how it may be influenced by various environmental factors) can be a great way of investigating and interpreting program impact.Qualitative data are also essential for interpreting the results of a quantitative inquiry. It makes no sense, for instance, to declare that a program had no discernable effect on its target population or area, if observation has revealed that the intended activities described by the program objectives were delivered in a qualitatively different way than intended.Finally, in cases where program impacts cannot be empirically measured, a process evaluation or case study may serve as a surrogate source of information about the nature of a particular program. Although the information will necessarily focus more on "what happened" than whether particular goals were achieved, case material may nonetheless shed important light on some of the apparent strengths and limitations of the program.When you are evaluating process, there are three possible resultant scenarios – program intervention sets into motion a causal process that leads to the desired effect; program intervention does not contribute to causal process which would have led to desired effect (theory of change failed); program intervention contributes to causal process but does not lead to desired effect (effect theory failed). Weiss, C.H.In the first instance, you’re celebrating your successes with staff and perhaps thinking about ways of replicating or expanding the program. In the second instance, there are clear issues with the way the program is being delivered. In the third instance, the program is being delivered in the intended manner, but it’s not making a different within the targeted population. Perhaps in the latter case, a different or modified approach needs to be considered; perhaps the approach that is being used is not appropriate given the target population.
5Process EvaluationFormative evaluation: use of evaluation to improve program during the development phaseProcess evaluation: evaluates actual delivery of services vs. those intendedImplementation: evaluation of all of the activities focused on the actual operation of a programQuality assurance: systematic process of checking to see whether a product or service being developed is meeting specified requirementsOkay; so you’re starting to understand the value of process evaluation. So you decide to do some further reading as to what process evaluation is all about and how to go about conducting a process evaluation. The problem is that there are so many terms out there to describe what appears to be the same approach. This can be very confusing.Essentially all of these terms – formative, process, implementation, and quality assurance – describe multiple evaluative approaches to effective program management. Formative evaluation is more appropriate to a piloted project; to identify and make necessary changes to improve implementation and delivery prior to full implementation.Quality assurance or quality improvement – another process that can be incorporated with process evaluation as a management tool assesses a set of standards or tools
6Process Evaluation - What What was the program intended to be? Standards of Care? Best practices? Reach? (Specification of program components, review of historical program data)What is delivered in reality?(Methods for measuring program functioning)Identification of gaps between the intended and the actual program delivery and reachProcess and ‘mplementation evaluation, on the other hand, are often used interchangeably. However, I think the one distinction between the two terms is the fact that process evaluation is comparative, measuring what is being delivered with what was intended. Implementation evaluation, on the other hand, focuses on all inputs, activities, and outputs and assesses their current functioning without necessarily making any comparisons.
7Process Evaluation - What “The use of empirical data to assess the delivery of programs…...verifies what the program is, whether or not it is delivered as intended to the targeted recipients and in the intended dosage”(Scheirer, p.40, in Handbook of Practical Program Evaluation, Wholey et al., 1994)So exactly how is process evaluation defined? Sheirer identifies it as … But I like Patton’s (1997) more simplified way of defining process evaluation. He says that process evaluation is all about “finding out if the program has all its parts, and if the parts are functioning as they are supposed to be functioning.
8Process Evaluation - Why Provides information on quality of services, and whether program was implemented as plannedAssesses whether or not program is reaching the intended recipients, and their level of satisfactionIncreases knowledge about program components contributing to outcomes – extent to which they’re working or not workingIncreases understanding about successful implementation of programs in complex organizational and community contexts
9Process Evaluation - Why Identifies and addresses program implementation problems in order to improve the programLinks program processes to program outcomesIdentifies organizational, program, and individual factors facilitating or impeding program implementationHere are some additional reasons for conducting a process evaluation:Provides positive reinforcementProcess evaluation provides the reinforcement that you’ll need to keep your project going; there is nothing like positive feedback to boost your morale and that of your group members.Highlights errorsIt will reveal the errors and miscalculations that are bound to arise. Although it may seem threatening to have your mistakes highlighted in this way, the negative feedback that comes up through the course of the process evaluation can actually provide you with great insights, and help to ensure the quality of your project.Promotes self-analysisProcess evaluation sets up a healthy climate of self-analysis and reflection that is essential in any community-based mental health program. Sometimes it’s helpful to set up regular review sessions, where you can monitor your progress on reaching the goals and objectives of your project. In fact, you’re probably already engaging in management activities relating to program assessment, and these activities can be integrated into an ongoing monitoring of process information.Provides a historical recordIf your project achieves good outcomes, those wishing to replicate it would need a clear description of exactly what was done, and how it was done. If your project does not achieve its objectives, process evaluation findings will help you to pinpoint why - whether your objectives were unrealistic, or whether the implementation or program delivery did not go as planned.Keeps you on trackWithout conducting frequent internal checks, it’s easy for programs to lose their momentum and drift off target. Regular, informal evaluations at the end of meetings (How are we doing? Is everyone feeling well/being heard! What are the challenges/barriers to program delivery? What are your staffing needs?) or even more formal (brief evaluation forms for participants and/or client satisfaction surveys) will help to keep your project on track.
10Process Evaluation - Who FundersAgency and program staffCommunity stakeholdersProgram participantsYou must determine your audience for process evaluation: external audiences include funders, clients, and community advocates; internal audiences include program staff, managers, and the board of directors.These audiences, or stakeholders, need to be involved in the evaluation planning from the beginning stages. And not only involved, but their opinions valued and ideas integrated into the plan. If, for example, your evaluation turns up findings relating to the success of your program and the need to expand services, but you failed to involve your funder until the end of the project, they’re unlikely to attach as much credibility to the findings. In order to ensure they are vested in the results, you must include them from the beginning. And who is responsible for ensuring the implementation of the evaluation findings? Who is going to ensure that needed changes illuminated over the evaluation are made to improve the program? Those persons should be closely involved and informed over the term of the evaluation.The perspectives of program participants, such as youth and parents, are also key stakeholders for the process evaluation. For example, if program participation or retention are the focus of the process evaluation, having feedback from families or youth can provide valuable insights.
11Process Evaluation - How Successful programs based on set of standards and objectives:Competent staffGood managementAdequate facilitiesSufficient fundingProducing sufficient outputs to meet client needsOutput levelOutput qualityReaching the intended target populationAs was mentioned earlier, the whole point of process evaluation is to measure actual program functioning and reach (extent to which you are reaching your intended target population) against the intended program functioning. In the initial planning stages, you must identify the intended optimum delivery of your program. As we go through the six steps of developing a process evaluation, you will understand how this all works…
12Process Evaluation – How Six Steps (Steckler & Linnan, 2002) Full description of program: Specification of purpose, underlying theory, objectives, strategies, expected impacts and outcomes. (conveyed in program logic model)Describe all elements consistent with optimum delivery of program – strategies, activities, staffing. This may include identification of factors (internal and external) contributing to program implementation and delivery problemsImportant issues that must be addressed:Understanding your program and how it is supposed to workConsidering program characteristics and context and how these may affect implementationBefore a program’s effectiveness can be judged, it is important to understand its operating environment. The program analysis stage requires a detailed description of the environment at the time of the evaluation. Included must be a description of any problems encountered in the implementation, how they were resolved, whether all planned activities were implemented; if not, what remains to be done, if plans and timetables were revised and why it was necessary, what new objectives were added and why, what changes occurred in leadership or staffing and what effect these changes have had, whether costs exceeded initial projections and why, support from referral sources and programs, lessons learned. For programs that have been functioning for some time, you will need to gather this information from program documentation.A process evaluation should provide detailed information about the program as it was actually implemented and as it is actually functioning in order to determine what is working and what is not working. A thorough process evaluation should include the following elements:Description of the program environmentDescription of process used to design and implement the programDescription of program operationsIdentification and description of intervening events that may have affected implementation and outcomesDocumentation such as meeting minutes, reports, memos, newsletters, and forms can be used in determining whether the intended has been achievedConsider how the goals and objectives of the program are translated into practice. What are the major project activities and how do these relate to the anticipated outcomes? Why and how is the program supposed to work? What are the underlying reasons the intervention is presumed to be effective? Who is the intended target population? What are the expectations of partnering programs. With respect to this last item – context – generally, there are participating agencies and partners as well as your target population – children/youth, and their caregivers/families. Assessment of program delivery must take into account the surrounding social system – not only the characteristics of the organization in which the program is being administered.All of these questions relate to the theory of the program. The program theory is a statement of the mechanisms through which the intervention is to work. Most importantly from an evaluation standpoint, it tells us what constructs to measure. One of the initial tasks over the term of the capacity-building grant was to develop a program logic model. This diagram essentially outlines your program theory.One of the subsequent tasks of evaluation planning is to construct a clear explication of the theory behind the program. What should the program deliver that will result in the intended changes within the targeted population? Each potential area of program emphasis implies a different casual process designed to change behaviour. Although program designers and administrators may not always claim to have employed theory in the creation of the program, theory is implicit in all forms of intervention.The literature review you have conducted should also shed light on your program theory, and to help identify what program elements contribute to changes in the outcomes. You may also want to consider here how new staff are oriented or trained on the program. If there are several staff delivering the program, how do you ensure everyone is on the same page? While it may appear quite straightforward to specify what the program is, specifying the content, what is actually going on, may be more difficult. There may be substantial variation in operational procedures, and among program staff. The more complex the program and varied the components, the more difficult and crucial this task is. You can see how potentially valuable this stage of the evaluation planning can be towards building program consensus, and the tremendous value of involving all program staff in the planning stages.Step 2 – describe high quality implementation and delivery based on direct experience and observing models similar. Here, you will want to describe what the program would look like in the ideal situation.If all these sound too complicated and difficult to do for your program, it may be a sign that your program is not yet ready for an outcome evaluation. The program may need to be further strengthened, and you may need to consider focusing on process evaluation.
13Process Evaluation Questions - How Develop list of potential process evaluation questionsThe most useful questions reflect:A diversity of stakeholder perspectivesKey components of your programPriority information needsAvailable resources to answer your key evaluation questionsWhile process evaluation has the potential to contribute to a better understanding of outcome results, these evaluations can also be flawed with vague questions phrased in unanswerable ways (McClure, Turner, & Yorkstons, 2005).Step 3 involves developing a list of potential process evaluation questions.
14Process Evaluation Questions - How Process evaluation questions address theWhoWhatWhenHow manyin relation to program inputs, activities, outputs, and target population – the left hand side of your program logic model
15Process Evaluation - Categories Steckler and Linnan (2002) recommend a minimum of four elements or ‘categories’ of process evaluation:Program activities deliveredProgram activities receivedReachFidelityThey also recommend documenting recruitment procedures and describing the context of the intervention.
16Process Evaluation Questions - How Coverage – extent of target population participation in programActivity, service, or agency levelSystem levelService Delivery – extent to which program components are being delivered as plannedEssentially, these components boil down to two forms of data around which the process evaluation can be organized: data on the program itself (service delivery), and data about the program participants (coverage). Program specific data would include the purposes of the program taken from program statements as well as staff descriptions, resource allocation, methods of operation, day to day procedures, staffing patterns, location, size of program, management structure, and inter-organizational relationships. Plus, every evaluation should carefully document the content, duration, and intensity of treatment involved in the intervention.Service delivery can be assessed from both an internal and external perspective. For example, agencies are increasingly comparing performance data against a set of external benchmarks – the same measurement data for similar agencies and programs (Bruder and Gray, 1994; Morley, Bryant, and Hatry, 2001). This can help an agency assess its own performance and set future target levels.The second type of input data – coverage – concerns characteristics of program clients. Coverage relates to how people enter the treatment service and what happens to them once they’re there. Data includes demographic and personal characteristics and assessments gathered at the initial point of intake. You can use this data to make comparisons between the actual and the intended target population. In addition, depending upon their relevance to the program, you may also wish to collect data on the attitude and perspective of program participants on a variety of issues that may be related to his or her performance in the project. These areas might include the youth's relationship to family and peers, attitude about the program, motivation for participation, and self concept. While the program may be reasonably expected to alter some of these factors, others are quite impervious to change. It is useful to collect data on these and other control variables to determine the types of clients who are more likely to be successful in the program.Measures: BCFPI, a standardized intake form, clinical assessment tools, motivation for treatment scale (e.g., Treatment Entry Questionnaire – Wild 1996)
17Questions about coverage – At the activity, service, or agency level Has the activity, service, or agency served the intended clients?What were the demographic and clinical characteristics of clients?What proportion of clients completed treatment, how many sessions were completed, and at what point did they drop-out? What were the characteristics of those who dropped out?CASE EXAMPLE: Substance-abuse programEVALUATION QUESTIONS: Has the number of referrals from external sources increased from the previous year? What percentage of clients are ages 15-18? What is the average age of program participants from each of the referral sources?HOW WAS DATA COLLECTED? As each client was admitted to the program, counselors recorded required demographics. At the end of the year, all forms were forwarded to the data clerk for tabulationHOW WAS DATA ANALYZED? Number and percent of clients from each referral source was calculated and compared to the previous year’s figures. Average age overall was calculated as well as average age from each referral source.WHAT DID THEY FIND OUT? The results indicated that the number of referrals from the legal system and from schools had increased substantially from the previous year. 87% of participants were in the age group. The average age was 17.8 years. Average age did not vary based on referral source.WHAT DID THEY DO WITH THE INFORMATION? Program staff anticipate continuing increased numbers of referrals and plans to expand program in the coming year.
18Questions about coverage – At the system level How many treatment programs exist for this target population within the region?How many clients are seen by each program in a year?Are there differences in the types of clients seen at each program?CASE EXAMPLE: New substance abuse program for youth within a particular region with other substance abuse programsEVALUATON QUESTIONS – What percent of clients live in the outlying area and what percent come from within city boundaries?Are clients from the different programs different in terms of age, sex or level of substance abuse?RESOURCES NEEDED?Managers from different centres collaborated to devise a one page data collection form. Distributed to all clinicians.HOW WAS DATA COLLECTED?Upon admission to each program, the admitting clinician competed a one page form including client’s age, sex, address, and substance use (type and frequency). Forms were forwarded to a central site where they were recorded in a log book and maintained by the program data managerHOW WAS DATA ANALYZED?Client data was divided according to different substance abuse sites. Within the new program site, clients’ addresses were coded for (1) living within the outlying area or (2) living within the city. The percentage of clients from each geographic area was calculated. Clients’ demographics were calculated across all of the centres.HOW WAS DATA USED?To ensure no overlap of services and to ensure there are no gaps in delivery of program across the community; to determine whether programs were reaching their target population from both the city and outlying areas; to determine client focus for each of the participating agencies
19Questions about service delivery – At the activity, service, or agency levels By what route did clients enter treatment?What services were actually delivered to clients in treatment and is this what was intended?What was the average length of stay or the average number of appointments kept?CASE EXAMPLE: Substance abuse program: Evaluation questions: What was the average number of days between the intake and the first scheduled appointment? What percentage of clients did not show for their first scheduled appointment?RESOURCES NEEDED TO ANSWER QUESTIONS: Data entry clerk to maintain database on a weekly basis in order to keep statistics on appointments and attendance.HOW WAS DATA COLLECTED? Attendance data was recorded on daily appointment logs kept by the assessment workers. The intake worker kept a record of the initial intake and the date of the initially scheduled assessment appointment.HOW WAS DATA ANALYZED? Person doing the analysis completed descriptive statistics on the average number of days between intake and the first scheduled appointment, and the percent of referrals who attended at least one assessment appointment during the past year.WHAT DID THEY DO WITH THE INFORMATION? Based on the findings that 34% of clients were not meeting the mandated time frame of 2 months between assessment and first appointment, and that 40% were no shows to their appointments, the agency decided to reorganize so that first appointments occurred on a specified day of the week at a specific time with several staff on call during that time. As a result, more patients could be scheduled earlier and the inefficiency of waiting for no shows was greatly reduced.
20Questions about service delivery – At the system level Are different treatment programs aware of one another?Do different treatment programs refer clients to one another?What is the relationship between general medical services and specialized treatment programs?CENTRAL FOCUS: The coordination among specialized treatment programs and between these program and other services that clients may need. First steps in measuring service deliver at the system level is to define the boundaries of the system itself. Focus – degree of communication and collaboration between and among staff in a given network of agencies.FOCUS OF REPORTS: Mutual awareness, frequency of interaction, frequency of cross referrals, information exchange, staff sharing, other resources exchanged (e.g, meeting rooms, materials, etc.)CASE EXAMPLE: Network of substance abuse treatment centres.EVALUATION QUESTIONS: For each centre, from where do referrals originate? What is the waiting time for services within each centre?WHAT RESOURCES? Case workers recorded referral information as part of their intake interviews. Consolidation of this data and tabulation of results were completed by data entry clerk, requiring approximately one hour per month. Each program manager completed an interview and questionnaire that assessed current treatment services.HOW WAS DATA COLLECTED? Data collection occurred in two parts – assessing the referral source, case workers recorded this information as part of their intake interviews. Consolidation of this data and tabulation of results was completed by a data entry clerk on a monthly basis. For the second part, assessing treatment services, two interviewers were dispatched to each program to collect data on waiting times for current treatment.HOW ANALYZED? All information was entered by a data entry clerk into a central records notebook. The clerk calculated the percentage of referrals from each referral source for each program, and categorized waiting times for the different types of treatment services offered.WHAT DID THEY FIND OUT? For the downtown program, the greatest number of clients (37%) were referred by the local emergency department. However, the emergency department tended not to make referrals to outlying clinics, even for clients who lived in these regions. Most referrals from outlying clinics came from family members or were self-referrals. The outlying program overlapped considerably with the downtown program, but tended to have shorter wait times.WHAT DID THEY DO WITH THIS INFORMATION? Evaluators concluded that more education of emergency department staff about the outlying programs (including shorter wait times) was warranted. They instituted a brochure campaign and a series of presentations at emergency department staff meetings to accomplish this goal.
21Process Evaluation Questions - How Steps to identifying questions with staff and stakeholders:Brainstorm questions relating to program, using logic model to guide processSort questions from brainstorming session into relevant categoriesUse logic model to guide development of questions – focus on inputs, activities and outputs to generate specific questions.Priorities should be supported by your funders, clients, staff, board, and any other stakeholders.
22Process Evaluation: Six Steps Determine methods for process evaluationQualitative and quantitative data sourcesQualitative methods: interviews, focus groups, logs, case studies, open-ended survey questions, content analysis of videotaped sessionsQuantitative methods: surveys, direct observation, checklists, attendance logs, document review(Baranowski & Stables, 2000; Devaney & Rossi, 1997; McGraw et al, 2000)A lot of process evaluation measures are transactional data that agencies collect on an ongoing basis – requests for service, clients admitted/discharged, production records, activity logs, treatments administered, follow-up or repeat sessions conducted, complaints from clients, etc. This information is usually maintained in a database and therefore readily accessible. Additional measures may be qualitative (e.g., focus groups, interviews).
23Process Evaluation: Six Steps Consider program resources and program characteristics and contextTeam must consider resources needed to answer questions from step 3 using methods from step 4Considerations: time for planning, pilot testing instruments, any training needed, data collection, entry, analysis, and reportingStaff time, client burden, disruption to the interventionAnd finally, practical considerations and cost must weigh usefulness of the measures and the quality of the data against issues of feasibility, time, effort, and costs.
24Process Evaluation: Six Steps Finalize the process evaluation planSet priorities and ensure feasibility of planDetermine prioritiesImportant to staff and stakeholdersAddress important program needsReflect key elements of program logic modelDetermine feasibilityCan be answered with available resources and within timeframeStep 6 emerges from the planning involved over steps 3 to 5.EXAMPLE of use of process evaluation across the four elements:Program activities delivered – consistency in the approach being used with families.Program activities received – number of sessions actually attended by program participants.Reach – extent to which program is reaching their intended target population.Fidelity – extent to which program has been implemented as intended.
25Process Evaluation: Methods QualitativeFocus groupsKey informant interviewsDirect observationQuantitativeChart or record reviewsActivity logsDemographicsQualitative and QuantitativeIn-person interviews (clients, staff, management)Measurement issues:Perhaps there is no more critical issue in evaluation than defining measurement tools. The validity of the evaluation depends on appropriate measures of project activities and outcomes. Indeed, the final judgment of the program may depend upon how the program operations are conceptualized and measured. The choice of measure and design of the evaluation will to a large degree, determine if the evaluation is to be useful and credible.
26Process Evaluation: Selection of Measures How meaningful is the measure?How feasible is the measure?How useful is the measure?Similar to the prioritizing of evaluation questions in step 6, you need to set priorities with regards to process measures. They must be meaningful and understandable – meaningful to stakeholders and focused on goals and objectives, priorities and dimensions of the program – and the findings not so obvious that the information is not that useful.
27Process Evaluation: Selection of Measures Questions involving qualityBudget process – e.g., review of minutesQuality of services – standard client recordsQuestions re: program operationsChecklist of intended programming/servicesQuestions on why a program was developedPlanning documents, meeting minutes, funding proposals, needs assessmentsBudget process – E.g., reviewing meeting minutes related to budget to determine how budget decisions are made.Quality of services – E.g., standard client records; also, program policy manuals, program logs, observations, media reports, legislative reports, interviews.Questions re: program operations – E.g., checklist of intended programming/services; also program policies and manuals, program logs, observations, client records, and interviews.
28Process Evaluation - Methods Program coverage or reachCompare actual participation to intended participationProvides information about the level of program participation by the target groupRequires clear definition of the intended target group, monitoring of key target group characteristics, and rate of participationIn evaluating program coverage or reach, you are comparing actual participation to intended participation in the program. Coverage analysis provides feedback about participation in the program and can detect biases by examining differences in participation by subgroups of the target population (such as voluntary self-selection, intentional “creaming” of easier-to-serve clients by staff, and expansion of the target population to include those not intended for the program). Measures you would use, in this case, would be program demographics and participation records.
29Process Evaluation – Methods to Evaluate Coverage Develop clear definition of target populationIdentify key characteristics of target population for monitoring purposesCollect data on identified characteristicsAnalyze data to determine if actual population served is the intended populationAnalyze data to determine if persons served by program meet eligibility criteriaUsing characteristics data, determine which subgroups are over/under representedAnalyze characteristics of program drop-outsKey characteristics will vary by program; they could include age, gender, ethnicity, geographic location, place of residence, etc. Also, identify any specific characteristics that may affect ability to attend program (e.g., transportation issues, child care issues, distance from program site, problem with hours of operation, etc.)
30Process Evaluation: Methods to Evaluate Delivery Program records:Specify evaluation questions re: program deliveryIdentify sources of data to answer each questionAssess adequacy of current sources of dataDevelop a data collection plan – sources of data, who will collect data, etc.Develop an analysis plan (e.g., tables, graphs, figures)Program records – these typically include clients files and administrative data that provide a valuable source of information regarding client needs and characteristics, program resources and delivery, and program outputs. But in order to make good use of these documents, an agency has to have a good mastery of target group needs, program theory, and program operations.Sources of data typically include client intake/referral forms, assessments, service activity forms (type, duration, intensity, units of service), case notes, records of goal attainment and client progress, financial records, service statistics reports, quality assurance reports, quarterly and annual reports.One way of capturing intended program activities and outputs is by constructing service delivery pathways. These are flow diagrams that map the key program activities needed to achieve intended outcomes. This involves:Developing a logic modelStudying current operationsConducting a literature review re: best practices of similar programsDeveloping a service delivery pathways diagram identifying program milestones that need to be achieved to reach each outcome target. (When used for monitoring program delivery, they can help to identify problems and ensure achievement of intended outcomes, service targets, and quality of services).
31Process Evaluation: Methods Client satisfaction or perception of careClient surveys can assess overall satisfaction with services as well as information regarding content of program and manner in which it was deliveredEstablished measures:Client Satisfaction Questionnaire (CSQ-8)Youth Services Survey for FamiliesConsumer-Oriented Mental Health SurveyKansas Family Satisfaction Survey
32Process Evaluation: Selection of Measures Standardized measures:Example: The National Inventory of Mental Health Quality Measures (developed by the Center for Quality Assessment and Improvement in Mental Health (CQAIMH):Future launch of the Centre’s Measures DatabaseThe Centre is also developing a Measures database, which will be accessible from the Centre’s website. We hope to launch this within the next few months. This is a database of the measures that are being used in various programs here in Ontario, with a review of the psychometric properties and how to get copies of the measure.
33Process Evaluation – Tips for Data Collection Create a schedule for collecting process data and stick to itDivide tasks & assign individuals to each taskMaximize existing opportunities for data collectionOnly identify questions of interest/value and keep the evaluation focusedPeriodically review your instruments to ensure evaluation approach is still usefulIn a process evaluation, it is often difficult to decide what information is truly key to describing program operations and what information is simply extraneous detail. In selecting relevant data and posing questions about program operations, the evaluation team needs to refer carefully to the logic model prepared at the start of the project. Keep in mind that it is not only permissible but also important in process evaluation to revise the original logic model in light of findings during the evaluation.Analysis of qualitative data requires considerable substantive knowledge on the part of the evaluator. The person doing the analysis needs to be familiar with similar projects, respondents, and responses, and the context in which the project is operating. He or she will also need to be able to understand the project's historical and political context as well as the organizational setting and culture in which services are delivered. At the same time, the challenge will be for that person to maintain some objectivity in order to be able to make an unbiased assessment of whether responses support or refute hypotheses about the way the project works and the effects it has.Collecting qualitative data also requires skills in interviewing and observing; reading all about how to conduct an interview or a focus group in advance is important in order to understand the intent of each question, the possible variety of answers that respondents might give, and ways to probe to ensure that full information about the issues under investigation is obtained. Data must be carefully recorded or taped. Notes on contextual factors and interim hypotheses need to be recorded as soon as possible after data collection. There are many sophisticated data measurement applications for tracking process related data, but for evaluation purposes, Microsoft Excel may be all that is needed.
34Next Steps March 27: webinar on writing the final report March 31: Award term endsApril 30: Final report dueMay 1 to 14: Evaluation readiness assessment for the Evaluation Implementation Grants (EIG)June 2: Deadline for EIG grant applicationsJune webinar: The road to data collection
35For more information Susan Kasprzak firstname.lastname@example.org / Ext. 3320Tanya Witteveen/ Ext. 3483