Program Evaluation

Overview and Discussion of: Objectives of evaluation Process evaluation Outcome evaluation Indicators & Measures Small group discussions

Program Evaluation: Here we go again! WHY WHY WHY do evaluation? Time Resources Validity and reliability Action

Why Evaluate? Increase knowledge & understanding of the program being evaluated Identify specific problems/needs, as well as effective practices Develop capacity for ongoing assessment and improvement Evaluation is of little value unless it leads to further decisions and activities re: program improvement

Why Evaluate? In the past… Most often, the focus of evaluation was on what agencies did – how many clients did they see, how many clinical hours were spent, how many sessions did they deliver Now… There is an increasing focus on what happens when agencies deliver services – what changes, what improves for the client

Thorny Issues in Program Evaluation ?

Thorny Issues in Program Evaluation Cost and human resources Buy-in at all levels Logic models that defy logic (overly ambitious) Process versus outcomes/impacts Choosing the right indicators/measures Moving data to action

Capacity to do evaluation Capacity to use evaluation Human Resources Staffing Technical Skills Communication Skills Professional Development Leadership Organizational Resources Budget Ongoing Data Collection Infrastructure Evaluation Literacy Results-Management Orientation Involvement-Participation Organizational Decision- Making Management Processes Decision Support Learning Benefits Instrumental-Conceptual Use Process Use { { Evaluation Planning and Activities Evaluation Plan Use of Consultants Information Sharing Organizational Linkages External Supports Framework of Evaluation Capacity, Isabelle Bourgeois. Do not cite without authors permission.

Ensuring support & buy-in Efforts to ensure support & buy-in for evaluation from staff & stakeholders should happen from the onset of deciding to conduct an evaluation Hold preliminary meetings Clarify roles & responsibilities over the evaluation Identify why the evaluation is being done; clarify any misconceptions; provide reassurance Identify who the end-user will be Develop agreement on what is to be evaluated Assess readiness for evaluation

Logic Models Keep it as simple and as focused as possible!

Process Evaluations Key Issues: Service Delivery – the extent to which program components are being delivered as planned Coverage – extent of target population participation in program Bias – extent to which subgroups of the designated target population participate (or dont participate)

Process Evaluations – Strengths & Weaknesses Strengths: Measures how well program meets client needs Measures the extent to which the program was implemented as intended Ensures a consumer-driven program Easy & inexpensive Weaknesses: Bias can be significant oFew dissatisfied participants return surveys oClients may report inflated levels of satisfaction oVery subjective

Outcome Evaluations Focus: program results, program effectiveness Examples of Outcomes: Change in circumstances Change in understanding/knowledge Change in attitude Change in behaviour

What Are Some of Your Outcomes? How are you measuring them? ?

Indicators and Measures – Measurement Strategy OutcomeIndicator(s)Source of data (records, clients, caregivers, etc.) Method to Collect Data & Frequency Who collects data When collects data EXAMPLE: Skill acquisition – self management, adaptive Increase in scores on administered psychometric test, caregiver survey Caregivers, Psychometric measures Pre and post administered tests, surveys Evaluation Researcher At the beginning and end of program, and at six months post program

Moving Results Into Action Use results Develop action plans for improving program effectiveness Develop plans to disseminate and share findings – be innovative! Plans should be developed from the beginning and continue throughout Linkages & partnerships developed in the initial stages of the evaluation form the basis of plans for information sharing

Centre of Excellence Activities Consultation services –Development of evaluation framework, logic model, indicators and measures –Training or workshops –Facilitating or developing networks/ communities of practice (e.g., Triple P) –Review of grant applications Evaluation grants

Objectives – Evaluation Grants Build skills –Developing evaluation framework and logic model –Implementing evaluation plan and analyzing results –Using evaluation findings Build support –Frequent communications between the Centre and agencies –Engage internal and external stakeholders Build communities –Establishing communities of practice

Communities of Practice – Triple P Nine agencies – key contacts for Triple P within Ontario Developing a common approach to evaluation – measuring same set of outcomes with common measures Facilitating a provincial roll-out of Triple P data

Communities of Practice – Triple P What made it work? Clearly identified purpose Willingness to work together Face-to-face meeting at strategic points Communication portal to facilitate active learning and sharing of ideas Centre as facilitator

Small Group Discussion 1.Think of a program in your agency. List all the external and internal stakeholders for the evaluation. What are some challenges in meeting the interests of all stakeholders in your program? What are some ways your program has been successful in getting buy-in from key stakeholders?

Small Group Discussion 2. In your local area or region, what community of practice would you like to be involved in? What topics or issues would you like the community of practice to discuss and meet about? How can the Centre of Excellence help you in forming and maintaining this community of practice?

Program Evaluation Contact Information Contact Information: Susan Kasprzak Research Associate (613) ext Tanya Witteveen Research Associate (613) ext

Visit our website for more information: