1. 1 Outcomes and Program Improvement: Designing Effective Evaluations Presented by Christine A. Ameen, Ed.D. Ameen Consulting & Associates (ameenca@ameenconsulting.com)

2. ameenconsulting & associates 2 Reflection “If you don’t know where you’re going, you’ll end up somewhere…” - Yogi Berra

3. ameenconsulting & associates 3 Some “Reasons” for Doing Outcome Evaluation... I need to be able to fill out these forms that the state department gave me. I need a plan for a grant I’m writing. I need to get my agency accredited. I need to show outcomes for the grant project we just completed.

4. ameenconsulting & associates 4 Other Reasons for Doing Outcome Evaluation... Our staff needs to know the impact they make Our agency needs to know the impact it makes Our funders need to see the return on their investment Our accrediting body needs to see evidence of our intent to provide quality services

5. ameenconsulting & associates 5 The #1 Reason for Doing Outcome Evaluation... Our clients deserve the very best service!

6. 6 How effective is your program? Let me tell you about the kids that are in my group!

7. 7 How effective is your program? Here’s what’s working… Here’s what we’re learning… Here’s how we’ve increased effectiveness…

8. ameenconsulting & associates 8 What is Continuous Program Enhancement (CPE)? A commitment to ongoing assessment of service delivery and client outcomes to achieve the best outcomes possible.

9. ameenconsulting & associates 9 CPE assumes that: Every program has strengths and weaknesses, and opportunities for improvement The organization values learning Administrators nurture a climate of trust and welcome feedback Staff want clients/customers to be successful Staff are open to learning new and different ways of providing service

10. ameenconsulting & associates 10 Determining Purpose Assess the impact of a program Improve program implementation Identify unmet client needs Respond to accountability. CPE may be used to:

11. ameenconsulting & associates 11 LET’S FIND OUT! ARE YOU READY?

12. ameenconsulting & associates 12 Its purpose is clearly understood by everyone The organization’s environment is a risk-free learning environment where a climate of trust is nurtured and feedback is welcomed Internal collaboration is encouraged and expected. Adapted from Brinkerhoff, R.O.; Brethower, D.M.; Hluctyl, T.; and Nowakowski, J.R. Program Evaluation: A Practitioner’s Guide for Trainers and Educators. Boston Kluwer-Nijhoff, 1983. You’re ready if:

13. ameenconsulting & associates 13 Results will be shared with those who have a right to know The results will be useful and used The process of doing CPE is humane The benefits of the CPE justify the cost. Adapted from Brinkerhoff, R.O.; Brethower, D.M.; Hluctyl, T.; and Nowakowski, J.R. Program Evaluation: A Practitioner’s Guide for Trainers and Educators. Boston Kluwer-Nijhoff, 1983. You’re ready if:

14. ameenconsulting & associates 14 Don’t Ask a Question for Which You Don’t Want the Answer! Respond to the results Be prepared to learn that perceptions don’t always reflect reality Accept the fact that CPE is an innovation and there will be some natural resistance to it Be patient — CPE will take some time before its payoffs are realized You must get administrative buy-in first!

15. ameenconsulting & associates 15 Planning for CPE 1.A program model which describes the services for delivering services to clients and the outcomes clients should achieve 2.An objectives model which defines objectives for the services and objectives for the outcomes 3.An evaluation model which defines how data will be collected and used to monitor the attainment of process objectives and outcome objectives.

16. ameenconsulting & associates 16 The Processes Used to Develop the CPE Models 1.Review contracts, licensing agreements, program proposals for service requirements, information requirements and pre-determined outcomes 2.Assess what information needs or concerns board members, administrators, staff, clients, customers, volunteers, policy-makers and public-at-large may have about the program 3.Convene a work group of staff and administrators to develop the CPE models and support implementation

17. 17 Building Your Program Model Defining Program Services and Client Outcomes

18. ameenconsulting & associates 18 Defining Your Program: The Program Model

19. ameenconsulting & associates 19 Some Definitions Some Definitions Client Conditions - demographic characteristics and other factors that describe client need, assets and client risk for not being successful in program Program Services - the services delivered to address client needs and risks Client Outcomes - a desired effect or impact of a service, intervention or experience on a client

20. ameenconsulting & associates 20 Residential Program Model Residential Program Model

21. ameenconsulting & associates 21 Residential Program Model Residential Program Model

22. ameenconsulting & associates 22 Residential Program Model Residential Program Model

23. ameenconsulting & associates 23 A Conceptual Framework for Behavior Change 1.Awareness: the initial consciousness, perception, or sense of a concept 2.Knowledge/Skills: understanding, comprehension of a concept; demonstration of ability of that understanding or comprehension 3.Behavior: performance or conduct in a specified way 4.Modeling behavior: demonstration of specified behavior to others, most often to teach the behavior to others.

24. ameenconsulting & associates 24 How Far Does Your Program Go? How long are clients in the program? How intense is the program? How extensive is content covered in the program?

25. 25 LET’S TRY IT! Build a Program Model

26. ameenconsulting & associates 26 Defining Your Program: The Program Model

27. 27 Building Your Objectives Model Defining Objectives for Services and Outcomes

28. ameenconsulting & associates 28 Your Program Objectives Model

29. ameenconsulting & associates 29 Some Definitions Service Objective - a specific statement about how the service is to be delivered, usually including a timeframe and benchmark Outcome Objective - a specific statement about how much desired effect or impact of a service is going to be achieved.

30. ameenconsulting & associates 30 Establishing Objectives for Program Services What clients? What services will be provided or experienced? During what time frame?

31. ameenconsulting & associates 31 Examples of Service Objectives l100% of clients served will have written treatment plans within 30 days of intake l85% of clients served will have weekly visits with their families throughout their treatment stay l85% of clients will attend school 90% of the time throughout their treatment stay l100% of clients will attend all individual therapy sessions designed in their treatment plans throughout their treatment stay

32. ameenconsulting & associates 32 Establishing Objectives for Client Outcomes Desired effect or impact On whom or what The specific outcome By how much, when.

33. ameenconsulting & associates 33 Writing Client Outcome Objectives

34. ameenconsulting & associates 34 Examples of Client Outcomes lNo youth served will experience recurrence of abuse/neglect while in care. l75% of youth will increase their academic level of functioning by at least one grade level upon release from the program. lNo more than 15% of youth completing treatment will be arrested within six months of release from the program

35. ameenconsulting & associates 35 Residential Program Objectives Model Residential Program Objectives Model

36. ameenconsulting & associates 36 Residential Program Objectives Model Residential Program Objectives Model

37. ameenconsulting & associates 37 Residential Program Objectives Model Residential Program Objectives Model

38. 38 LET’S TRY IT! Build an Objectives Model

39. ameenconsulting & associates 39 Your Program Objectives Model

40. 40 Building Your Evaluation Plan Defining Data Collection to Monitor the Attainment of Service and Outcome Objectives

41. ameenconsulting & associates 41 Your Program Evaluation Model

42. ameenconsulting & associates 42 Data Collection Plan for the Objectives Create a data collection plan that includes: Information source Measurement method Timeline for when collected Identification of parties responsible for data

43. ameenconsulting & associates 43 Some Issues to Resolve Related to Service Objectives Data Service objectives usually report the status of service delivery or the status of the client at a specific point in time Attendance versus participation Opportunity to participate versus actual participation Using currently existing data, e.g., rating forms, attendance records, incident reports, etc.

44. ameenconsulting & associates 44 Some Common Methods for Collecting Outcome Objectives Data Formal testing or assessment, e.g., vocational skills Attitude measurement, e.g., empathy Surveys, e.g., therapeutic environment Interviews, e.g., follow up with the family upon the youth returning home

45. ameenconsulting & associates 45 Data Collection for Outcomes Use both qualitative and quantitative information : Qualitative Used when the program is aimed at individualized outcomes Quantitative Used when the program is aimed at common outcomes for all participants

46. ameenconsulting & associates 46 Some Issues to Resolve Related to Outcome Objectives Data Will service delivery result in the impact on the client? Will service delivery result in the client being able to maintain a positive impact after leaving the program? An increase or a decrease in something, e.g., empathy, requires the measure be used at the beginning and the end of the program

47. ameenconsulting & associates 47 Some Issues to Resolve Related to Outcome Objectives Data The achievement of a specific level of an outcome, e.g., the family will be able to name 3 community resources they can utilize, requires the measure be used at the end of the program only Will instruments such as surveys be read to clients? If read to some, read to all Instruments should be administered by staff with whom client have limited or no interaction, to reduce the potential, unintended bias of client responding based upon the relationship to the staff administering the instrument

48. ameenconsulting & associates 48 Some Tips for Selecting Measures Does the instrument directly measure the target behavior or issue the program is intended to change? Is the instrument able to measure change? Is the instrument appropriate for the population? Can the instrument be used in ways that are respectful of the clients and the program? Can the instrument be used consistently? What is the reading level?

49. ameenconsulting & associates 49 Data Utilization for the Objectives Create a data utilization plan that includes: Who will review the data How data will be interpreted Timeline for when reviewed How data will be used

50. ameenconsulting & associates 50 Some Tips for Writing the Data Utilization Plan Plan on including direct service staff in the review and utilization process Results will be interpreted against the benchmark you set for each objective Try to anticipate how the information will be used.

51. ameenconsulting & associates 51 Some Tips for Writing the Data Utilization Plan Review often enough to allow for changes to be made if the objectives aren’t met Consider reviewing data about service objectives on a monthly basis Consider reviewing data about outcome objectives on a quarterly basis Consider reviewing all of the data on an annual basis, to consider setting new benchmarks or new objectives

52. ameenconsulting & associates 52 Residential Program Evaluation Model Residential Program Evaluation Model

53. ameenconsulting & associates 53 Residential Program Evaluation Model Residential Program Evaluation Model

54. ameenconsulting & associates 54 Residential Program Evaluation Model Residential Program Evaluation Model

55. 55 LET’S TRY IT! Build an Evaluation Model

56. ameenconsulting & associates 56 Your Program Evaluation Model

57. ameenconsulting & associates 57 Where do I start? Rome wasn’t built in a day! Focus on 2-3 key services and 2-3 outcomes. Don’t reinvent the wheel! Use available resources, beg, borrow and “steal” Don’t go it alone! Get together with colleague agencies to learn from one another, pool resources Don’t wait for perfection! Measuring outcomes is an involving field

58. ameenconsulting & associates 58 Why Quality Improvement Efforts? We need to provide the highest quality services possible We need to learn what’s working and what isn’t We need to let our staff know how effective our services are We need to be accountable to our funders We need to make a difference

59. 59 Thank you for attending my session today! If you have any feedback for how I can improve my training style or content, please email me your suggestions! (ameenca@ameenconsulting.com)ameenca@ameenconsulting.com