1. Basic Program Evaluation NTSC Training Materials

2. Purpose/Objectives Increase your knowledge of processes involved in program evaluation Provide information and resources to help you design and conduct your own program evaluation

3. Program Evaluation Training This training presentation is in 16 modules, encompassing 9 steps in the evaluation process

4. Program Evaluation Training – Modules Module 1 – Introduction Module 2 – Overview Module 3 – Defining the Purpose Module 4 – Specifying the Questions Module 5 – Identifying Evidence Needed Module 6 – Specifying the Design

5. Program Evaluation Training – Modules Module 7 – Data Collection Plan Module 8 – How to Collect Data Module 9 – Using Commercial Instruments Module 10 – Using Self-Constructed Instruments Module 11 – Collecting Data

6. Program Evaluation Training – Modules Module 12 – Analyzing Data Module 13 – Drawing Conclusions and Documenting Findings Module 14 – Disseminating Information Module 15 – Feedback for Program Improvement Module 16 – Conclusion

7. Program Evaluation Training – Modules Module 1 – Introduction Module 2 – Overview Module 3 – Defining the Purpose Module 4 – Specifying the Questions Module 5 – Identifying Evidence Needed Module 6 – Specifying the Design

8. Module 1 – Introduction Why evaluate? What is evaluation? What does evaluation do? Kinds of evaluation

9. Why Evaluate? Determine program outcomes Identify program strengths Identify and improve weaknesses Justify use of resources Increased emphasis on accountability Professional responsibility to show effectiveness of program

10. What is Program Evaluation? Purposeful, systematic, and careful collection and analysis of information used for the purpose of documenting the effectiveness and impact of programs, establishing accountability, and identifying areas needing change and improvement

11. What Evaluation Does Looks at the results of your investment of time, expertise, and energy, and compares those results with what you said you wanted to achieve

12. Kinds of Evaluation Outcome Implementation Formative Summative

13. Outcome Evaluation What: Identifies the results or effects of a program When: You want to measure students’ or clients’ knowledge, attitudes, and behaviors as a result of a program Examples: Did program increase achievement, reduce truancy, create better decision-making?

14. Implementation Evaluation What: Documents what the program is and to what extent it has been implemented When: A new program is being introduced; identifies and defines the program; identifies what you are actually evaluating Examples: Who receives program, where is program operating; is it being implemented the same way at each site?

15. Timing of Evaluation Formative – as the program is happening to make changes as program is being implemented Summative – at the end of a program to document results

16. Module 2 – Overview Module 1 – Introduction Module 2 – Overview Module 3 – Defining the Purpose Module 4 – Specifying the Questions Module 5 – Identifying Evidence Needed Module 6 – Specifying the Design

17. Planning Development Implementation Feedback Overview – The 9- step Process

21. Module 3 – Defining the Purpose Module 1 – Introduction Module 2 – Overview Module 3 – Defining the Purpose Module 4 – Specifying the Questions Module 5 – Identifying Evidence Needed Module 6 – Specifying the Design

22. 9-step Evaluation Process Step 1: Define Purpose and Scope

23. Step 1: Scope/Purpose of Evaluation Why are you doing the evaluation? – mandatory? program outcomes? program improvement? What is the scope? How large will the effort be? – large/small; broad/narrow How complex is the proposed evaluation? – many variables, many questions? What can you realistically accomplish?

24. Resource Considerations Resources –$$ –Staff who can assist? need to bring in expertise? do it yourself? advisory team? –Time Set priorities How you will use the information

25. Module 4 – Specifying the Questions Module 1 – Introduction Module 2 – Overview Module 3 – Defining the Purpose Module 4 – Specifying the Questions Module 5 – Identifying Evidence Needed Module 6 – Specifying the Design

26. 9-step Evaluation Process Step 2: Specify Evaluation Questions

27. Evaluation Questions What is it that you want to know about your program? operationalize it (make it measurable) Do not move forward if you cannot answer this question.

28. Sources of Questions Strategic plans Mission statements Policies Needs assessment Goals and objectives National standards and guidelines

29. Broad Questions Broad Scope – Do our students contribute positively to society after graduation? – Do students in our new mentoring program have a more positive self-concept and better decision-making skills than students without access to the mentoring program? – To what extent does the state’s career development program contribute to student readiness for further education and training and success in the workforce?

30. Narrow Questions Narrow Scope – Can our 6th grade students identify appropriate and inappropriate social behaviors? – How many of our 10th grade students have identified their work-related interests using an interest inventory? – Have 100% of our 10th grade students identified at least 3 occupations to explore further based on their interests, abilities, and knowledge of education and training requirements?

31. Exercise 1 – Scope (p. 2 of Workbook) From the list of questions, identify those that might be considered broad and those that might be considered narrow How large will the resources need to be to answer the question

32. Exercise 2 – Scope Write (p. 3 of Workbook) List one broad evaluation question and one narrow evaluation question

33. Module 5 – Identifying Evidence Needed Module 1 – Introduction Module 2 – Overview Module 3 – Defining the Purpose Module 4 – Specifying the Questions Module 5 – Identifying Evidence Needed Module 6 – Specifying the Design

34. Identifying Evidence Needed to Answer Your Questions What evidence do you have to answer your question?

35. Identifying Evidence Needed to Answer Your Questions Need to think about what information you need in order to answer your evaluation questions

36. Example Evidence – Broad Scope Do our students contribute positively to society after graduation? Percent of our students that are employed, in education or training programs, in the military, are supporting a family by working at home, and/or are volunteering for charitable causes 3 years after high school graduation Percent of our students that vote in local and national elections 5 years after graduation

37. Example Evidence – Narrow Scope Have 100% of our 10th grade students identified at least 3 occupations to further explore that are based on their interests, abilities, and knowledge of the education and training requirements? Number of 11th and 12th grade students participating in the career class that demonstrated increased career maturity from a pre- and post-test

38. Exercise 3 – Evidence (p. 4 of Workbook) List evidence you need to have to answer the question

39. Module 6 – Specifying the Design Module 1 – Introduction Module 2 – Overview Module 3 – Defining the Purpose Module 4 – Specifying the Questions Module 5 – Identifying Evidence Needed Module 6 – Specifying the Design

40. 9-step Evaluation Process Step 3: Specify Evaluation Design

41. Types of Designs Relates to when data should be collected Status (here and now; snapshot) Comparison (group A vs. group B; program A vs. program B) Change (what happened as a result of a program; what differences are there between time A and time B) Longitudinal (what happens over extended time)

42. Exercise 4 – Design (p. 5 of Workbook) What type of design fits each evaluation question? – Status – Comparison – Change – Longitudinal

43. Module 7 – Data Collection Plan Module 8 – How to Collect Data Module 9 – Using Commercial Instruments Module 10 – Using Self-Constructed Instruments Module 11 – Collecting Data

44. 9-step Evaluation Process Step 4: Create a Data Collection Action Plan

45. Organize Your Evaluation With a Data Collection Action Plan Evaluation Question 1 What is Collected How Collected/What Technique From Whom/ Data Sources When Collected and By Whom How Data are to be Analyzed

46. Components of a Data Collection Action Plan What Will be Collected? –based on evidence required How Collected? Instrumentation –surveys? published instrument? focus group? observations?

47. Components of a Data Collection Action Plan From Whom Collected? –who or what provides evidence When Collected and by Whom? –specific dates, times, persons How Data are to be Analyzed?

48. Data Sources — Who and What Students Parents Teachers Counselors Employers Friends Documents and other records

49. Exercise 5 – Data Sources (p. 6 of Workbook) Who/what are the data sources for the following questions?

50. Module 8 – How to Collect Data Module 7 – Data Collection Plan Module 8 – How to Collect Data Module 9 – Using Commercial Instruments Module 10 – Using Self-Constructed Instruments Module 11 – Collecting Data

51. Data Collection Options Commercial instrument Survey/questionnaire Focus group/interviews Observations Archived information

52. Module 9 – Using Commercial Instruments Module 7 – Data Collection Plan Module 8 – How to Collect Data Module 9 – Using Commercial Instruments Module 10 – Using Self-Constructed Instruments Module 11 – Collecting Data

53. Commercial Instruments Sometimes best to use published or research instruments – particularly for tough constructs – since its not made specifically for you, may not answer your question entirely

54. Sources of Information on Instruments Counselors Guide to Career Assessment Instruments Relevance, the Missing Link ─ A Guide for Promoting Student Success Through Career Development Education, Training, and Counseling The Buros Institute ETS Test Collection The Association for Assessment in Counseling and Education

55. Exercise 6: Decision- Making Checklist (p. 7 of Workbook) This checklist will help you conduct a review of data collection instruments that you are considering using in your evaluation

56. Module 10 – Using Self- Constructed Instruments Module 7 – Data Collection Plan Module 8 – How to Collect Data Module 9 – Using Commercial Instruments Module 10 – Using Self-Constructed Instruments Module 11 – Collecting Data

57. Self-Constructed Instruments: Questionnaires Focus on evidence you need Use simple language Ask only what you need; keep it short Don’t use jargon Each question should focus on one idea Make sure terms are clear Make it easy for person to answer the questions (check rather than write, where possible) Use extended response when you want details

58. Types of Scales (1) Specific (yes, no; number; gender) Extended (1-3, 1-5, 1-7)

59. Types of Scales (2)

60. Anchored Scales

61. Scales for Younger Students

62. Self-Constructed Instruments: Focus Groups/Interviews Good to use when you want extended and detailed responses Craft an agenda and stick to it Keep groups small (6-10); time short (1- 1.5 hours) Specify objectives of session Questions need to be clear; one question at a time Encourage everyone to participate Use opportunity to probe deeper on a topic

63. Observations and Observational Checklist “You can observe a lot just by watching.” -- Yogi Berra Go to pages 8 and 9 of your Workbook and review an example of an observational checklist

64. Archives and Documents Examine What’s Already Available Examples – Attendance records – Truancy reports – Grades – Bullying incidents – Report cards – Portfolios – Discipline referrals – Public service hours – Police reports

65. Exercise 7 - Data Collection Action Plan Review examples of a completed Data Collection Action Plan on Pages 10-12 of the Workbook

66. Module 11 – Collecting Data Module 7 – Data Collection Plan Module 8 – How to Collect Data Module 9 – Using Commercial Instruments Module 10 – Using Self-Constructed Instruments Module 11 – Collecting Data

67. 9-step Evaluation Process Step 5: Collect Data

68. How Much Data Should You Collect? How much data do you need? – 100% of target audience is ideal; may be too expensive and time consuming – If not 100%, sample is OK if group is representative of group as a whole (population)

69. Types of Samples

70. Data Collection Considerations When should you collect the information? Who should collect it?

71. Module 12 – Analyzing Data Module 13 – Drawing Conclusions and Documenting Findings Module 14 – Disseminating Information Module 15 – Feedback for Program Improvement Module 16 – Conclusion

72. 9-step Evaluation Process Step 6: Analyze Data

73. What is Data Analysis? Data collected during program evaluation are compiled and analyzed (counting; number crunching) Inferences are drawn as to why some results occurred and others did not Can be very complex depending on your evaluation questions We will focus on simple things that can be done without expert consultants

74. Types of Data Analysis Simple Frequency Counts

75. Types of Data Analysis Sort by Relevant Categories

76. Types of Data Analysis Calculate Percentages – Exercise 8 (p.13 of Workbook)

77. Types of Data Analysis Showing Change or Differences

78. Types of Data Analysis Reaching an Objective or Goal

79. Types of Data Analysis Observing Trends

80. Types of Data Analysis Graph Results

81. Types of Data Analysis Calculate Averages – Exercise 9 (p. 14 of Workbook)

82. Types of Data Analysis Calculate Weighted Averages

84. Types of Data Analysis Rank Order Weighted Averages

85. Types of Data Analysis Graph Weighted Averages

86. Using Focus Group/Interview Information Qualitative findings from focus groups, extended response items, etc., should be analyzed in a different way – Code words/frequency – Identify themes – Pull quotes – Summarize and draw conclusions

87. Module 13 – Drawing Conclusions and Documenting Findings Module 12 – Analyzing Data Module 13 – Drawing Conclusions and Documenting Findings Module 14 – Disseminating Information Module 15 – Feedback for Program Improvement Module 16 – Conclusion

88. 9-step Evaluation Process Step 7: Drawing Conclusions and Documenting Findings

89. Drawing Conclusions Examine results carefully and objectively Draw conclusions based on your data What do the results signify about your program?

90. Exercise 10 – Interpreting Results (p. 15-16 of Workbook) Complete the Interpreting Results Exercise on pages 15-16 of the Workbook.

91. Unintended Consequences Watch for positive and negative outcomes that you did not plan on - For example, if your career development program focuses on increasing students’ awareness of how to identify their interests and skills, it may have the unintended consequence of leaving little time for students to explore occupations and jobs in their area. - Or, if your program has overemphasized the importance of getting a college education, students may not be considering the positive benefits of other kinds of postsecondary training.

92. What to Include in Your Documentation Program description Evaluation questions Methodology (how and from whom and when) Response rate Methods of analysis Conclusions listed by evaluation question General conclusions and findings Action items Recommendations for program improvement and change

93. Document the Successes and Shortfalls Highlight and brag about positive outcomes Document shortfalls - Provides opportunities to - improve program - make recommendations to benefit the program

94. Module 14 – Disseminating Information Module 12 – Analyzing Data Module 13 – Drawing Conclusions and Documenting Findings Module 14 – Disseminating Information Module 15 – Feedback for Program Improvement Module 16 – Conclusion

95. 9-step Evaluation Process Step 8: Disseminate Information

96. Determining Dissemination Methods Inform all your relevant stakeholders on results Dissemination methods should differ by your target audience

97. Potential Audiences Your program staff Businesses Partners that work with your program Employers School Level School administrators Counselors Teachers Students Parents

98. Potential Audiences Media Local newspaper TV station Radio program Community or school newsletter Education Researchers Members of Community or faith based organizations Church members Religious leaders Rotary club Boys or girls club Anyone who participated in your evaluation!

99. Dissemination Techniques Reports Journal articles Conferences Career Newsletter/Tabloids Presentations Brochures TV and newspaper interviews Executive summary Posting on Web site

100. Exercise 11 – Disseminating Information (p. 17 of Workbook) Using the information provided in exercise 6, describe how you would disseminate the information to – Program funders – Parents

101. Module 15 – Feedback for Program Improvement Module 12 – Analyzing Data Module 13 – Drawing Conclusions and Documenting Findings Module 14 – Disseminating Information Module 15 – Feedback for Program Improvement Module 16 – Conclusion

102. 9-step Evaluation Process Step 9: Feedback to Program Improvement

103. Opportunities to Fix Shortfalls Evaluation results may show areas where improvement is necessary - 25% of 11th graders are unable to complete a skills based resume - 85% of our students drop out of college in the first year - Most employers do not want your students to serve an interns in their companies

104. Feedback to Program Improvement You can use evaluation findings to make program improvements – Consider adjustments – Re-examine/revise program strategies – Change programs or methodologies – Increase time with the program Use your results as a needs assessment for future efforts

105. Module 16 - Conclusion Module 12 – Analyzing Data Module 13 – Drawing Conclusions and Documenting Findings Module 14 – Disseminating Information Module 15 – Feedback for Program Improvement Module 16 – Conclusion

106. Conclusion Evaluation helps you: determine the effects of the program on recipients know if you have reached your objectives improve your program

107. Conclusion The 9-step process works A credible evaluation can be done with careful planning and some basic math skills

108. Exercise 12 – Developing a Data Collection Action Plan (page 18 of Workbook) Using all the information you have gathered from the workbook exercises and the powerpoint slides, you can develop your own Data Collection Action Plan on page 18 of your Workbook