1. Are we there yet? Evaluating your graduation SiMR

2. 2

3. Everyone has a plan, but… Your plan and everything it specifies, including evaluation steps, should fit your state’s context—its needs, strengths and goals. Identify the steps, processes and supports needed to implement the practices or programs that will support your goals. Design and incorporate evaluation pieces to support formative and summative evaluation of the parts of your plan. 3

4. What will you evaluate? The quality or fidelity of implementation of the programs / interventions in your SSIP AND The results or impact of those programs / interventions being implemented to improve student outcomes 4

5. Fidelity of implementation Look at the process Look for evidence of progress… Did the activities specified in the intervention occur? – How do you know? Look at the evidence… the outputs: a product or participation in an activity 5

6. Results or impact Look for evidence of progress… Were the intended outcomes of each activity achieved? – How do you know? Look for a change in knowledge, skills, behavior, student outcomes, etc. 6

7. General reaction to evaluation 7

8. Assessing results: start with the ending 1.What do you want to change? 2.How will you know if it happened? 3.What data do you need to answer your questions? 4.How will you gather this information? 8

9. 1. What do you want to change? Identify your specific intended outcomes and some milestones along the path to each.  Short-term outcomes  Medium-term outcomes  Long-term outcomes 9

10. Outcomes should be: Developed with representative stakeholder input Measurable without too much drama / trauma Realistic and achievable Clear, specific and distinct Logically linked to your program activities 10

11. 2. How will you know if it happened? Review each outcome and think about how to determine whether it has been attained. What are the indicators of progress for each outcome? Write a few evaluation questions for each outcome. Make them clear and specific. Consider the level of evaluation you want… 11

12. 12 NSTTAC’s Six Critical Levels of Evaluation Evaluation LevelQuestions to addressData sourcesIndicators 1. Quality, Usefulness, and Relevance Were participants satisfied? Were info & materials useful? Were info & materials relevant? Survey or questionnaire Participants indicate satisfaction 2. Participant Learning Outcomes Did participants meet outcomes? Did participants gain knowledge? Did participants gain skills needed? Pre/post test Demonstration Reflection Participants acquired knowledge / skills 3. Organizational Policies, Procedures, and Support Were barriers addressed efficiently? Was implementation advocated? Was the organization impacted? Policies & docs Surveys Interviews Org advocacy, support, facilitation 4. Program Implementation Did participants implement the new knowledge & skills effectively? What is degree of implementation? Observation Surveys Interviews Degree / quality of implementation 5. In-school and Post- school Outcomes What was impact on kids? Are grad rates increasing? Are PSO improving? School records Structured interviews Improved student outcomes (in- and post-school) 6. Evaluation Use and Dissemination Were results disseminated clearly & effectively to the stakeholders? Were the evaluation results used in making decisions? Surveys Organizational records Data-based decision making Data sharing

13. 3. What data do you need to answer your questions? Identify the types and sources of data that could inform each of the evaluation questions. Consider when and how often you will need to collect the data for each indicator. Minimize the burden and maximize the yield! 13

14. 4. How will you gather this information? Identify the methods and tools you will use to collect the data (e.g., NSTTAC planning matrix, NDPC-SD data tools, the Risk Calculator, SWIS, the District or School Attendance Tracking Tools (DATTs or SATTs), the School Climate Survey, interviews, focus groups, etc.). 14

15. Other considerations What resources are available to support the evaluation (buy-in, $$, people, time)? Who will do each portion of the work? Format(s) of the data (quantitative, qualitative) How and where data will be stored and analyzed? How sensitive the tools are to error, how reliable, how valid for your purpose? How the results of evaluation will be shared with stakeholders? 15

16. Gratuitous cat picture 16

17. References/Useful Sites NTACT website: www.transitionta.orgwww.transitionta.org NSTTAC website - The Evaluation Toolkit: www.nsttac.org/content/evaluation-toolkit-third-edition www.nsttac.org/content/evaluation-toolkit-third-edition Coalition for Community Schools evaluation toolkit: www.communityschools.org/assets/1/AssetManager/Evaluation_Toolkit_ March2010.pdf www.communityschools.org/assets/1/AssetManager/Evaluation_Toolkit_ March2010.pdf IES pub on making evaluation comprehensible: ies.ed.gov/ncser/pubs/20133000/pdf/20133000.pdf ies.ed.gov/ncser/pubs/20133000/pdf/20133000.pdf IES website: www.ies.ed.gov/funding/resources.aspwww.ies.ed.gov/funding/resources.asp 17

18. Contact Matthew Klare, Ph.D. NTACT mklare@uncc.edu 18