1. From Program Evaluation to Program Improvement 21 st CCLC Training Meeting January 14, 2011 Presented by: Carrie Furrer & Dunya Minoo, Portland Public Schools Greg Belisle, Impact NW

2. Evaluation  Improvement Formative: Plan & think about your work Implementation: Understand service delivery & participation Summative: Find out if program is making a difference Improvement: Figure out where to make changes

3. Demystifying the ‘black box’ of program improvement Program Inputs Resources Planning Staff Partners 21 st CCLC Programming Program Outputs  Short-term Student Outcomes Long-term Student Outcomes

4. What’s in the Black Box? Theory of Change What is your theory about how your program works to improve children’s lives? –PROGRAM OUTPUTS Quantity: number of students served; hours served Quantity: number of students served; hours served Quality: positive staff-student interactions; prosocial climate; social support Quality: positive staff-student interactions; prosocial climate; social support Communication/collaboration: relationships with partners; collaboration with school staff Communication/collaboration: relationships with partners; collaboration with school staff –SHORT-TERM OUTCOMES: Student engagement in program, motivation to participate in activities, sense of belonging

5. Logic Model = Theory of Change Program Inputs Program Outputs Short-term Outcomes Long-term Outcomes Homework club Students served 4 hours/week Students served 4 hours/week Collaboration with school day teachers Collaboration with school day teachers Qualified staff Qualified staff Students motivated to complete homework Students motivated to complete homework Students turn in homework Students turn in homework Better understanding of class material Better understanding of class material Improved grades

6. Program Improvement: A Systematic Review of…. Program Inputs: Program Planning & Design Program Inputs: Program Planning & Design Program Outputs: Quality, Quantity, Communication/Collaboration, Other Program Outputs: Quality, Quantity, Communication/Collaboration, Other Short-term Outcomes: Students’ “baby steps” toward the long-term outcomes Short-term Outcomes: Students’ “baby steps” toward the long-term outcomes

7. Resources Bagby, J. H. (Ed.). (2008). A resource guide for planning and operating afterschool programs (3rd ed.). Austin, TX: SEDL. www.sedl.org/pubs/fam95/.www.sedl.org/pubs/fam95/ Chapin Hall at the University of Chicago. www.chapinhall.org/research/areas/Youth- Development-and-Afterschool-Initiatives.www.chapinhall.org/research/areas/Youth- Development-and-Afterschool-Initiatives Eccles, J., & Gootman, J. A. (Eds.). (2002). Community programs to promote youth development. Washington, DC: National Academies Press. www.nap.edu/catalog/10022.html. www.nap.edu/catalog/10022.html Education Northwest. educationnorthwest.org/resource/focus/Evaluation.educationnorthwest.org/resource/focus/Evaluation Harvard Family Research Project. www.hfrp.org. Little, P. M. D. (Ed.). (2003, Spring). Evaluating out-of-school time. The Evaluation Exchange, 9(1). www.gse.harvard.edu/hfrp/eval/issue21www.gse.harvard.edu/hfrp/eval/issue21 SEDL. www.sedl.org/expertise/afterschool_products.html.www.sedl.org/expertise/afterschool_products.html U.S. Department of Education. (2003, February 3). Evaluation shows improvements needed in after school program. Washington, DC: Author. www.ed.gov/PressReleases/02-2003/02032003c.html.ed.gov/PressReleases/02-2003/02032003c.html