1. Evaluating your 21st CCLC Program and Lessons Learned by Veteran Program Directors FY13 Program Training August 7, 2012
Presenter: Michael Thaler 21st CCLC State Program Manager (404)

2. Goals of this Presentation
Understand the importance of program evaluation
Discuss the components of the evaluation reports
Understand the roles and expectations of an external evaluator
Learn about tools other 21st CCLC programs are using
Receive tips and knowledge that our most veteran program directors wish they had known their first year

3. “We are doing a great job of running our programs and we are effective, but we aren’t telling our story and that’s hurting us.”
Dr. Sylvia Lyles
Beyond School Hours XIII Conference
February, 2010

4. Perception vs. Reality - Daycare centers
- Unorganized or ineffective activities
- Minimal impact on schools, families and community
- Adult employment agency
Reality
- Highly organized out-of-school programs
- Utilize research-based activities that specifically target ‘at-risk’ or ‘high opportunity’ students
- Provide critical services to a number of stakeholders

5. Our job is to tell the story of our 21st CCLC programs to our various stakeholders so we can change their perception and help educate them about positive impact of our programs. This is accomplished by developing and utilizing an effective evaluation plan and following the principles of effectiveness.

6. Why should you do it?
Principles of Effectiveness (Sec (b)) states ‘the program or activity shall undergo a periodic evaluation to assess its progress toward achieving its goal of providing high quality opportunities for academic enrichment’

7. Principles of Effectiveness (Sec. 4205 (b))
The program or activity should be based upon: 1. An assessment of objective data regarding the need for the program 2. An established set of performance measures 3. Scientifically-based research that demonstrates the activity will help the students meet State or local standards

8. What can evaluations provide?
In addition to letting you know how you are doing, effective evaluation plans can help:
Increase student and parent participation
Increase school support
Obtain additional partnerships
Obtain additional funding
Increase the effectiveness of your program

9. Essential Steps for an Effective Evaluation Plan
Determine the needs of your students and families
Establish clear goals, objectives, and activities based on the identified needs
Develop the timeline for the evaluation plan
Collect and analyze baseline, formative, and summative data
Disseminate the evaluation results to your stakeholders
Refine the program and its activities for next year, based upon the evaluation results

10. Evaluation Requirements
3 Major Components:
- Formative (mid-year) Assessment
- Summative (end of year) Assessment
- Cayen/Afterschool 21 data entry system
Summer Evaluation, if your program will run one in 2013

11. Formative Assessment
Must undergo at least one and it must be written by the external evaluator
Evaluates from the beginning of the program year to the time of the evaluation
At a minimum, the report must include:
Student Attendance: Data on average daily attendance and enrollment at each site;
Program Operation: Synopsis of current level of operation at each site;
Objective Assessment: Data analysis and indication of progress towards achieving EACH objective (as required, all objectives must include measures that allow for continuous (formative) assessment); and
Recommendations: Recommendations for programmatic refinement for all objectives where progress is not being achieved and/or where the program is not likely to achieve the stated objective by the end of the program year.

12. Formative Assessment Due February 1, 2013
Important to continuously assess your program throughout the year (i.e., grades, attendance)
Effective programs make revisions throughout, based upon their formative assessments
Share the results with stakeholders

15. Summative Assessment
Each program must submit a summative report by June 28th, 2012
Evaluates the entire program year
The results of your summative report will be used to help determine continuation of funding in subsequent years
Must be written by your external evaluator and must follow the template on the following slides

16. Summative Report Template
Each summative report must include, at minimum, the following major sections:
Overview and History: Provide a general overview of the 21st CCLC program, including a history of previous operation and how the program has progressed and grown over the past year(s) of operation.
Student Attendance and Enrollment: Total and Regularly-Participating Student Enrollment (per site); Student Demographics (e.g., gender, ethnicity); Average Daily Attendance (per site)
Program Operation: The summative report must include information on operation for EACH site within the program (e.g., days, hours, and weeks of operation for each component).
Quality of Staffing: The composition of site staff is one method for assessing the quality and breadth of 21st CCLC programs. For each site within the program, the summative report must include information about staffing (e.g., Staff Demographics, Student: Staff Ratio, Staff Training, and Use of Certified Teachers).

17. Summative Report Template
Objective Assessment: Within the grant application, 21st CCLC programs proposed both objectives and methods of evaluating progress towards achieving the objectives. The summative report must include detailed information for EACH OBJECTIVE approved by GaDOE (e.g., activities, data collected, timeline, analysis and results). Programs are NOT permitted to revise, reword, or otherwise change their objectives without specific written approval from GaDOE.
Other Observations (Optional): If appropriate, provide other relevant findings pertaining to this 21st CCLC program. Other findings could include qualitative and/or quantitative data not related to specific objectives; quotes and/or statements from students, parents, and/or teachers; success stories of students within the program; and photographs demonstrating unique program activities and services.

18. Summative Report Template
Progress towards Sustainability: The summative report must provide information about program partnerships (e.g., partnership development, partner maintenance, and contributions to the program).
Overall Recommendations: Provide an overall assessment of this 21st CCLC program, as well as any program-wide recommendations to enhance program quality.

19. Major Outcomes/GaDOE’s Expectations
GaDOE expects every 21st CCLC program to demonstrate the following:
Increase in grades from fall to spring (no less than 85% of student grades entered)
Increase in CRCT/EOCT scores (no less than 85% of students with
scores in tested grade levels)
Increase in school day and program attendance
Improvement in areas such as discipline, homework completion,
parent involvement, parent/student satisfaction, etc.

20. Cayen/ Afterschool 21 Data Entry System
Programs must enter data (grades, attendance, activities, etc.) into this system continuously throughout the life of the grant
GaDOE uses this to fulfill its requirement of reporting data to US ED
All new subgrantees must attend webinars and work closely with their assigned AS21 contractor

21. External Evaluator
Provides a critical service to our programs and should be an experienced professional
Contact multiple evaluators and ‘shop around’
- 21st CCLC experience?
- Speak with some of their current programs and see if they will provide you with one of the reports
- Ensure you hire one early in the program year
Can only use 3% of your budget for this and must enter in to one year contracts
Cannot be the person who wrote the grant
The quality of your evaluator can have a significant impact on the quality your program!

22. Role of an External Evaluator
Remember - They work for you, you don’t work for them
They are there to provide guidance and support, but ultimately the success of the program relies on the program administration
They are somewhat limited by the quality of data you provide to them
Refer to the operations manual for more information requirements/recommendations for the external evaluator

23. What must they provide? At a minimum: - One formative assessment
- Summative assessment after the conclusion of the program year
- A list of recommended revisions based upon the results of the formative and summative assessments

24. Suggestions: What to Request
Conduct site visits at least twice a year
This can vary depending on the size of your grant or location of your evaluator
Assist with developing surveys and other measurement instruments
Assist with the dissemination of evaluation results
i.e., One page flyer and/or brief PowerPoint presentation

25. Suggestions: What to Request
Participate and present at school/advisory/stakeholder meetings
Conduct focus groups with students, staff, and/or parents
Provide input in revising objectives and assessments
Monthly meetings or teleconferences to ensure the evaluation plan is on track

26. Helpful Hints
Set up your contract so your payments align with deliverables, not just timeframes
Clearly explain to them the evaluation requirements
If possible, bring them to state/national conferences and state meetings

27. Other Helpful Hints
Disaggregate the results of your summative evaluation based on number of days attended:
0-29 days
30-59 days
60-89 days
90+ days
Compare regularly participating students (30+ days) to non-21st CCLC students
Track students for the life of your grant

28. Additional Evaluation Tools
Needs Assessments or Interest Surveys
Students - determine activities that would interest them
Consider pre and post surveys
Staff - determine the type of training that would benefit them the most (example)
Staff - determine which types of enrichment activities they would like to teach
Parents - not only what type of activities, but when and where to hold them

29. ____________________________________
DJJ AFTER-SCHOOL PROGRAM
PROFESSIONAL DEVELOPMENT SURVEY
INTRODUCTION: This professional development survey was developed for DJJ After-School Program teachers. Survey results will help determine future professional development activities. Read each statement carefully and place a check mark in the appropriate space that best represents your answer for Very Interested (VI), Somewhat Interested (SI) or Not Interested.
Professional Development Activity VI SI NI
1. Transition school-day activities to After-School Program
activities
2. Integrate related reading and math learning activities for enrichment and non-remediation classes.
3. Evaluate and document learning in enrichment and non-remediation classes
4. Develop course outlines and weekly lessons in accordance with program objectives
5. Identify what teaching techniques promote learning in the After-School Program classroom.
6. Develop additional enrichment activities
7. Promote parental involvement and determine activities to keep them coming back
8. Discover supplemental teaching materials that work
9. Develop/refine classroom management skills for After-School Program
10. Maintain a safe, secure classroom environment while promoting learning and creative expression
11. Understand the influence of school, family and peers with delinquency
12. Gain national, state and DJJ perspectives of After-School Programs
13. How is the After-School Evaluated?
14. Time to share After-School Program experiences with other after-school teachers
15. How to get more community participation for speakers, classroom supplies, etc
Comments Suggestions: ________________________________________________________________________
____________________________________

31. Additional Evaluation Tools
Student progress reports
Classroom/teacher observation forms
Parent satisfaction survey

36. Lessons Learned by Veteran 21st CCLC Program Directors
I asked some of our most successful and experienced 21st CCLC program directors to come up with a list of tips and pieces of knowledge they wish they knew their first year
Review of Lessons Learned handout
This is GOLD

37. Any Questions?
The biggest thing to remember is you have a support system of GaDOE staff and other subgrantees to help you through this process.