1. Improving Program Quality in Central Texas

2. Agenda Overview Accomplishments Improvements Future Opportunities Break Assessors Methods Trainers

3. Community Action Framework for Positive Youth Development You Are Here! (Gambone et al., 2004)

4. 4 Overview STEP 1 Decide to build system STEP 2a Self- assessment STEP 2b External assessment STEP 3 Plan for improvement STEP 4 Carry out plan SAE System Accountability Environment PLC Professional Learning Community POS Point Of Service Quality Matters presentation PQA Basics PQA Intermediate Anchoring Planning with Data A la carte Methods Workshops TA Quality Coaching STEP 5 Measure change Program Self- Assessment External Assessment Observation- Reflection

5. Safe Environment Healthy food and drinks Program space and furnitureEmergency procedures Psychological and emotional safety Physically safe environment Reframing conflictEncouragement Skill building Active engagement Session flow Welcoming atmosphere What is Assessed? Partner with adults Lead and mentor Be in small groups Experience belonging Interaction Make choices Plan Reflect Engagement Healthy food and drinks Program space and furnitureEmergency procedures Psychological and emotional safety Physically safe environment Reframing conflictEncouragement Skill building Active engagement Session flow Welcoming atmosphere Supportive Environment

6. Alignment of Professional Development Opportunities Aligned Professional Development Courses

7. Cycle of Improvement Assess program Attend Professional Development Assess Program More Professional Development Start Here Over the Span of a Year

8. Highlights from Palm Beach County, FL: Prime Time Initiative Goal was to develop a comprehensive, county-wide system of supports and resources for OST programs Overall Reported Strengths Teamwork Support Opportunities for networking, program enhancements and training

9. Prime Time (continued) Specific Professional Development Strengths Refined offerings to meet shared learning needs Connect training to program improvement plans Variety of modules Quality of workshop leadership Tools and ideas for program improvement* Real changes Better communication Better activities More youth involvement Impetus to make time for change

10. Quality Improvement System QIS Process Baseline Assessment Quality Advisor Assignment Letter of Recommendation for Improvement Training Self-assessment* Program Improvement Plan* Implementation Reassessment

11. Prime Time (continued) Pilot Quality Improvement System 40 baseline programs; 37 follow-up Concerns about managing baseline assessments and how assessment would be used How were assessment handled better? Unclear. Just says “they learned to do it better” Internal/external assessment value became clearer to providers as it was used for quality improvement Satisfaction increased with use and understanding

12. Prime Time Lessons: It Takes Time! To collect baseline data and develop a quality improvement plan To form meaningful relationships with people in the improvement process/resource providers It takes initiative, time and energy for staff to reflect on work and implement new practices Also, the culture of expectations are as diverse as the participants and affect the process

13. Prime Time Recommendations: Ensure sufficient communication about the process, especially for new participants Consider presenting score reports in person for new participants (by a knowledgeable party) Bring agency/site directors together to discuss strategies to best support the improvement process (talk to each other about what helps) Strengthen rapport between sites and improvement facilitators (advisors/coaches)

14. Accomplishments

15. Educating Leaders – Quality Matters Building Local Capacity 27 trained self assessors 19 trained external assessors 6 methods trainers Improving Program Quality 3 methods trainings 22 attended Planning with Data

16. Comparison Scores National Average and Austin External Scores National Average: N= 735 offerings in 180 organizations Austin External Assessments: N=12 offerings

17. Comparison Scores Austin External and Self Assessment Scores Austin External Assessment: N=12offerings Austin Self Assessment: N=18 assessments

18. Additional Data Notes Engagement was significantly related to both supportive environment and interaction, but the relationship was strongest with interaction Thus, programs that were stronger in interaction were more likely than any others to be strong in engagement

19. Key factors for high engagement Opportunity to develop a sense of belonging Opportunities to act as group facilitators* Opportunity to partner with adults Support youth with encouragement Support youth with building new skills Support active engagement* Provide welcoming atmosphere * Explains 56% of variance in engagement

20. Improvements

21. Score Improvement Suggestions Our goal is for each person to measure similar actions the same way each time we go to a location, regardless of where it is (reliability) - and for this action to be measured correctly (validity)

22. More Suggestions Provide supporting evidence Scores can only be: 1, 3, 5 or X Small groups (III-M) (2) If no small groups are used, then “1” not NR (X) (3) Quality of small groups Conflict/Supportive Environment (II-K) If not observed ask the follow-up questions Evidence: no conflict or no procedure for conflict

23. What else can be improved? Table discussions Report out

24. Future Opportunities

25. Quality Coach Quality Advisor Youth PQA Trainers Quality Matters Presentation More Methods Trainings & Assessments

26. Quality Coach Prepares youth program professional to help youth works improve their program or the way they work with youth. Supervisors or Managers Network Consultants September 2009 *CYPQ Quality Coaching Endorsement

27. Quality Advisors Prepares individuals to be network consultants to work with programs and staff throughout the quality improvement process. October/November 2009

28. Youth PQA Trainers Prepares participants to train others in Youth PQA related workshops including: Youth PQA Basics Youth PQA Intermediate Planning with Data June/July 2009 *Participants must have extensive experience collecting PQA data, participated in Basics, Intermediate and Planning with Data trainings. External Assessor

29. Methods Trainings & Assessments 12 - month calendar Increase in cost per training $20 for programs involved in assessment process $25 for anyone interested in the modules More baseline assessments coming in April/May

30. Quality Matters Presentation Present recent findings from the YPQI- Setting Change Study, highlight the data and success of local QIS and facilitate a discussion about embedding the improvement work. Decision-makers, stakeholders, funders and programs.

31. New Cycle of Improvement Methods Trainers Methods Trainers Self Assessor External Assessor Self Assessor External Assessor Over the Span of a Year Quality Coach Quality Coach Start Here Quality Advisor Youth PQA Trainer

32. Professional Development Professional Development and Quality Assessment2009 JANFEBMARAPRMAYJUNJULAUGSEPOCTNOVDEC External Assessments Baselines continue Planning w/data Professional Development Outcomes and Indicators, Youth Service Mapping - Compact Process Begins Annual External Assessments Annual - Self Assessments Quality Coaching Workshop Youth PQA Training Of Trainers Quality Advising Workshop1 Quality Matters Presentation Quality Matters Presentation Professional Development QI Overview External Assessor Self Assessor

33. What is next? short- term March 31 st Ask, Listen and Encourage Module Reframing Conflict April / May – External Assessments Method Schedule of Modules

34. Break

35. Score Improvement Suggestions Our goal is for each person to measure similar actions the same way each time we go to a location, regardless of where it is (reliability) - and for this action to be measured correctly (validity)

36. More Suggestions Provide supporting evidence Scores can only be: 1, 3, 5 or X Small groups (III-M) (2) If no small groups are used, then “1” not NR (X) (3) Quality of small groups Conflict/Supportive Environment (II-K) If not observed ask the follow-up questions Evidence: no conflict or no procedure for conflict

37. Example: Supportive Environment (II-J. Staff Support) “Teacher asked many open-ended questions: ‘What’s happening in this poem?’ ‘What does it mean to rearrange the earth?’ ‘What are some of the other things these poems have in common?’ ‘How do you feel about this?’ ‘Anything you didn’t like or a weird word you would change?’”

38. Example: Supportive Environment (II-G. Session Flow) “Teacher thoroughly explains writing exercise twice. A student comes late. Teacher has a student explain it to the late student and then follows up with additional directions.”