Improving Program Quality in Central Texas

Agenda Overview Accomplishments Improvements Future Opportunities Break Assessors Methods Trainers

Community Action Framework for Positive Youth Development You Are Here! (Gambone et al., 2004)

4 Overview STEP 1 Decide to build system STEP 2a Self- assessment STEP 2b External assessment STEP 3 Plan for improvement STEP 4 Carry out plan SAE System Accountability Environment PLC Professional Learning Community POS Point Of Service Quality Matters presentation PQA Basics PQA Intermediate Anchoring Planning with Data A la carte Methods Workshops TA Quality Coaching STEP 5 Measure change Program Self- Assessment External Assessment Observation- Reflection

Safe Environment Healthy food and drinks Program space and furnitureEmergency procedures Psychological and emotional safety Physically safe environment Reframing conflictEncouragement Skill building Active engagement Session flow Welcoming atmosphere What is Assessed? Partner with adults Lead and mentor Be in small groups Experience belonging Interaction Make choices Plan Reflect Engagement Healthy food and drinks Program space and furnitureEmergency procedures Psychological and emotional safety Physically safe environment Reframing conflictEncouragement Skill building Active engagement Session flow Welcoming atmosphere Supportive Environment

Alignment of Professional Development Opportunities Aligned Professional Development Courses

Cycle of Improvement Assess program Attend Professional Development Assess Program More Professional Development Start Here Over the Span of a Year

Highlights from Palm Beach County, FL: Prime Time Initiative Goal was to develop a comprehensive, county-wide system of supports and resources for OST programs Overall Reported Strengths Teamwork Support Opportunities for networking, program enhancements and training

Prime Time (continued) Specific Professional Development Strengths Refined offerings to meet shared learning needs Connect training to program improvement plans Variety of modules Quality of workshop leadership Tools and ideas for program improvement* Real changes Better communication Better activities More youth involvement Impetus to make time for change

Quality Improvement System QIS Process Baseline Assessment Quality Advisor Assignment Letter of Recommendation for Improvement Training Self-assessment* Program Improvement Plan* Implementation Reassessment

Prime Time (continued) Pilot Quality Improvement System 40 baseline programs; 37 follow-up Concerns about managing baseline assessments and how assessment would be used How were assessment handled better? Unclear. Just says “they learned to do it better” Internal/external assessment value became clearer to providers as it was used for quality improvement Satisfaction increased with use and understanding

Prime Time Lessons: It Takes Time! To collect baseline data and develop a quality improvement plan To form meaningful relationships with people in the improvement process/resource providers It takes initiative, time and energy for staff to reflect on work and implement new practices Also, the culture of expectations are as diverse as the participants and affect the process

Prime Time Recommendations: Ensure sufficient communication about the process, especially for new participants Consider presenting score reports in person for new participants (by a knowledgeable party) Bring agency/site directors together to discuss strategies to best support the improvement process (talk to each other about what helps) Strengthen rapport between sites and improvement facilitators (advisors/coaches)

Accomplishments

Educating Leaders – Quality Matters Building Local Capacity 27 trained self assessors 19 trained external assessors 6 methods trainers Improving Program Quality 3 methods trainings 22 attended Planning with Data

Comparison Scores National Average and Austin External Scores National Average: N= 735 offerings in 180 organizations Austin External Assessments: N=12 offerings

Comparison Scores Austin External and Self Assessment Scores Austin External Assessment: N=12offerings Austin Self Assessment: N=18 assessments

Additional Data Notes Engagement was significantly related to both supportive environment and interaction, but the relationship was strongest with interaction Thus, programs that were stronger in interaction were more likely than any others to be strong in engagement

Key factors for high engagement Opportunity to develop a sense of belonging Opportunities to act as group facilitators* Opportunity to partner with adults Support youth with encouragement Support youth with building new skills Support active engagement* Provide welcoming atmosphere * Explains 56% of variance in engagement

Improvements

Score Improvement Suggestions Our goal is for each person to measure similar actions the same way each time we go to a location, regardless of where it is (reliability) - and for this action to be measured correctly (validity)

More Suggestions Provide supporting evidence Scores can only be: 1, 3, 5 or X Small groups (III-M) (2) If no small groups are used, then “1” not NR (X) (3) Quality of small groups Conflict/Supportive Environment (II-K) If not observed ask the follow-up questions Evidence: no conflict or no procedure for conflict

What else can be improved? Table discussions Report out

Future Opportunities

Quality Coach Quality Advisor Youth PQA Trainers Quality Matters Presentation More Methods Trainings & Assessments

Quality Coach Prepares youth program professional to help youth works improve their program or the way they work with youth. Supervisors or Managers Network Consultants September 2009 *CYPQ Quality Coaching Endorsement

Quality Advisors Prepares individuals to be network consultants to work with programs and staff throughout the quality improvement process. October/November 2009

Youth PQA Trainers Prepares participants to train others in Youth PQA related workshops including: Youth PQA Basics Youth PQA Intermediate Planning with Data June/July 2009 *Participants must have extensive experience collecting PQA data, participated in Basics, Intermediate and Planning with Data trainings. External Assessor

Methods Trainings & Assessments 12 - month calendar Increase in cost per training $20 for programs involved in assessment process $25 for anyone interested in the modules More baseline assessments coming in April/May

Quality Matters Presentation Present recent findings from the YPQI- Setting Change Study, highlight the data and success of local QIS and facilitate a discussion about embedding the improvement work. Decision-makers, stakeholders, funders and programs.

New Cycle of Improvement Methods Trainers Methods Trainers Self Assessor External Assessor Self Assessor External Assessor Over the Span of a Year Quality Coach Quality Coach Start Here Quality Advisor Youth PQA Trainer

Professional Development Professional Development and Quality Assessment2009 JANFEBMARAPRMAYJUNJULAUGSEPOCTNOVDEC External Assessments Baselines continue Planning w/data Professional Development Outcomes and Indicators, Youth Service Mapping - Compact Process Begins Annual External Assessments Annual - Self Assessments Quality Coaching Workshop Youth PQA Training Of Trainers Quality Advising Workshop1 Quality Matters Presentation Quality Matters Presentation Professional Development QI Overview External Assessor Self Assessor

What is next? short- term March 31 st Ask, Listen and Encourage Module Reframing Conflict April / May – External Assessments Method Schedule of Modules

Break

Score Improvement Suggestions Our goal is for each person to measure similar actions the same way each time we go to a location, regardless of where it is (reliability) - and for this action to be measured correctly (validity)

More Suggestions Provide supporting evidence Scores can only be: 1, 3, 5 or X Small groups (III-M) (2) If no small groups are used, then “1” not NR (X) (3) Quality of small groups Conflict/Supportive Environment (II-K) If not observed ask the follow-up questions Evidence: no conflict or no procedure for conflict

Example: Supportive Environment (II-J. Staff Support) “Teacher asked many open-ended questions: ‘What’s happening in this poem?’ ‘What does it mean to rearrange the earth?’ ‘What are some of the other things these poems have in common?’ ‘How do you feel about this?’ ‘Anything you didn’t like or a weird word you would change?’”

Example: Supportive Environment (II-G. Session Flow) “Teacher thoroughly explains writing exercise twice. A student comes late. Teacher has a student explain it to the late student and then follows up with additional directions.”