Presentation on theme: "1 Youth PQA Basics Center For Youth Program Quality Youth Program Quality Assessment."— Presentation transcript:
1 Youth PQA Basics Center For Youth Program Quality Youth Program Quality Assessment
Agenda Welcome Opening: Intro to the Youth PQA Observational Note Taking Fitting & Scoring Building Your Pyramid Self Assessment Q&A
The Quality Counts Initiative Why Quality Counts, Who We Are, & Where We’re Going
4 The American Reality Only 4 in 10 youth ready, only 1 in 3 youth supported, too few leaders effective. The American Dilemma Fragmentation, complacency, and low expectations of youth, communities and leaders The American Dream All youth ready, every family and community supportive, each leader effective. The Ready by 21™ Challenge Change the odds for youth by changing the way we do business WHY? Because Rhetoric and Reality Don’t Match
Ready for Work Youth Employment Outcomes Ready for College Academic Outcomes Ready for Life Youth Development Outcomes 21 st Century Skills & Content Information & Media Literacy * Creativity * Intellectual Curiosity * Critical & Systems Thinking * Accountability and Adaptability * Communication * Problem Solving * Interpersonal Skills * Social Responsibility * Financial Literacy * Global Awareness * Civic Literacy * Self-Direction Cultural, Physical & Behavioral Health Knowledge & Skills Specific Vocational Knowledge & Skills …There is Increasing Agreement on Skills Needed for the 21 st Century… Subject Matter Knowledge
6 Researchers Agree on What It Takes to Support Development The National Research Council & Institute for Medicine list the following key features of positive youth development settings: Physical and psychological safety Appropriate structure Supportive relationships Opportunities to belong Positive social norms Support for efficacy and mattering Opportunities for skill-building Integration of family, school and community efforts - Community Programs to Promote Youth Development, 2002
7 Longitudinal Studies Confirm that these Supports Make a Difference
8 Providing these Supports Can Change the Odds from 4 in 10 doing well to 7 in 10 doing well*. Gambone/Connell’s research suggests that if all young people got the supports they needed in early adolescence, the picture could change…
9 mbus etroit Minneapolis Kentucky Iowa Oklahoma New York Rhode Island Austin Georgetown Divide Columbus Indianapolis Grand Rapids Nashville St. Louis Washington* West Palm Beach County Rochester Chicago Systems for Quality Accountability Policies in Places YPQA is part of state and county accountability policies: –Cross sector (DHS& DOE) snapshots: Iowa, Washington, Arkansas –Statewide 21st Century: Michigan, Maine, Minnesota, Rhode Island, New Mexico, –Cities and Counties: Rochester, Detroit, Grand Rapids, Palm Beach
10 WHO WE ARE - 3 Nationals, 5+ States, 7 Localities STATES (w/ participating localities) Iowa Linn County, N. Central Iowa, Polk County/Des Moines Kentucky Lexington, Louisville New York Broome, Onondaga, Orange & Rockland Counties Oklahoma Norman, Tulsa Rhode Island Central Falls, Newport, Pawtucket Washington State (Honorary QC member) LOCALITIES Austin, TX Georgetown Divide (Black Oak Mine), CA Columbus (Bartholomew County), IN Grand Rapids, MI Indianapolis, IN Nashville, TN St. Louis, MO NATIONAL PARTNERS The Forum for Youth Investment The Center For Youth Program Quality (formerly High/Scope Youth Development Group) AED National Training Institute for Community Youth Work
WHERE WE’RE GOING - Four Domains The Quality Counts initiative is designed to help you maximize your capacity to improve the quality and reach of youth programs by taking concrete steps in four areas (domains) to ensure that you have:. 11 Capacity to Recruit, Train, Retain Workforce Strong Policy / Leadership Horsepower Capacity to Assess & Improve Programs A Strong & Stable Program Base
12 Why bother paying attention to youth program quality? Network leaders build awareness and communication Programs see their strengths and areas to improve, receive critical feedback, plus improve cmty networking and collaboration Funders gain an important metric Youth can have better program experiences
13 What is program quality? ??? inputs outcomes youth program Another way to say it: What do we want to see in high quality youth programs?
High POS Quality Several ways to organize: Readin’ + ‘Ritin + ‘Rithmatic (old-school) Affect + Active Learning + Metacognition (Education) Relationships + Relevance + Rigor (Education 2.0) Relatedness + Autonomy + Competence (Psychology SDT) Safety + Belonging + Esteem (Psychology old-school) 8 Features of Positive Youth Development Settings (PYD) Relationship + Task + Increasing Complexity ContentTherapeutic process
15 The Pyramid of Program Quality Plan Make choices Reflect Partner with adults Lead and mentor Be in small groups Experience belonging Engagement Reframing conflictEncouragement Skill building Active engagement Session flow Welcoming atmosphere Supportive Environment Interaction Safe Environment Healthy food and drinks Program space and furnitureEmergency procedures Psychological and emotional safety Physically safe environment Professional Learning Community Youth Voice and Governance
16 What is the Youth PQA? 1. A validated instrument designed to assess the quality of youth programs and identify staff training needs. 2. A set of items that measures youth access to key developmental experiences. 3. A tool which produces scores that can be used for comparison and assessment of progress over time.
18 Program Offering Level Validation Study: Outside Observer (N=140 offerings) Self-Assessment Pilot (N=24 orgs) Safe environment Supportive environment Opportunities for interaction Engagement Organization Level(N=51 orgs)(N=24 orgs) Youth centered policies and practices High expectations for students and staff Access Scores From Validation Studies
Contrasting Pedagogy Profiles N=599 offerings in 120 organizations 5 = occurred for everyone 3 = occurred for some 1 = did not occur
How we think about DDCI - People change not programs Contemplation Preparation Action Maintenance Quality assessment Planning with data Implementation & coaching Repeat cycle SAE System Accountability Environment PLC Professional Learning Community POS Point Of Service Individual Change Model Organizational Context Prochaska, J.O., & DiClemente, C.C. (1982). Transtheoretical therapy toward a more integrative model of change. Psychotherapy: Theory, Research and Practice, 19(3), /13/200821The Center for Youth Program Quality
22 Designing Quality Improvement Systems (QIS) System Capacity Program Staff Skill & Knowl Self- assessment of Quality (b) Planning with Data (c) External quality assessment (a) Quality Advisor (e) TOTs for quality assess, coaching, and youth work methods (f,g) Coaching & Training Phase 1: Readiness & Capacity Phase 2: Impact & Sustainability Targeted youth work methods training for direct staff (h) Quality coaching by managers (i) Use of on-line dashboards and training (d) External Quality Report with Norms Self-Assessed Quality Report Use of on-line dashboards and training (d) Mostly Mangers Managers and direct staff
23 Defining the Purpose of Your QIS Lower Stakes Program Self-Assessment Rough data to get staff thinking and discussing program quality in the context of best practice Less time Less money Impact internal audiences Higher Stakes External Assessment Precise data for internal and external audiences for evaluation, monitoring, accountability, improvement, reporting More time More money Impact internal and external audiences (the creative middle)
Columbus Indiana Phase 1: Building Local Capacity STEP 1 Decide to build system STEP 2a Self- assessment STEP 2b External assessment STEP 3 Plan for improvement STEP 4 Carry out plan SAE System Accountability Environment PLC Professional Learning Community August 25 Quality Matters Presentation August 26 Youth PQA Basics January Ext Assessment October 8 Planning with Data Improvement Plan Opt Phase 2 Method Workshops Quality Coaching STEP 5 Measure change Annually Program SA Ext Assessment Observe-Reflection Planning with Data POS Point Of Service
Step 2: Pilots Select Self-Assessment Teams and Develop Schedule 26
Step 3: Self Assessment 27
Step 4: Submit Scores 28
High/Scope’s eTools Web Site etools.highscope.org Your one stop shop for: –Online training Intro to Youth PQA Intro to Scores Reporter –Online assessment Submitting Youth PQA Scores Generate reports
Online PQA Scores Reporter etools.highscope.org 3 levels of users -Youth Program Facilitators -Youth Program Directors -Network Users
Online PQA Scores Reporter Enter Data with mouse or keyboard
Online PQA Scores Reporter View & Print Reports Your programs scores can automatically be compared to national norms
Step 5: Teams attend Planning with Data and create Program Improvement Plans 34
Step 6: Staff members work the plan! Repeat Youth PQA annually… 35
Does it Work? Findings from Several Samples POS quality-outcomes findings: –Supportive environment related to: Attendance –Interaction related to: Interest in program –Engagement related to: Sense of challenge, sense of growth, school-day reading, school-day suspension –Note: No offerings get to high engagement without high support and high interaction Quality Improvement (YPQI) Findings –Scores increase from pre to post –Scores increase in the targeted areas more –Management practices are related to quality change (Vision, Feedback, Continuity)
Questions about… Purpose? Process? Pilot? Next Steps… 37
38 Youth Program Quality Assessment
39 Structure of the Youth PQA Form B Organizational Interview Ask questions, write, score (2 hours) Form A Observation Watch, write, score (3 hours) Organization Program Offering 1 Program Offering 2 Program Offering 3 Program Offering 4
40 Sample indicator III. Interaction III-L. Youth have opportunities to develop a sense of belonging. Note: Structured refers to the quality of being intentional, planned, and/or named; it does not refer to informal conversation. IndicatorsSupporting Evidence 1Youth have no opportunities to get to know each other (beyond self- selected pairs or small cliques). 3Youth have informal opportunities to get to know each other (e.g., youth engage in informal conversations before, during, or after session. 5Youth have structured opportunities to get to know each other (e.g., there are team- building activities, introductions, personal updates, welcomes of new group members, icebreakers, and a variety of groupings for activities) The Youth PQA consists of 2 scales (Form A & Form B); 7 subscales (4 in A, 3 in B) 30 items (18 in A, 12 in B); 103 indicator rows (60 in A, 43 in B) “subscale” “item” “indicator row”
41 Indicator Scramble What are the key words? What are the key quantities? What are we counting?
42 Sample Anecdotes Youth sort of identified with program offering. III-L (3 rd row) Kids sometimes taunt each other and try to annoy each other, but they seem to get along, too. I-A (1st row) Kevin dominated committee meetings he sat in on. Did better job of listening and acknowledging comments in the large group. III-O (1st row)
43 Observational Note Taking EffectiveIneffective Objective as possible Rich detail in snapshot form Focus on interactions between –Staff and youth –Youth and youth –Youth and environment Allow time for interactions to reach completion State the outcome of interactions Who, what, when, where Quotes: what youth and staff say Lists of materials What you see in the room Anecdote can stand alone Subjective terms such as ‘good’ or ‘bad’ Rater’s opinions Assumptions about internal states: she felt angry; he did not get it Anecdotes are too vague; lack detail Lacking facts: what you see and hear Summary in the place of quotes Raters repeat what the indicator says For a 3 involving some positive and some negative, raters have one but not the other Anecdote does not fit the indicator Anecdote could support more than one score
45 Follow-up questions Only ask for items with question(s) in the evidence column. Ask the questions as they are printed. Form A items with at least one “Ask” indicators (24 total) I-C indicator rows 1-6 I-D indicator row 4 II-K indicator rows 1-4 III-L indicator row 1 III-N indicator rows 2-3 IV-P indicator rows 1-2 IV-Q indicator rows 1-2 IV-Rindicator rows 2-4
46 Fit and Score (label practice) 1.Try to place each label in the correct row for each item. Look over the anecdotal evidence for each item and see how the anecdotes are a “fit” with the indicator rows. 2.Score each indicator row. For each row, read through the levels of the indicator row and select the level - 1, 3, or 5 - that most closely agrees with your evidence. Pay close attention to words like some, most, and always. Start at level 5 to see the full description of necessary evidence and check agreement with the anecdotal evidence by looking to the lower levels.
47 More on Scoring the YPQA Fitting data to the indicator rows can eventually happen in your head while you are observing. Always try to see multiple items in every interaction and cross-reference constantly. Look to a preponderance of evidence but favor higher scores. If you lack evidence with “fit,” collect more data.
48 Assessing POS Quality using Youth PQA Let’s Try It! Watch, Write, Score Collecting objective anecdotal evidence Scoring rubrics Reliability Instructions –In this activity you will watch a short (4 minute) video clip, take notes, and score a few indicators based on the video. –Before you watch the video, get ready: Make sure you have your hardcopy of Form A of the Youth PQA in front of you for this activity. Make sure you have scrap paper and a pen or pencil handy Review items II-J, III-M, and IV-Q (you will be scoring one indicator from each) While you watch the video take notes by hand. Make sure your notes are as objective as possible. Try to capture actual quotes.
49 More on Self-Assessment Self-assessment only works if program (direct) staff are involved. Try to get a good mix of program times and offerings. Assessment is different from evaluation. Assessment gives you the opportunity to see where you are, whereas evaluation suggests judgment. Remember, the conversation is the most important part of self-assessment.
50 Youth PQA Step by Step Self- Assessment STEP External Assessment Team that includes as many frontline staff as possible 1. Select Assessor(s) Use anchored rater if possible When will data be collected? Who watches whom? 2. Plan the Data Collection Check time & schedule Make sure frontline staff knows you’re coming Staff observe each other in minute chunks & take notes 3. Collect Data 1-2 hr visit Take notes Ask follow-up questions Have a meeting. Go indicator- by-indicator, sharing evidence and coming to consensus on each score 4. Fit & Score Fit & score each indicator Enter indicator scores into Scores Reporter 5. Generate Report Enter indicator scores into Scores Reporter Use the report to make plans for items to work on 6. Plan for Improvement Use the report to make plans for items to work on
51 How will you use this data to promote the right kinds of Accountability Behaviors? Focus Us Survive Engage Me ComplyAvoid/Resist LowHigh Commitment