Presentation is loading. Please wait.

Presentation is loading. Please wait.

Center For Youth Program Quality

Similar presentations

Presentation on theme: "Center For Youth Program Quality"— Presentation transcript:

1 Center For Youth Program Quality
Youth Program Quality Assessment Youth PQA Basics Center For Youth Program Quality 1

2 Agenda Welcome Opening: Intro to the Youth PQA
Observational Note Taking Fitting & Scoring Building Your Pyramid Self Assessment Q&A

3 The Quality Counts Initiative
Why Quality Counts, Who We Are, & Where We’re Going 3

4 WHY? Because Rhetoric and Reality Don’t Match
The American Dream All youth ready, every family and community supportive, each leader effective. The American Reality Only 4 in 10 youth ready, only 1 in 3 youth supported, too few leaders effective. The American Dilemma Fragmentation, complacency, and low expectations of youth, communities and leaders The Ready by 21™ Challenge Change the odds for youth by changing the way we do business 4 4

5 Ready for Work Youth Employment Outcomes
…There is Increasing Agreement on Skills Needed for the 21st Century… Ready for Work Youth Employment Outcomes Ready for College Academic Outcomes Specific Vocational Knowledge & Skills 21st Century Skills & Content Information & Media Literacy * Creativity * Intellectual Curiosity * Critical & Systems Thinking * Accountability and Adaptability * Communication * Problem Solving * Interpersonal Skills * Social Responsibility * Financial Literacy * Global Awareness * Civic Literacy * Self-Direction Subject Matter Knowledge Cultural, Physical & Behavioral Health Knowledge & Skills Ready for Life Youth Development Outcomes 5

6 Researchers Agree on What It Takes to Support Development
The National Research Council & Institute for Medicine list the following key features of positive youth development settings: Physical and psychological safety Appropriate structure Supportive relationships Opportunities to belong Positive social norms Support for efficacy and mattering Opportunities for skill-building Integration of family, school and community efforts - Community Programs to Promote Youth Development, 2002 6 6

7 Longitudinal Studies Confirm that these Supports Make a Difference
7 7

8 Providing these Supports Can Change the Odds
Gambone/Connell’s research suggests that if all young people got the supports they needed in early adolescence, the picture could change… from 4 in 10 doing well to 7 in 10 doing well* . 8 8

9 Systems for Quality Accountability Policies in Places
YPQA is part of state and county accountability policies: Cross sector (DHS& DOE) snapshots: Iowa, Washington, Arkansas Statewide 21st Century: Michigan, Maine, Minnesota, Rhode Island, New Mexico, Cities and Counties: Rochester, Detroit, Grand Rapids, Palm Beach `` ` l Rochester Washington* M i n n e a p o l i s Grand Rapids Chicago e t New York r o i t Iowa Indianapolis Rhode Island Georgetown Divide m b u s Columbus St. Louis Kentucky Oklahoma Nashville Austin West Palm Beach County

10 WHO WE ARE - 3 Nationals, 5+ States, 7 Localities
STATES (w/ participating localities) Iowa Linn County, N. Central Iowa, Polk County/Des Moines Kentucky Lexington, Louisville New York Broome, Onondaga, Orange & Rockland Counties Oklahoma Norman, Tulsa Rhode Island Central Falls, Newport, Pawtucket Washington State (Honorary QC member) LOCALITIES Austin, TX Georgetown Divide (Black Oak Mine), CA Columbus (Bartholomew County), IN Grand Rapids, MI Indianapolis, IN Nashville, TN St. Louis, MO NATIONAL PARTNERS The Forum for Youth Investment The Center For Youth Program Quality (formerly High/Scope Youth Development Group) AED National Training Institute for Community Youth Work 10

11 WHERE WE’RE GOING - Four Domains
The Quality Counts initiative is designed to help you maximize your capacity to improve the quality and reach of youth programs by taking concrete steps in four areas (domains) to ensure that you have:. Capacity to Assess & Improve Programs Strong Policy / Leadership Horsepower A Strong & Stable Program Base Capacity to Recruit, Train, Retain Workforce 11

12 Why bother paying attention to youth program quality?
Network leaders build awareness and communication Programs see their strengths and areas to improve, receive critical feedback, plus improve cmty networking and collaboration Funders gain an important metric Youth can have better program experiences 12 12

13 What is program quality?
??? inputs inputs outcomes inputs youth program This problem plagued school research for a long time too (really still does). E.g. NCLB Introduce structure versus process. We call process the POINT OF SERVICE (POS) Form B= structure; Form A = process. We heavily focus on process because we believe it is critical. But structure is important too--structure can be varied and different high quality programs can look very different in structure. Another way to say it: What do we want to see in high quality youth programs? 13

14 High POS Quality Relationship + Task + Increasing Complexity
Several ways to organize: Readin’ + ‘Ritin + ‘Rithmatic (old-school) Affect + Active Learning + Metacognition (Education) Relationships + Relevance + Rigor (Education 2.0) Relatedness + Autonomy + Competence (Psychology SDT) Safety + Belonging + Esteem (Psychology old-school) 8 Features of Positive Youth Development Settings (PYD) Just a small point for this crowd: task can be content (as traditionally conceived; sports, arts, activities) OR it can represent therapy or whatever the goals are for what you do with the kids in the POS. Relationship + Task + Increasing Complexity Content Therapeutic process

15 The Pyramid of Program Quality
Plan Make choices Engagement Reflect Lead and mentor Be in small groups Partner with adults Interaction Experience belonging Encouragement Reframing conflict Supportive Environment Skill building Session flow Active engagement Welcoming atmosphere Psychological and emotional safety Safe Environment Program space and furniture Emergency procedures Healthy food and drinks Physically safe environment Youth Voice and Governance Professional Learning Community

16 What is the Youth PQA? 1. A validated instrument
designed to assess the quality of youth programs and identify staff training needs. 2. A set of items that measures youth access to key developmental experiences. 3. A tool which produces scores that can be used for comparison and assessment of progress over time. The Youth PQA represents widely shared ideas about how programs can best promote youth development and learning. The structure of the pyramid above (which has parallels to Maslow’s Hierarchy of Needs) shows common trends in scoring. Youth programs tend to have higher scores in the area of Safe Environment. They then taper off in the areas of Supportive Environment, Interaction and are often lowest in Engagement. So, the pyramid shape represents the way scores tend to look. The top two categories—Interaction and Engagement—while often the lowest scoring, are the most connected to positive youth outcomes.

17 The Youth PQA and Maslow
Actualization B-Needs Esteem Needs Engagement Interaction (youth-youth; peer community) Love/ Belonging Needs D-Needs Supportive Environment (adult-youth relationships) Safety Needs Safe Environment Maslow categories PQA Subscales (within pyramid)

18 Scores From Validation Studies
Program Offering Level Validation Study: Outside Observer (N=140 offerings) Self-Assessment Pilot (N=24 orgs) Safe environment 4.35 4.39 Supportive environment 3.75 4.16 Opportunities for interaction 3.11 3.73 Engagement 2.83 3.37 Organization Level (N=51 orgs) Youth centered policies and practices 3.92 3.20 High expectations for students and staff 3.86 3.90 Access 4.18

19 Youth PQA Domain Scores N= 735 offerings (all unique staff) in 180 organizations
So when we look at scores on the four domains this is what we see in nearly all samples. When we disaggregate to the item level (lowest level of the tool) we get more detail – but really too much. This is what many quality assessments give us: global ratings or the kitchen sink of items that include everything that anyone thinks is important. Neither provide very focused emphasis about what is most important across many different types of settings.

20 Contrasting Pedagogy Profiles N=599 offerings in 120 organizations
5 = occurred for everyone 3 = occurred for some 1 = did not occur 20

21 How we think about DDCI - People change not programs
Prochaska, J.O., & DiClemente, C.C. (1982). Transtheoretical therapy toward a more integrative model of change. Psychotherapy: Theory, Research and Practice, 19(3), Maintenance SAE System Accountability Environment PLC Professional Learning Community POS Point Of Service Repeat cycle Action Implementation & coaching Preparation Planning with data Contemplation QIS = a change initiative People change, not just organizations. Change is a big deal for people. This is called a transtheoretical model and is popular in health research & applications. Prochaska, J.O., & DiClemente, C.C. (1982). Transtheoretical therapy toward a more integrative model of change. Psychotherapy: Theory, Research and Practice, 19(3), Quality assessment Individual Change Model Organizational Context 8/13/2008 The Center for Youth Program Quality

22 Designing Quality Improvement Systems (QIS)
Mostly Mangers Managers and direct staff TOTs for quality assess, coaching, and youth work methods (f,g) System Capacity Use of on-line dashboards and training (d) External quality assessment (a) Quality Advisor (e) Coaching & Training External Quality Report with Norms Self-Assessed Quality Report Use of on-line dashboards and training (d) Targeted youth work methods training for direct staff (h) Program Staff Skill & Knowl This is a clearer representation of how the intervention is actually enacted. Appendix 1 provides a detailed description of each of the elements listed in the arrows. The bottom arrow represents the skill set of the manager and administrative decision-making to institutionalize the intervention in year 1. In year two, direct staff are brought in to the process based on the mangers year-1 decision making and team building with her/his staff. Self-assessment of Quality (b) Quality coaching by managers (i) Planning with Data (c) Phase 1: Readiness & Capacity Phase 2: Impact & Sustainability

23 Defining the Purpose of Your QIS
Lower Stakes Program Self-Assessment Rough data to get staff thinking and discussing program quality in the context of best practice Less time Less money Impact internal audiences (the creative middle) Higher Stakes External Assessment Precise data for internal and external audiences for evaluation, monitoring, accountability, improvement, reporting More time More money Impact internal and external audiences

24 Columbus Indiana Phase 1: Building Local Capacity
POS Point Of Service SAE System Accountability Environment PLC Professional Learning Community STEP 2a Self-assessment STEP 1 Decide to build system STEP 3 Plan for improvement STEP 4 Carry out plan STEP 5 Measure change STEP 2b External assessment After working with numerous networks, we developed this diagram which represents the path that a network or program goes through. Talk through slide (training workshops at bottom) Mention High/Scope eTools—online Youth PQA scores reporter and online PQA training—will be live in January. August 26 Youth PQA Basics January Ext Assessment October 8 Planning with Data Improvement Plan Annually Program SA Ext Assessment Observe-Reflection Planning with Data August 25 Quality Matters Presentation Opt Phase 2 Method Workshops Quality Coaching

25 Step 1: Pilot Sites Attend Training- Youth PQA Basics, External Assessor

26 Step 2: Pilots Select Self-Assessment Teams and Develop Schedule

27 Step 3: Self Assessment

28 Step 4: Submit Scores

29 High/Scope’s eTools Web Site
Your one stop shop for: Online training Intro to Youth PQA Intro to Scores Reporter Online assessment Submitting Youth PQA Scores Generate reports

30 Online PQA Scores Reporter
3 levels of users Youth Program Facilitators Youth Program Directors Network Users

31 Online PQA Scores Reporter Enter Data with mouse or keyboard

32 Online PQA Scores Reporter View & Print Reports
Your programs scores can automatically be compared to national norms

33 Online PQA Scores Reporter View & Print Reports

34 Step 5: Teams attend Planning with Data and create Program Improvement Plans

35 Step 6: Staff members work the plan! Repeat Youth PQA annually…

36 Does it Work? Findings from Several Samples
POS quality-outcomes findings: Supportive environment related to: Attendance Interaction related to: Interest in program Engagement related to: Sense of challenge, sense of growth, school-day reading, school-day suspension Note: No offerings get to high engagement without high support and high interaction Quality Improvement (YPQI) Findings Scores increase from pre to post Scores increase in the targeted areas more Management practices are related to quality change (Vision, Feedback, Continuity)

37 Purpose? Process? Pilot? Next Steps…
Questions about… Purpose? Process? Pilot? Next Steps…

38 Youth Program Quality Assessment

39 Structure of the Youth PQA
Form B Organizational Interview Ask questions, write, score (2 hours) Form A Observation Watch, write, score (3 hours) Program Offering 1 Program Offering 2 Organization Here’s the overview (more details coming): You will be completing one Form A and one Form B, but your Form A will involve as many of your program offerings as possible. A program offering is a group of youths and at least one adult who meet together 1 or more times for a purpose. For example, dance, art, math, girls group, etc. For Form B, you ask the questions printed in the tool (on the right facing pages), write the answers, and score based on those answers. Since you are doing self-assessment you can probably answer all of the questions on your own or together with your team. For Form A you have staff observe your different program offerings and gather anecdotal evidence in minute chunks. You then come together as a team, compile all of the information you have gathered from you observations, insert it into the tool (Form A) and score it. Program Offering 3 Program Offering 4

40 Sample indicator III. Interaction “subscale” “item” “indicator row”
III-L. Youth have opportunities to develop a sense of belonging. Note: Structured refers to the quality of being intentional, planned, and/or named; it does not refer to informal conversation. Indicators Supporting Evidence 1 Youth have no opportunities to get to know each other (beyond self-selected pairs or small cliques). 3 Youth have informal opportunities to get to know each other (e.g., youth engage in informal conversations before, during, or after session. 5 Youth have structured opportunities to get to know each other (e.g., there are team-building activities, introductions, personal updates, welcomes of new group members, icebreakers, and a variety of groupings for activities) “item” “indicator row” Here’s a sample indicator. The written evidence goes in the white space on the right, then the score goes in the box – 1, 3, or 5 (no 2s or 4s!). This sample demonstrates how the tool addresses a common positive youth development topic. It’s generally agreed that youth do better when they feel a sense of belonging. This slide shows one indicator we use to get at that idea. Notice how this converts an inner state (whether youth feel like they belong) to a measurable behavior (whether staff provide get to know you activities). Each booklet is a scale. To see how Form A breaks down, look at the last page (titled “Youth PQA Summary Sheet”). The boldface words are the subscales and the words labeled A, B, C, etc. are items. An item is a page. An item contains 2 to 6 indicators like the one on this slide. The Youth PQA consists of 2 scales (Form A & Form B); 7 subscales (4 in A, 3 in B) 30 items (18 in A, 12 in B); 103 indicator rows (60 in A, 43 in B)

41 What are the key quantities?
Indicator Scramble What are the key words? What are the key quantities? What are we counting?

42 Sample Anecdotes Youth sort of identified with program offering. III-L (3rd row) Kids sometimes taunt each other and try to annoy each other, but they seem to get along, too. I-A (1st row) Kevin dominated committee meetings he sat in on. Did better job of listening and acknowledging comments in the large group. III-O (1st row)

43 Observational Note Taking
Effective Ineffective Objective as possible Rich detail in snapshot form Focus on interactions between Staff and youth Youth and youth Youth and environment Allow time for interactions to reach completion State the outcome of interactions Who, what, when, where Quotes: what youth and staff say Lists of materials What you see in the room Anecdote can stand alone Subjective terms such as ‘good’ or ‘bad’ Rater’s opinions Assumptions about internal states: she felt angry; he did not get it Anecdotes are too vague; lack detail Lacking facts: what you see and hear Summary in the place of quotes Raters repeat what the indicator says For a 3 involving some positive and some negative, raters have one but not the other Anecdote does not fit the indicator Anecdote could support more than one score


45 Follow-up questions Only ask for items with question(s) in the evidence column. Ask the questions as they are printed. Form A items with at least one “Ask” indicators (24 total) I-C indicator rows 1-6 I-D indicator row 4 II-K indicator rows 1-4 III-L indicator row 1 III-N indicator rows 2-3 IV-P indicator rows 1-2 IV-Q indicator rows 1-2 IV-R indicator rows 2-4

46 Fit and Score (label practice)
Try to place each label in the correct row for each item. Look over the anecdotal evidence for each item and see how the anecdotes are a “fit” with the indicator rows. Score each indicator row. For each row, read through the levels of the indicator row and select the level - 1, 3, or 5 - that most closely agrees with your evidence. Pay close attention to words like some, most, and always. Start at level 5 to see the full description of necessary evidence and check agreement with the anecdotal evidence by looking to the lower levels.

47 More on Scoring the YPQA
Fitting data to the indicator rows can eventually happen in your head while you are observing. Always try to see multiple items in every interaction and cross-reference constantly. Look to a preponderance of evidence but favor higher scores. If you lack evidence with “fit,” collect more data.

48 Assessing POS Quality using Youth PQA Let’s Try It! Watch, Write, Score
Collecting objective anecdotal evidence Scoring rubrics Reliability Instructions In this activity you will watch a short (4 minute) video clip, take notes, and score a few indicators based on the video. Before you watch the video, get ready: Make sure you have your hardcopy of Form A of the Youth PQA in front of you for this activity. Make sure you have scrap paper and a pen or pencil handy Review items II-J, III-M, and IV-Q (you will be scoring one indicator from each) While you watch the video take notes by hand. Make sure your notes are as objective as possible. Try to capture actual quotes. 48

49 More on Self-Assessment
Self-assessment only works if program (direct) staff are involved. Try to get a good mix of program times and offerings. Assessment is different from evaluation. Assessment gives you the opportunity to see where you are, whereas evaluation suggests judgment. Remember, the conversation is the most important part of self-assessment.

50 Youth PQA Step by Step STEP Self- Assessment External Assessment
Team that includes as many frontline staff as possible 1. Select Assessor(s) Use anchored rater if possible When will data be collected? Who watches whom? 2. Plan the Data Collection Check time & schedule Make sure frontline staff knows you’re coming Staff observe each other in minute chunks & take notes 3. Collect Data 1-2 hr visit Take notes Ask follow-up questions Have a meeting. Go indicator-by-indicator, sharing evidence and coming to consensus on each score 4. Fit & Score Fit & score each indicator Enter indicator scores into Scores Reporter 5. Generate Report Use the report to make plans for items to work on 6. Plan for Improvement

51 Engage Survive Comply Avoid/Resist
How will you use this data to promote the right kinds of Accountability Behaviors? Focus Us Survive Engage Me Comply Avoid/Resist Low High Commitment

Download ppt "Center For Youth Program Quality"

Similar presentations

Ads by Google