1. Center For Youth Program Quality
Youth Program Quality Assessment
Youth PQA Basics
Center For Youth Program Quality 1

2. Agenda Welcome Opening: Intro to the Youth PQA
Observational Note Taking
Fitting & Scoring
Building Your Pyramid
Self Assessment Q&A

3. The Quality Counts Initiative
Why Quality Counts, Who We Are, & Where We’re Going 3

4. WHY? Because Rhetoric and Reality Don’t Match
The American Dream
All youth ready, every family and community supportive, each leader effective.
The American Reality
Only 4 in 10 youth ready, only 1 in 3 youth supported, too few leaders effective.
The American Dilemma
Fragmentation, complacency, and low expectations of youth, communities and leaders
The Ready by 21™ Challenge
Change the odds for youth by changing the way we do business 4 4

5. Ready for Work Youth Employment Outcomes
…There is Increasing Agreement on Skills Needed for the 21st Century…
Ready for Work
Youth Employment Outcomes
Ready for College
Academic Outcomes
Specific Vocational Knowledge & Skills
21st Century
Skills & Content
Information & Media Literacy * Creativity * Intellectual Curiosity * Critical & Systems Thinking * Accountability and Adaptability * Communication * Problem Solving * Interpersonal Skills * Social Responsibility * Financial Literacy * Global Awareness * Civic Literacy * Self-Direction
Subject
Matter
Knowledge
Cultural, Physical & Behavioral Health Knowledge & Skills
Ready for Life
Youth Development Outcomes 5

6. Researchers Agree on What It Takes to Support Development
The National Research Council & Institute for Medicine list the following key features of positive youth development settings:
Physical and psychological safety
Appropriate structure
Supportive relationships
Opportunities to belong
Positive social norms
Support for efficacy and mattering
Opportunities for skill-building
Integration of family, school and community efforts
- Community Programs to Promote Youth Development, 2002 6 6

7. Longitudinal Studies Confirm that these Supports Make a Difference7 7

8. Providing these Supports Can Change the Odds
Gambone/Connell’s research suggests that if all young people got the supports they needed in early adolescence, the picture could change…
from 4 in 10 doing well
to 7 in 10 doing well* . 8 8

9. Systems for Quality Accountability Policies in Places
YPQA is part of state and county accountability policies:
Cross sector (DHS& DOE) snapshots: Iowa, Washington, Arkansas
Statewide 21st Century: Michigan, Maine, Minnesota, Rhode Island, New Mexico,
Cities and Counties: Rochester, Detroit, Grand Rapids, Palm Beach
`` ` l
Rochester
Washington* M i n n e a p o l i s
Grand Rapids
Chicago e t
New York r o i t
Iowa
Indianapolis
Rhode Island
Georgetown
Divide m b u s
Columbus
St. Louis
Kentucky
Oklahoma
Nashville
Austin
West Palm Beach County

10. WHO WE ARE - 3 Nationals, 5+ States, 7 Localities
STATES (w/ participating localities)
Iowa
Linn County, N. Central Iowa, Polk County/Des Moines
Kentucky
Lexington, Louisville
New York
Broome, Onondaga, Orange & Rockland Counties
Oklahoma
Norman, Tulsa
Rhode Island
Central Falls, Newport, Pawtucket
Washington State
(Honorary QC member)
LOCALITIES
Austin, TX
Georgetown Divide (Black Oak Mine), CA
Columbus (Bartholomew County), IN
Grand Rapids, MI
Indianapolis, IN
Nashville, TN
St. Louis, MO
NATIONAL PARTNERS
The Forum for Youth Investment
The Center For Youth Program Quality (formerly High/Scope Youth Development Group)
AED National Training Institute for Community Youth Work 10

11. WHERE WE’RE GOING - Four Domains
The Quality Counts initiative is designed to help you maximize your capacity to improve the quality and reach of youth programs by taking concrete steps in four areas (domains) to ensure that you have:.
Capacity to Assess & Improve Programs
Strong Policy / Leadership Horsepower
A Strong & Stable Program Base
Capacity to Recruit, Train, Retain Workforce 11

12. Why bother paying attention to youth program quality?
Network leaders build awareness and communication
Programs see their strengths and areas to improve, receive critical feedback, plus improve cmty networking and collaboration
Funders gain an important metric
Youth can have better program experiences 12 12

13. What is program quality?
???
inputs
inputs
outcomes
inputs
youth program
This problem plagued school research for a long time too (really still does). E.g. NCLB
Introduce structure versus process. We call process the POINT OF SERVICE (POS)
Form B= structure; Form A = process. We heavily focus on process because we believe it is critical. But structure is important too--structure can be varied and different high quality programs can look very different in structure.
Another way to say it: What do we want to see in high quality youth programs? 13

14. High POS Quality Relationship + Task + Increasing Complexity
Several ways to organize:
Readin’ + ‘Ritin + ‘Rithmatic (old-school)
Affect + Active Learning + Metacognition (Education)
Relationships + Relevance + Rigor (Education 2.0)
Relatedness + Autonomy + Competence (Psychology SDT)
Safety + Belonging + Esteem (Psychology old-school)
8 Features of Positive Youth Development Settings (PYD)
Just a small point for this crowd: task can be content (as traditionally conceived; sports, arts, activities) OR it can represent therapy or whatever the goals are for what you do with the kids in the POS.
Relationship + Task + Increasing Complexity
Content
Therapeutic process

15. The Pyramid of Program Quality
Plan
Make choices
Engagement
Reflect
Lead and mentor
Be in small groups
Partner with adults
Interaction
Experience belonging
Encouragement
Reframing conflict
Supportive Environment
Skill building
Session flow
Active engagement
Welcoming atmosphere
Psychological and emotional safety
Safe Environment
Program space and furniture
Emergency procedures
Healthy food and drinks
Physically safe environment
Youth Voice and Governance
Professional Learning Community

16. What is the Youth PQA? 1. A validated instrument
designed to assess the quality
of youth programs and identify
staff training needs.
2. A set of items that measures youth access to key developmental experiences.
3. A tool which produces scores that can be used for comparison and assessment of progress over time.
The Youth PQA represents widely shared ideas about how programs can best promote youth development and learning. The structure of the pyramid above (which has parallels to Maslow’s Hierarchy of Needs) shows common trends in scoring. Youth programs tend to have higher scores in the area of Safe Environment. They then taper off in the areas of Supportive Environment, Interaction and are often lowest in Engagement. So, the pyramid shape represents the way scores tend to look. The top two categories—Interaction and Engagement—while often the lowest scoring, are the most connected to positive youth outcomes.

17. The Youth PQA and Maslow
Actualization
B-Needs
Esteem Needs
Engagement
Interaction (youth-youth; peer community)
Love/ Belonging Needs
D-Needs
Supportive Environment (adult-youth relationships)
Safety Needs
Safe Environment
Maslow categories
PQA Subscales (within pyramid)

18. Scores From Validation Studies
Program Offering Level
Validation Study: Outside Observer
(N=140 offerings)
Self-Assessment
Pilot
(N=24 orgs)
Safe environment
4.35
4.39
Supportive environment
3.75
4.16
Opportunities for interaction
3.11
3.73
Engagement
2.83
3.37
Organization Level
(N=51 orgs)
Youth centered policies and practices
3.92
3.20
High expectations for students and staff
3.86
3.90
Access
4.18

19. Youth PQA Domain Scores N= 735 offerings (all unique staff) in 180 organizations
So when we look at scores on the four domains this is what we see in nearly all samples. When we disaggregate to the item level (lowest level of the tool) we get more detail – but really too much. This is what many quality assessments give us: global ratings or the kitchen sink of items that include everything that anyone thinks is important. Neither provide very focused emphasis about what is most important across many different types of settings.

20. Contrasting Pedagogy Profiles N=599 offerings in 120 organizations
5 = occurred for everyone
3 = occurred for some
1 = did not occur 20

21. How we think about DDCI - People change not programs
Prochaska, J.O., & DiClemente, C.C. (1982). Transtheoretical therapy toward a more integrative model of change. Psychotherapy: Theory, Research and Practice, 19(3),
Maintenance
SAE
System Accountability
Environment
PLC
Professional Learning Community
POS
Point Of Service
Repeat cycle
Action
Implementation & coaching
Preparation
Planning with data
Contemplation
QIS = a change initiative
People change, not just organizations.
Change is a big deal for people.
This is called a transtheoretical model and is popular in health research & applications.
Prochaska, J.O., & DiClemente, C.C. (1982). Transtheoretical therapy toward a more integrative model of change. Psychotherapy: Theory, Research and Practice, 19(3),
Quality assessment
Individual Change Model
Organizational Context
8/13/2008
The Center for Youth Program Quality

22. Designing Quality Improvement Systems (QIS)
Mostly Mangers
Managers and direct staff
TOTs for quality assess, coaching, and youth work methods (f,g)
System Capacity
Use of on-line dashboards and training (d)
External quality assessment (a)
Quality Advisor (e)
Coaching & Training
External Quality Report with Norms
Self-Assessed Quality Report
Use of on-line dashboards and training (d)
Targeted youth work methods training for direct staff (h)
Program Staff Skill & Knowl
This is a clearer representation of how the intervention is actually enacted. Appendix 1 provides a detailed description of each of the elements listed in the arrows. The bottom arrow represents the skill set of the manager and administrative decision-making to institutionalize the intervention in year 1. In year two, direct staff are brought in to the process based on the mangers year-1 decision making and team building with her/his staff.
Self-assessment of Quality (b)
Quality coaching by managers (i)
Planning with Data (c)
Phase 1: Readiness &
Capacity
Phase 2: Impact & Sustainability

23. Defining the Purpose of Your QIS
Lower Stakes
Program
Self-Assessment
Rough data to get staff thinking and discussing program quality in the context of best practice
Less time
Less money
Impact internal
audiences
(the creative middle)
Higher Stakes
External Assessment
Precise data for internal and external audiences for evaluation, monitoring, accountability, improvement, reporting
More time
More money Impact internal and
external audiences

24. Columbus Indiana Phase 1: Building Local Capacity
POS Point Of Service
SAE System Accountability Environment
PLC Professional Learning Community
STEP 2a
Self-assessment
STEP 1
Decide to build system
STEP 3
Plan for improvement
STEP 4
Carry out plan
STEP 5
Measure change
STEP 2b
External assessment
After working with numerous networks, we developed this diagram which represents the path that a network or program goes through.
Talk through slide (training workshops at bottom)
Mention High/Scope eTools—online Youth PQA scores reporter and online PQA training—will be live in January.
August 26
Youth PQA Basics
January Ext Assessment
October 8
Planning with Data
Improvement Plan
Annually
Program SA
Ext Assessment
Observe-Reflection
Planning with Data
August 25
Quality Matters
Presentation
Opt Phase 2
Method Workshops
Quality Coaching

25. Step 1: Pilot Sites Attend Training- Youth PQA Basics, External Assessor

26. Step 2: Pilots Select Self-Assessment Teams and Develop Schedule

27. Step 3: Self Assessment

28. Step 4: Submit Scores

29. High/Scope’s eTools Web Site etools.highscope.org
Your one stop shop for:
Online training
Intro to Youth PQA
Intro to Scores Reporter
Online assessment
Submitting Youth PQA Scores
Generate reports

30. Online PQA Scores Reporter etools.highscope.org
3 levels of users
Youth Program Facilitators
Youth Program Directors
Network Users

31. Online PQA Scores Reporter Enter Data with mouse or keyboard

32. Online PQA Scores Reporter View & Print Reports
Your programs scores can automatically be compared to national norms

33. Online PQA Scores Reporter View & Print Reports

34. Step 5: Teams attend Planning with Data and create Program Improvement Plans

35. Step 6: Staff members work the plan! Repeat Youth PQA annually…

36. Does it Work? Findings from Several Samples
POS quality-outcomes findings:
Supportive environment related to: Attendance
Interaction related to: Interest in program
Engagement related to: Sense of challenge, sense of growth, school-day reading, school-day suspension
Note: No offerings get to high engagement without high support and high interaction
Quality Improvement (YPQI) Findings
Scores increase from pre to post
Scores increase in the targeted areas more
Management practices are related to quality change (Vision, Feedback, Continuity)

37. Purpose? Process? Pilot? Next Steps…
Questions about…
Purpose? Process? Pilot? Next Steps…

38. Youth Program Quality Assessment

39. Structure of the Youth PQA
Form B Organizational Interview
Ask questions, write, score
(2 hours)
Form A Observation
Watch, write, score (3 hours)
Program Offering 1
Program Offering 2
Organization
Here’s the overview (more details coming):
You will be completing one Form A and one Form B, but your Form A will involve as many of your program offerings as possible. A program offering is a group of youths and at least one adult who meet together 1 or more times for a purpose. For example, dance, art, math, girls group, etc.
For Form B, you ask the questions printed in the tool (on the right facing pages), write the answers, and score based on those answers. Since you are doing self-assessment you can probably answer all of the questions on your own or together with your team.
For Form A you have staff observe your different program offerings and gather anecdotal evidence in minute chunks. You then come together as a team, compile all of the information you have gathered from you observations, insert it into the tool (Form A) and score it.
Program Offering 3
Program Offering 4

40. Sample indicator III. Interaction “subscale” “item” “indicator row”
III-L. Youth have opportunities to develop a sense of belonging.
Note: Structured refers to the quality of being intentional, planned, and/or named; it does not refer to informal conversation.
Indicators
Supporting Evidence
1 Youth have no opportunities to get to know each other (beyond self-selected pairs or small cliques).
3 Youth have informal opportunities to get to know each other (e.g., youth engage in informal conversations before, during, or after session.
5 Youth have structured opportunities to get to know each other (e.g., there are team-building activities, introductions, personal updates, welcomes of new group members, icebreakers, and a variety of groupings for activities)
“item”
“indicator row”
Here’s a sample indicator. The written evidence goes in the white space on the right, then the score goes in the box – 1, 3, or 5 (no 2s or 4s!).
This sample demonstrates how the tool addresses a common positive youth development topic. It’s generally agreed that youth do better when they feel a sense of belonging. This slide shows one indicator we use to get at that idea. Notice how this converts an inner state (whether youth feel like they belong) to a measurable behavior (whether staff provide get to know you activities).
Each booklet is a scale. To see how Form A breaks down, look at the last page (titled “Youth PQA Summary Sheet”). The boldface words are the subscales and the words labeled A, B, C, etc. are items. An item is a page. An item contains 2 to 6 indicators like the one on this slide.
The Youth PQA consists of
2 scales (Form A & Form B); 7 subscales (4 in A, 3 in B)
30 items (18 in A, 12 in B); 103 indicator rows (60 in A, 43 in B)

41. What are the key quantities?
Indicator Scramble
What are the key words?
What are the key quantities?
What are we counting?

42. Sample Anecdotes
Youth sort of identified with program offering. III-L (3rd row)
Kids sometimes taunt each other and try to annoy each other, but they seem to get along, too. I-A (1st row)
Kevin dominated committee meetings he sat in on. Did better job of listening and acknowledging comments in the large group. III-O (1st row)

43. Observational Note Taking
Effective
Ineffective
Objective as possible
Rich detail in snapshot form
Focus on interactions between
Staff and youth
Youth and youth
Youth and environment
Allow time for interactions to reach completion
State the outcome of interactions
Who, what, when, where
Quotes: what youth and staff say
Lists of materials
What you see in the room
Anecdote can stand alone
Subjective terms such as ‘good’ or ‘bad’
Rater’s opinions
Assumptions about internal states: she felt angry; he did not get it
Anecdotes are too vague; lack detail
Lacking facts: what you see and hear
Summary in the place of quotes
Raters repeat what the indicator says
For a 3 involving some positive and some negative, raters have one but not the other
Anecdote does not fit the indicator
Anecdote could support more than one score

45. Follow-up questions
Only ask for items with question(s) in the evidence column.
Ask the questions as they are printed.
Form A items with at least one “Ask” indicators (24 total)
I-C indicator rows 1-6
I-D indicator row 4
II-K indicator rows 1-4
III-L indicator row 1
III-N indicator rows 2-3
IV-P indicator rows 1-2
IV-Q indicator rows 1-2
IV-R indicator rows 2-4

46. Fit and Score (label practice)
Try to place each label in the correct row for each item. Look over the anecdotal evidence for each item and see how the anecdotes are a “fit” with the indicator rows.
Score each indicator row. For each row, read through the levels of the indicator row and select the level - 1, 3, or 5 - that most closely agrees with your evidence. Pay close attention to words like some, most, and always. Start at level 5 to see the full description of necessary evidence and check agreement with the anecdotal evidence by looking to the lower levels.

47. More on Scoring the YPQA
Fitting data to the indicator rows can eventually happen in your head while you are observing.
Always try to see multiple items in every interaction and cross-reference constantly.
Look to a preponderance of evidence but favor higher scores.
If you lack evidence with “fit,” collect more data.

48. Assessing POS Quality using Youth PQA Let’s Try It! Watch, Write, Score
Collecting objective anecdotal evidence
Scoring rubrics
Reliability
Instructions
In this activity you will watch a short (4 minute) video clip, take notes, and score a few indicators based on the video.
Before you watch the video, get ready:
Make sure you have your hardcopy of Form A of the Youth PQA in front of you for this activity.
Make sure you have scrap paper and a pen or pencil handy
Review items II-J, III-M, and IV-Q (you will be scoring one indicator from each)
While you watch the video take notes by hand. Make sure your notes are as objective as possible. Try to capture actual quotes. 48

49. More on Self-Assessment
Self-assessment only works if program (direct) staff are involved.
Try to get a good mix of program times and offerings.
Assessment is different from evaluation. Assessment gives you the opportunity to see where you are, whereas evaluation suggests judgment.
Remember, the conversation is the most important part of self-assessment.

50. Youth PQA Step by Step STEP Self- Assessment External Assessment
Team that includes as many frontline staff as possible
1. Select Assessor(s)
Use anchored rater if possible
When will data be collected? Who watches whom?
2. Plan the Data Collection
Check time & schedule Make sure frontline staff knows you’re coming
Staff observe each other in minute chunks & take notes
3. Collect Data
1-2 hr visit
Take notes
Ask follow-up questions
Have a meeting. Go indicator-by-indicator, sharing evidence and coming to consensus on each score
4. Fit & Score
Fit & score each indicator
Enter indicator scores into Scores Reporter
5. Generate Report
Use the report to make plans for items to work on
6. Plan for Improvement

51. Engage Survive Comply Avoid/Resist
How will you use this data to promote the right kinds of Accountability Behaviors?
Focus Us
Survive
Engage Me
Comply
Avoid/Resist
Low
High
Commitment