Presentation on theme: "Center For Youth Program Quality"— Presentation transcript:
1 Center For Youth Program Quality Youth Program Quality AssessmentYouth PQA BasicsCenter For Youth Program Quality1
2 Agenda Welcome Opening: Intro to the Youth PQA Observational Note TakingFitting & ScoringBuilding Your PyramidSelf Assessment Q&A
3 The Quality Counts Initiative Why Quality Counts, Who We Are, & Where We’re Going3
4 WHY? Because Rhetoric and Reality Don’t Match The American DreamAll youth ready, every family and community supportive, each leader effective.The American RealityOnly 4 in 10 youth ready, only 1 in 3 youth supported, too few leaders effective.The American DilemmaFragmentation, complacency, and low expectations of youth, communities and leadersThe Ready by 21™ ChallengeChange the odds for youth by changing the way we do business44
5 Ready for Work Youth Employment Outcomes …There is Increasing Agreement on Skills Needed for the 21st Century…Ready for WorkYouth Employment OutcomesReady for CollegeAcademic OutcomesSpecific Vocational Knowledge & Skills21st CenturySkills & ContentInformation & Media Literacy * Creativity * Intellectual Curiosity * Critical & Systems Thinking * Accountability and Adaptability * Communication * Problem Solving * Interpersonal Skills * Social Responsibility * Financial Literacy * Global Awareness * Civic Literacy * Self-DirectionSubjectMatterKnowledgeCultural, Physical & Behavioral Health Knowledge & SkillsReady for LifeYouth Development Outcomes5
6 Researchers Agree on What It Takes to Support Development The National Research Council & Institute for Medicine list the following key features of positive youth development settings:Physical and psychological safetyAppropriate structureSupportive relationshipsOpportunities to belongPositive social normsSupport for efficacy and matteringOpportunities for skill-buildingIntegration of family, school and community efforts- Community Programs to Promote Youth Development, 200266
7 Longitudinal Studies Confirm that these Supports Make a Difference 77
8 Providing these Supports Can Change the Odds Gambone/Connell’s research suggests that if all young people got the supports they needed in early adolescence, the picture could change…from 4 in 10 doing wellto 7 in 10 doing well*.88
9 Systems for Quality Accountability Policies in Places YPQA is part of state and county accountability policies:Cross sector (DHS& DOE) snapshots: Iowa, Washington, ArkansasStatewide 21st Century: Michigan, Maine, Minnesota, Rhode Island, New Mexico,Cities and Counties: Rochester, Detroit, Grand Rapids, Palm Beach`` `lRochesterWashington*MinneapolisGrand RapidsChicagoetNew YorkroitIowaIndianapolisRhode IslandGeorgetownDividembusColumbusSt. LouisKentuckyOklahomaNashvilleAustinWest Palm Beach County
10 WHO WE ARE - 3 Nationals, 5+ States, 7 Localities STATES (w/ participating localities)IowaLinn County, N. Central Iowa, Polk County/Des MoinesKentuckyLexington, LouisvilleNew YorkBroome, Onondaga, Orange & Rockland CountiesOklahomaNorman, TulsaRhode IslandCentral Falls, Newport, PawtucketWashington State(Honorary QC member)LOCALITIESAustin, TXGeorgetown Divide (Black Oak Mine), CAColumbus (Bartholomew County), INGrand Rapids, MIIndianapolis, INNashville, TNSt. Louis, MONATIONAL PARTNERSThe Forum for Youth InvestmentThe Center For Youth Program Quality (formerly High/Scope Youth Development Group)AED National Training Institute for Community Youth Work10
11 WHERE WE’RE GOING - Four Domains The Quality Counts initiative is designed to help you maximize your capacity to improve the quality and reach of youth programs by taking concrete steps in four areas (domains) to ensure that you have:.Capacity to Assess & Improve ProgramsStrong Policy / Leadership HorsepowerA Strong & Stable Program BaseCapacity to Recruit, Train, Retain Workforce11
12 Why bother paying attention to youth program quality? Network leaders build awareness and communicationPrograms see their strengths and areas to improve, receive critical feedback, plus improve cmty networking and collaborationFunders gain an important metricYouth can have better program experiences1212
13 What is program quality? ???inputsinputsoutcomesinputsyouth programThis problem plagued school research for a long time too (really still does). E.g. NCLBIntroduce structure versus process. We call process the POINT OF SERVICE (POS)Form B= structure; Form A = process. We heavily focus on process because we believe it is critical. But structure is important too--structure can be varied and different high quality programs can look very different in structure.Another way to say it: What do we want to see in high quality youth programs?13
14 High POS Quality Relationship + Task + Increasing Complexity Several ways to organize:Readin’ + ‘Ritin + ‘Rithmatic (old-school)Affect + Active Learning + Metacognition (Education)Relationships + Relevance + Rigor (Education 2.0)Relatedness + Autonomy + Competence (Psychology SDT)Safety + Belonging + Esteem (Psychology old-school)8 Features of Positive Youth Development Settings (PYD)Just a small point for this crowd: task can be content (as traditionally conceived; sports, arts, activities) OR it can represent therapy or whatever the goals are for what you do with the kids in the POS.Relationship + Task + Increasing ComplexityContentTherapeutic process
15 The Pyramid of Program Quality PlanMake choicesEngagementReflectLead and mentorBe in small groupsPartner with adultsInteractionExperience belongingEncouragementReframing conflictSupportive EnvironmentSkill buildingSession flowActive engagementWelcoming atmospherePsychological and emotional safetySafe EnvironmentProgram space and furnitureEmergency proceduresHealthy food and drinksPhysically safe environmentYouth Voice and GovernanceProfessional Learning Community
16 What is the Youth PQA? 1. A validated instrument designed to assess the qualityof youth programs and identifystaff training needs.2. A set of items that measures youth access to key developmental experiences.3. A tool which produces scores that can be used for comparison and assessment of progress over time.The Youth PQA represents widely shared ideas about how programs can best promote youth development and learning. The structure of the pyramid above (which has parallels to Maslow’s Hierarchy of Needs) shows common trends in scoring. Youth programs tend to have higher scores in the area of Safe Environment. They then taper off in the areas of Supportive Environment, Interaction and are often lowest in Engagement. So, the pyramid shape represents the way scores tend to look. The top two categories—Interaction and Engagement—while often the lowest scoring, are the most connected to positive youth outcomes.
18 Scores From Validation Studies Program Offering LevelValidation Study: Outside Observer(N=140 offerings)Self-AssessmentPilot(N=24 orgs)Safe environment4.354.39Supportive environment3.754.16Opportunities for interaction3.113.73Engagement2.833.37Organization Level(N=51 orgs)Youth centered policies and practices3.923.20High expectations for students and staff3.863.90Access4.18
19 Youth PQA Domain Scores N= 735 offerings (all unique staff) in 180 organizations So when we look at scores on the four domains this is what we see in nearly all samples. When we disaggregate to the item level (lowest level of the tool) we get more detail – but really too much. This is what many quality assessments give us: global ratings or the kitchen sink of items that include everything that anyone thinks is important. Neither provide very focused emphasis about what is most important across many different types of settings.
20 Contrasting Pedagogy Profiles N=599 offerings in 120 organizations 5 = occurred for everyone3 = occurred for some1 = did not occur20
21 How we think about DDCI - People change not programs Prochaska, J.O., & DiClemente, C.C. (1982). Transtheoretical therapy toward a more integrative model of change. Psychotherapy: Theory, Research and Practice, 19(3),MaintenanceSAESystem AccountabilityEnvironmentPLCProfessional Learning CommunityPOSPoint Of ServiceRepeat cycleActionImplementation & coachingPreparationPlanning with dataContemplationQIS = a change initiativePeople change, not just organizations.Change is a big deal for people.This is called a transtheoretical model and is popular in health research & applications.Prochaska, J.O., & DiClemente, C.C. (1982). Transtheoretical therapy toward a more integrative model of change. Psychotherapy: Theory, Research and Practice, 19(3),Quality assessmentIndividual Change ModelOrganizational Context8/13/2008The Center for Youth Program Quality
22 Designing Quality Improvement Systems (QIS) Mostly MangersManagers and direct staffTOTs for quality assess, coaching, and youth work methods (f,g)System CapacityUse of on-line dashboards and training (d)External quality assessment (a)Quality Advisor (e)Coaching & TrainingExternal Quality Report with NormsSelf-Assessed Quality ReportUse of on-line dashboards and training (d)Targeted youth work methods training for direct staff (h)Program Staff Skill & KnowlThis is a clearer representation of how the intervention is actually enacted. Appendix 1 provides a detailed description of each of the elements listed in the arrows. The bottom arrow represents the skill set of the manager and administrative decision-making to institutionalize the intervention in year 1. In year two, direct staff are brought in to the process based on the mangers year-1 decision making and team building with her/his staff.Self-assessment of Quality (b)Quality coaching by managers (i)Planning with Data (c)Phase 1: Readiness &CapacityPhase 2: Impact & Sustainability
23 Defining the Purpose of Your QIS Lower StakesProgramSelf-AssessmentRough data to get staff thinking and discussing program quality in the context of best practiceLess timeLess moneyImpact internalaudiences(the creative middle)Higher StakesExternal AssessmentPrecise data for internal and external audiences for evaluation, monitoring, accountability, improvement, reportingMore timeMore money Impact internal andexternal audiences
24 Columbus Indiana Phase 1: Building Local Capacity POS Point Of ServiceSAE System Accountability EnvironmentPLC Professional Learning CommunitySTEP 2aSelf-assessmentSTEP 1Decide to build systemSTEP 3Plan for improvementSTEP 4Carry out planSTEP 5Measure changeSTEP 2bExternal assessmentAfter working with numerous networks, we developed this diagram which represents the path that a network or program goes through.Talk through slide (training workshops at bottom)Mention High/Scope eTools—online Youth PQA scores reporter and online PQA training—will be live in January.August 26Youth PQA BasicsJanuary Ext AssessmentOctober 8Planning with DataImprovement PlanAnnuallyProgram SAExt AssessmentObserve-ReflectionPlanning with DataAugust 25Quality MattersPresentationOpt Phase 2Method WorkshopsQuality Coaching
34 Step 5: Teams attend Planning with Data and create Program Improvement Plans
35 Step 6: Staff members work the plan! Repeat Youth PQA annually…
36 Does it Work? Findings from Several Samples POS quality-outcomes findings:Supportive environment related to: AttendanceInteraction related to: Interest in programEngagement related to: Sense of challenge, sense of growth, school-day reading, school-day suspensionNote: No offerings get to high engagement without high support and high interactionQuality Improvement (YPQI) FindingsScores increase from pre to postScores increase in the targeted areas moreManagement practices are related to quality change (Vision, Feedback, Continuity)
37 Purpose? Process? Pilot? Next Steps… Questions about…Purpose? Process? Pilot? Next Steps…
39 Structure of the Youth PQA Form B Organizational InterviewAsk questions, write, score(2 hours)Form A ObservationWatch, write, score (3 hours)Program Offering 1Program Offering 2OrganizationHere’s the overview (more details coming):You will be completing one Form A and one Form B, but your Form A will involve as many of your program offerings as possible. A program offering is a group of youths and at least one adult who meet together 1 or more times for a purpose. For example, dance, art, math, girls group, etc.For Form B, you ask the questions printed in the tool (on the right facing pages), write the answers, and score based on those answers. Since you are doing self-assessment you can probably answer all of the questions on your own or together with your team.For Form A you have staff observe your different program offerings and gather anecdotal evidence in minute chunks. You then come together as a team, compile all of the information you have gathered from you observations, insert it into the tool (Form A) and score it.Program Offering 3Program Offering 4
40 Sample indicator III. Interaction “subscale” “item” “indicator row” III-L. Youth have opportunities to develop a sense of belonging.Note: Structured refers to the quality of being intentional, planned, and/or named; it does not refer to informal conversation.IndicatorsSupporting Evidence1 Youth have no opportunities to get to know each other (beyond self-selected pairs or small cliques).3 Youth have informal opportunities to get to know each other (e.g., youth engage in informal conversations before, during, or after session.5 Youth have structured opportunities to get to know each other (e.g., there are team-building activities, introductions, personal updates, welcomes of new group members, icebreakers, and a variety of groupings for activities)“item”“indicator row”Here’s a sample indicator. The written evidence goes in the white space on the right, then the score goes in the box – 1, 3, or 5 (no 2s or 4s!).This sample demonstrates how the tool addresses a common positive youth development topic. It’s generally agreed that youth do better when they feel a sense of belonging. This slide shows one indicator we use to get at that idea. Notice how this converts an inner state (whether youth feel like they belong) to a measurable behavior (whether staff provide get to know you activities).Each booklet is a scale. To see how Form A breaks down, look at the last page (titled “Youth PQA Summary Sheet”). The boldface words are the subscales and the words labeled A, B, C, etc. are items. An item is a page. An item contains 2 to 6 indicators like the one on this slide.The Youth PQA consists of2 scales (Form A & Form B); 7 subscales (4 in A, 3 in B)30 items (18 in A, 12 in B); 103 indicator rows (60 in A, 43 in B)
41 What are the key quantities? Indicator ScrambleWhat are the key words?What are the key quantities?What are we counting?
42 Sample AnecdotesYouth sort of identified with program offering. III-L (3rd row)Kids sometimes taunt each other and try to annoy each other, but they seem to get along, too. I-A (1st row)Kevin dominated committee meetings he sat in on. Did better job of listening and acknowledging comments in the large group. III-O (1st row)
43 Observational Note Taking EffectiveIneffectiveObjective as possibleRich detail in snapshot formFocus on interactions betweenStaff and youthYouth and youthYouth and environmentAllow time for interactions to reach completionState the outcome of interactionsWho, what, when, whereQuotes: what youth and staff sayLists of materialsWhat you see in the roomAnecdote can stand aloneSubjective terms such as ‘good’ or ‘bad’Rater’s opinionsAssumptions about internal states: she felt angry; he did not get itAnecdotes are too vague; lack detailLacking facts: what you see and hearSummary in the place of quotesRaters repeat what the indicator saysFor a 3 involving some positive and some negative, raters have one but not the otherAnecdote does not fit the indicatorAnecdote could support more than one score
45 Follow-up questionsOnly ask for items with question(s) in the evidence column.Ask the questions as they are printed.Form A items with at least one “Ask” indicators (24 total)I-C indicator rows 1-6I-D indicator row 4II-K indicator rows 1-4III-L indicator row 1III-N indicator rows 2-3IV-P indicator rows 1-2IV-Q indicator rows 1-2IV-R indicator rows 2-4
46 Fit and Score (label practice) Try to place each label in the correct row for each item. Look over the anecdotal evidence for each item and see how the anecdotes are a “fit” with the indicator rows.Score each indicator row. For each row, read through the levels of the indicator row and select the level - 1, 3, or 5 - that most closely agrees with your evidence. Pay close attention to words like some, most, and always. Start at level 5 to see the full description of necessary evidence and check agreement with the anecdotal evidence by looking to the lower levels.
47 More on Scoring the YPQA Fitting data to the indicator rows can eventually happen in your head while you are observing.Always try to see multiple items in every interaction and cross-reference constantly.Look to a preponderance of evidence but favor higher scores.If you lack evidence with “fit,” collect more data.
48 Assessing POS Quality using Youth PQA Let’s Try It! Watch, Write, Score Collecting objective anecdotal evidenceScoring rubricsReliabilityInstructionsIn this activity you will watch a short (4 minute) video clip, take notes, and score a few indicators based on the video.Before you watch the video, get ready:Make sure you have your hardcopy of Form A of the Youth PQA in front of you for this activity.Make sure you have scrap paper and a pen or pencil handyReview items II-J, III-M, and IV-Q (you will be scoring one indicator from each)While you watch the video take notes by hand. Make sure your notes are as objective as possible. Try to capture actual quotes.48
49 More on Self-Assessment Self-assessment only works if program (direct) staff are involved.Try to get a good mix of program times and offerings.Assessment is different from evaluation. Assessment gives you the opportunity to see where you are, whereas evaluation suggests judgment.Remember, the conversation is the most important part of self-assessment.
50 Youth PQA Step by Step STEP Self- Assessment External Assessment Team that includes as many frontline staff as possible1. Select Assessor(s)Use anchored rater if possibleWhen will data be collected? Who watches whom?2. Plan the Data CollectionCheck time & schedule Make sure frontline staff knows you’re comingStaff observe each other in minute chunks & take notes3. Collect Data1-2 hr visitTake notesAsk follow-up questionsHave a meeting. Go indicator-by-indicator, sharing evidence and coming to consensus on each score4. Fit & ScoreFit & score each indicatorEnter indicator scores into Scores Reporter5. Generate ReportUse the report to make plans for items to work on6. Plan for Improvement
51 Engage Survive Comply Avoid/Resist How will you use this data to promote the right kinds of Accountability Behaviors?FocusUsSurviveEngageMeComplyAvoid/ResistLowHighCommitment