High Quality Performance Measures What is Performance Measurement? What makes a high quality performance measure?

High Quality Performance Measures Learning Objectives By the end of this module, you will be able to: Describe basic performance measurement concepts Describe how to strengthen performance measures through: o Alignment with theory of change o Alignment of outputs and outcomes o Selection of a meaningful outcome 2

High Quality Performance Measures Module Overview Performance Measurement Concept and definitions Comparison to impact evaluation Strengthening Performance Measures through: Alignment with the Theory of Change Alignment of outputs and outcomes Selection of a meaningful outcome 3

High Quality Performance Measures What is Performance Measurement? Systematic process of regularly measuring: Outputs Amount of service provided (e.g. people served, products created, or programs developed) Outcomes Reflect the changes or benefits that occur as the result of the service Can reflect changes in individuals, organizations, communities, or the environment Address changes in attitudes/beliefs, knowledge/skills, behavior, or conditions 4

High Quality Performance Measures Performance Measurement and Impact Evaluation Similarities Systematic data collection Measurement of progress Collect data on: What took place (intervention) Outputs (amount of service) Outcomes (changes) 5

High Quality Performance Measures Performance Measurement and Impact Evaluation Differences: Key Areas Performance Measurement Impact Evaluation CausalityAssumes causality; does not prove theory of change Seeks to show causality and prove theory of change ImplementationTracks outputs and outcomes on a regular, ongoing basis May occur from time to time, but not on a regular, ongoing basis Time FocusShorter term focus; what can be measured within one year Longer term focus 6

High Quality Performance Measures Performance Measurement and Impact Evaluation (continued) Differences: Key Areas Performance Measurement Impact Evaluation DesignBalances rigor* with practicality; measurement implemented by program staff with limited resources and expertise Uses most rigorous evaluation design and methods that are right for program (e.g. quasi- experimental design) 7 *Capable of producing valid (accurate) results.

High Quality Performance Measures Learning Exercise Check the best answer Performance Measurement Impact Evaluation Both 1. Seeks to provide evidence of the existence of a causal relationship between the intervention and the observed outcomes, that is, it attempts to prove the theory of change 2. The information collected can be used for decision-making on programmatic improvement or expansion 3. Usually planned and implemented by program staff 4. Careful and systematic and quality data collection is very important 8

High Quality Performance Measures Learning Exercise Check the best answer Performance Measurement Impact Evaluation Both 1. Seeks to provide evidence of the existence of a causal relationship between the intervention and the observed outcomes, that is, it attempts to prove the theory of change X 2. The information collected can be used for decision-making on programmatic improvement or expansion X 3. Usually planned and implemented by program staff X 4. Careful and systematic and quality data collection is very important X 9

High Quality Performance Measures Ensuring Quality Performance Measure Intervention Output Outcome Align output and outcome Review for meaningful outcome Planning Theory of Change (ToC) Problem/Need Intervention Outcome Align elements within ToC 10

High Quality Performance Measures Performance Measurement and Alignment Alignment in Performance Measurement: Logical flow and connection between the different elements Alignment within the theory of change Alignment between output and outcome 11

High Quality Performance Measures Alignment within Theory of Change 12 Evidence Guides choice of intervention Supports cause-effect relationship Evidence Guides choice of intervention Supports cause-effect relationship Community Need/Problem Specific Intervention cause Specific Intervention cause Intended Outcome effect Intended Outcome effect A theory of change defines a cause-and-effect relationship between a specific intervention and an intended outcome.

High Quality Performance Measures Which outcome is aligned with the intervention? NEED: Documented high rates of chronic absence from school lead to other problems, e.g. low school engagement, poor academic performance (Social Work Practices, 2010). INTERVENTION: Weekly individual and group mentoring to 6 th grade students with attendance problems over 10 months. Focus on promoting re-engagement with school through social development and leadership skills. OUTCOME: ________ Students participate in the mentoring program Students increase academic performance Students improve school attendance Mentoring Example*: Alignment within Theory of Change *Descriptions in examples are abbreviated.

High Quality Performance Measures Which outcome is aligned with the intervention? NEED: Documented high rates of chronic absence from school lead to other problems, e.g. low school engagement, poor academic performance (Social Work Practices, 2010). INTERVENTION: Weekly individual and group mentoring to 6 th grade students with attendance problems over 10 months. Focus on promoting re-engagement with school through social development and leadership skills. OUTCOME: ________ Students participate in the mentoring program Students increase academic performance Students improve school attendance Mentoring Example: Alignment within Theory of Change

High Quality Performance Measures NEED: Documented high rates of chronic absence from school lead to other problems, e.g. low school engagement, poor academic performance (Social Work Practices, 2010). INTERVENTION: Weekly individual and group mentoring to 6 th grade students with attendance problems over 10 months. Focus on promoting re- engagement with school through social development and leadership skills. OUTCOME: Students improve school attendance. Mentoring Example: Alignment within Theory of Change

High Quality Performance Measures NEED : 35% of young veterans (18-24 year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications and experience in professional settings. Intervention 1.Veterans participate in a drop-in job clinic offering job fairs, resume writing classes and skills in internet job searches. 2.Veterans supported in completing training programs through assistance in finding appropriate programs and financial aid, with tutoring resources, in internship placement. 3. Military families receive job placement counseling. Outcome a.Veterans report increased confidence in using the internet for a job search. b.Veteran family member report completion of skill classes. c.Veterans find or are placed in jobs. Learning Exercise

High Quality Performance Measures NEED: 35% of young veterans (18-24 year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications and experience in professional settings. Intervention 1.Veterans participate in a drop-in job clinic offering job fairs, resume writing classes and skills in internet job searches. 2.Veterans supported in completing training programs through assistance in finding appropriate programs and financial aid, with tutoring resources, in internship placement. 3. Military families receive job placement counseling. Outcome a.Veterans report increased confidence in using the internet for a job search. b.Veteran family member report completion of job skills classes. c.Veterans find or are placed in jobs. Learning Exercise (continued) Select the outcome.

High Quality Performance Measures NEED: 35% of young veterans (18-24 year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications and experience in professional settings. Intervention 1.Veterans participate in a drop-in job clinic offering job fairs, resume writing classes and skills in internet job searches. 2.Veterans supported in completing training programs through assistance in finding appropriate programs and financial aid, with tutoring resources, in internship placement. 3. Military families receive job placement counseling. Outcome Veterans find or are placed in jobs. Learning Exercise (continued) Select the intervention.

High Quality Performance Measures NEED: 35% of young veterans (18-24 year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications and experience in professional settings. Intervention Veterans supported in completing training programs through assistance in finding appropriate programs and financial aid, with tutoring resources, in internship placement. Outcome Veterans find or are placed in jobs. Learning Exercise (continued) Summary: Alignment within Theory of Change

High Quality Performance Measures Alignment of Outputs and Outcomes Output results from the intervention Outcome is likely to result from output Output and outcome measure the same intervention and beneficiaries 20 InterventionOutputOutcome

High Quality Performance Measures INTERVENTION: Construction teams assess housing request, plan and execute housing upgrades and repairs and complete inspection requirements for individuals with disabilities. OUTPUT: Individuals with disabilities receive housing service (upgrades and repairs). OUTCOME: Construction team members improve construction skills. Do the intervention and output align? Yes No Why or why not? Housing Example: Output- Outcome Alignment

High Quality Performance Measures INTERVENTION: Construction teams assess housing request, plan and execute housing upgrades and repairs and complete inspection requirements for individuals with disabilities. OUTPUT: Individuals with disabilities receive housing service (upgrades and repairs). OUTCOME: Construction team members improve construction skills. Do the intervention and output align? Yes No Why or why not? Logical to assume if well-designed and implemented Housing Example: Output- Outcome Alignment

High Quality Performance Measures INTERVENTION: Construction teams assess housing request, plan and execute housing upgrades and repairs, and complete inspection requirements for individuals with disabilities. OUTPUT: Individuals with disabilities receive housing services (upgrades and repairs). OUTCOME: Construction team members improve construction skills. Do the output and outcome align? Yes No Why or why not? Housing Example: Output- Outcome Alignment

High Quality Performance Measures INTERVENTION: Construction teams assess housing request, plan and execute housing upgrades and repairs and complete inspection requirements for individuals with disabilities. OUTPUT: Individuals with disabilities receive housing service (upgrades and repairs). OUTCOME: Construction team members improve construction skills. Do the output and outcome align? Yes No Why or why not? 2 different groups of beneficiaries Housing Example: Output- Outcome Alignment

High Quality Performance Measures INTERVENTION: National Service participants provide one-on-one tutoring to preschool children in math readiness activities, 5 times a week over 9 months. Output 1.Children complete participation in the early childhood program. 2.Teachers complete training with expert math teachers. 3.Children complete participation in preschool art group. Outcome 1.Children demonstrate gains in school readiness in terms of literacy skills. 2.Children demonstrate gains in school readiness in terms of numeracy (math) skills 3.Teachers implement new math curriculum in the classroom. Learning Exercise Output-Outcome Alignment

High Quality Performance Measures INTERVENTION: National Service participants provide one-on-one tutoring to preschool children in math readiness activities, 5 times a week over 9 months. Output 1.Children complete participation in the early childhood program 2.Teachers complete training with expert math teachers. 3.Children complete participation in preschool art group. Outcome 1.Children demonstrate gains in school readiness in terms of literacy skills. 2.Children demonstrate gains in school readiness in terms of numeracy (math) skills 3.Teachers implement new math curriculum in the classroom. Learning Exercise Output-Outcome Alignment

High Quality Performance Measures INTERVENTION: National Service participants provide one-on-one tutoring to preschool children in math readiness activities, 5 times a week over 9 months. Output Children complete participation in the early childhood program. Outcome 1.Children demonstrate gains in school readiness in terms of literacy skills. 2.Children demonstrate gains in school readiness in terms of numeracy (math) skills 3.Teachers implement new math curriculum in the classroom. Learning Exercise Output-Outcome Alignment

High Quality Performance Measures Examples of Aligned Output-Outcome National Performance Measures Focus Area OutputOutcome EducationNumber of children that completed participation in CNCS-supported early childhood education programs. (ED21) Number of children demonstrating gains in school readiness in terms of social and/or emotional development.(ED23) Healthy Futures Number of individuals receiving support, services, education and/or referrals to alleviate long-term hunger (H11) Number of individuals that reported increased food security of themselves and their children (household food security) as a result of CNCS-supported services (H12) National performance measures at the Resource Center:

High Quality Performance Measures Identifying a Meaningful Outcome The Outcome should: Be central to your theory of change Match the type of change (attitude, knowledge, behavior, or condition) you want to measure Be ambitious but manageable 29

High Quality Performance Measures Central to the Theory of Change 30 How is the proposed outcome supported by the theory of change? Evidence for intervention should support the outcome Evidence supports cause-and-effect relationship between intervention and outcome

High Quality Performance Measures Central to the Theory of Change 31 How is the proposed outcome central to the theory of change? Addresses the need/problem Likely change given the intervention A key, not preliminary or peripheral, outcome

High Quality Performance Measures Which outcome is central? NEED: Documented high rates of chronic absence from school lead to other problems, e.g. low school engagement, poor academic performance (Social Work Practices, 2010). INTERVENTION: Weekly individual and group mentoring to 6 th grade students with attendance problems over 10 months. Focus on promoting re-engagement with school through social development and leadership skills. 1.Families are satisfied with the mentoring program 2.Students improve engagement with peers 3.Students improve school attendance Mentoring Example: Central to Theory of Change

High Quality Performance Measures Type of outcome: Accurate and Clear 33 Is the outcome getting at what you wish to measure? Articulates the question you really want to answer Has the outcome type been identified: attitude, knowledge/skills, behavior or condition? Outcome type should be same as theory of change Outcome type will inform instrument

High Quality Performance Measures Which type of outcome? NEED: 35% of young veterans ( year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications… INTERVENTION: National service participants support veterans in completing training programs by assisting in locating appropriate programs, securing financial aid, and by providing tutoring resources and internship placements. 1.Veterans report increased confidence about finding employment. (attitude) 2.Veterans demonstrate new technical skills. (knowledge/skills) 3.Veterans are placed in jobs. (condition) Veteran Example: Type of Outcome

High Quality Performance Measures Ambitious and Manageable 35 Is the outcome meaningful AND manageable? Consider: Staff time and expertise Reporting constraints

High Quality Performance Measures Ambitious and Manageable 36 Is the outcome too modest? Consider: Outcome should reflect the main change expected from the intervention Is the outcome too ambitious? Consider: Program timeframe Scope of the intervention Severity of the problems being addressed

High Quality Performance Measures Which outcome is meaningful and manageable? NEED: Documented high rates of chronic absence from school… INTERVENTION: National Service participants provide individual and group mentoring to 6 th graders with documented attendance problems… 1. Student increase outdoor teamwork skills 2.Students increase graduation rates from high school 3.Students improve school attendance Mentoring Example: Ambitious and Manageable

High Quality Performance Measures Which outcome is meaningful and manageable? NEED: Documented high rates of chronic absence from school… INTERVENTION: National Service participants provide individual and group mentoring to 6 th graders with documented attendance problems… 1. Student increase outdoor teamwork skills 2.Students increase graduation rates from high school 3.Students improve school attendance Mentoring Example: Ambitious and Manageable

High Quality Performance Measures Learning Exercise 1.Which outcome is too ambitious? (select one) 2.Which outcome is too modest? (select one) 3.Which outcomes look ambitious but manageable? (select two) Outcomes a.Students know location of the library b.Student improve reading ability c.Students graduate from high school d.Students increase school attendance Intervention: Year long mentoring project with 4 th graders– literacy focus.

High Quality Performance Measures Learning Exercise 1.Which outcome is too ambitious? (select one) 2.Which outcome is too modest? (select one) 3.Which outcomes look ambitious but manageable? (select two) Outcomes a.Parents are interested in learning more about the program. b.There is reduced gang violence in the area. c.Donations for sustaining activities at the two sites are secured. d.Trained community volunteers are serving at the new sites. Intervention: An afterschool program for children ages 6-12 is expanded to two new sites.

High Quality Performance Measures Learning Exercise 1.Which outcome is too ambitious? (select one) 2.Which outcome is too modest? (select one) 3.Which outcomes look ambitious but manageable? (select two) Outcomes a.Youth develop long- term healthy eating habits. b.Youth increase healthy food intake. c.Youth improve cooking skills. d.Youth are motivated to eat more vegetables. Intervention: 6-month nutrition and health program for high school students.

High Quality Performance Measures Learning Exercise 1.Which outcome is too ambitious? (select one) 2.Which outcome is too modest? (select one) 3.Which outcomes look ambitious but manageable? (select two) Outcomes a.Youth increase knowledge of native stream bank plant species. b.250,000 residents in the county benefit from restored watershed. c.Youth increase interest in protecting the environment. d.Water quality is improved over 6 mile clean-up area. Intervention: Year-long riparian (river) restoration project involving youth volunteers.

High Quality Performance Measures Learning Exercise 43 The most meaningful and manageable outcome for this intervention is: a)Number of individuals satisfied with quality of the work. b)Number of individuals securing mortgage loan approval for house purchase within 5 years. c)Number of individuals transitioned into safe housing. INTERVENTION: National Service participants assess the housing up-grade requests from the community members and address dilapidated, unsafe or inaccessible housing. They plan and then execute housing remodels, upgrades and repairs for economically disadvantaged individuals and individuals with disabilities.

High Quality Performance Measures Summary of Key Points Performance measurement is a systematic process of measuring progress (outputs and outcomes) Performance measurement does not seek to prove a theory of change, but can provide snapshots. Impact evaluation can determine if results occurred because of the intervention. Strong performance measures align with the theory of change (need, intervention and outcome) –The intervention is based on evidence that supports a cause-and-effect relationship between the intervention and an intended outcome. –The theory of change helps identify an appropriate outcome to measure. 44

High Quality Performance Measures Summary of Key Points Strong performance measures are aligned –Output results from the intervention –Outcome is likely to result from output –Output and outcome measure the same intervention and beneficiaries Strong outcomes should: –Be central to the theory of change –Match the type of change you want to see (attitude, knowledge/skills, behavior, condition) –Be ambitious but manageable 45

High Quality Performance Measures Resources CNCS Priorities and Performance Measures: Program Specific Notices of Funding Opportunities and Application Instructions: Link to Evaluation 46