1 Creating Strong Reports New Mexico AmeriCorps April 20, 2006 Sue Hyatt, Project STAR Coach

2 Agenda Learn some dos and donts for performance measures (PMs) in the APR. Learn some dos and donts for performance measures (PMs) in the APR. Reporting Tips Reporting Tips Review of Your Most Recent Report Review of Your Most Recent Report

3 Project STAR Findings: Common PM and Reporting Issues Outputs versus Outcomes Outputs versus Outcomes Intermediate versus End Outcomes Intermediate versus End Outcomes Creative Writing - Indicators and Targets Creative Writing - Indicators and Targets Targets Not Related to Instruments Targets Not Related to Instruments Weak Community Need Statements Weak Community Need Statements

4 REVIEW: Types of Performance Measures Outputs Outputs Intermediate Outcomes Intermediate Outcomes End Outcomes End Outcomes

5 Outputs Are counts of the amount of services delivered, work, or products created that members or volunteers have completed. Usually number of people served Do not answer the question, what changed as a result of our work? Do not provide information on benefits or other changes in the lives of beneficiaries. Examples: Examples: 30 students tutored in reading 100 tress planted 25 houses built benefiting 25 families

6 End Outcomes Answer the question, So what? Changes that have occurred in the lives of beneficiaries that are significant Examples: Students will not engage in drug or alcohol use Increasing homeownership among clients Students demonstrate better in school behavior

7 Intermediate Outcomes Answer the question, what difference did it make? Do not represent the final result you hope to achieve for beneficiaries Likely preconditions for longer-term gains When positive are usually a sign that your project is on track to achieve the related end outcomes. Examples: Students report feeling safe and not pressured to engage in risky behavior. Decrease in volunteer attrition. Students will complete homework assignments.

8 Possible Beneficiary Changes Outputs: Number of People Served Outputs: Number of People Served Outcomes: How People Changed Outcomes: How People Changed Knowledge Knowledge Attitudes Attitudes Behavior or Skills Behavior or Skills IntermediateEnd

9 What is an Indicator? Information you collect to determine progress toward achieving results Information you collect to determine progress toward achieving results Concrete and measurable Concrete and measurable Answers the question, What is my that something has occurred? Answers the question, What is my evidence that something has occurred? Output indicator usually begins with Number of... Output indicator usually begins with Number of... Outcome indicator usually begins with Percent of... Outcome indicator usually begins with Percent of...

10 Internal Alignment: End Outcome Members will provide 1 on 1 reading tutoring to third graders at least 3 times a week for 20 minutes/session End Outcome Result: Students have improved proficiency in reading End Outcome Result: Students have improved proficiency in reading Indicator: Percentage of students tutored who have improved proficiency in reading Indicator: Percentage of students tutored who have improved proficiency in reading Target: 75% (263 of 350) students tutored will have improved proficiency in reading Target: 75% (263 of 350) students tutored will have improved proficiency in reading Instrument: Reading Grades Instrument: Reading Grades

11 Internal Alignment: End Outcome (NOT!) Members will provide 1 on 1 reading tutoring to third graders at least 3 times a week for 20 minutes/session End Outcome Result: Students have improved proficiency in reading End Outcome Result: Students have improved proficiency in reading Indicator: Teachers will report that 75% students have completed more reading homework. Indicator: Teachers will report that 75% students have completed more reading homework. Target: Students tutored will show improved behavior in the classroom and be sent to the Principals office less often. Target: Students tutored will show improved behavior in the classroom and be sent to the Principals office less often. Instrument: logs and surveys Instrument: logs and surveys

12 APR Dos and Donts: Do create performance measurement plans

13 What is a Performance Measurement Plan? Written description of steps you will take to obtain data to measure outputs and outcomes The PM plan specifies: Who will do each step When they will do it Also known as data collection plan PM plan is instrument-focused The PM plan is your roadmap during the program year. It helps you obtain reliable, useful information about the effect of your program on beneficiaries.

14 Performance Measurement Plans (continued)

15 Performance Measurement Plans (continued)

16 APR Dos and Donts: Do include strong descriptions of service activities

17 APR Dos and Donts: Do clearly identify instruments

18 APR Dos and Donts: Do talk about change in outcome results

19 APR Dos and Donts: Dont get carried away with output codes

20 Why Do Progress Reports? Enhances program accountability Enhances program accountability Opportunity for reflection and strategic thinkingsee the forest for the trees Opportunity for reflection and strategic thinkingsee the forest for the trees Helps you become a great communicatorthe more you do it, the better you get at it. Helps you become a great communicatorthe more you do it, the better you get at it. Program staff turnover: Progress report is a permanent document. Program staff turnover: Progress report is a permanent document. Helpful in identifying TA needs. Helpful in identifying TA needs.

21 Calculating Targets: Who Should I Report On? Entire needy population Your beneficiaries Beneficiaries you measured Successful beneficiaries

22 Example of Target Calculation Successful Beneficiaries You Measured Beneficiaries You Measured Target: Sixty percent of neighborhood residents will report an increase in recycling. 67%

23 Completing the APR Progress Report

24 Well-written Progress Reports Answer These Questions: How much work is complete? How much work is complete? What work is currently in progress? What work is currently in progress? What work hasnt started yet? What work hasnt started yet? What challenges have arisen? What challenges have arisen? What steps are being taken to address these challenges? What steps are being taken to address these challenges? These same questions apply to reporting on performance measurement!

25 Describing Your Work Before Results Are Available...We dont have any results yet.

26 Describing Your Work Before Results Are Available Report with respect to: Report with respect to: –Developing logic models and indicators –Refining data collection plans –Data access/confidentiality (human subjects) –Developing and pilot testing instruments –Training data collectors

27 Describing Your Results Tell the reader: Tell the reader: –How and when data were collected –How data were analyzed Account for any data that were inaccessible or unusable. Account for any data that were inaccessible or unusable. Include numbers, not just percentages. Include numbers, not just percentages. Dont delay analyzing and reporting baseline data! Dont delay analyzing and reporting baseline data!

28 Describing Challenges & Solutions Worst thing to report: No challenges Worst thing to report: No challenges Explain how challenges affect program operation. Explain how challenges affect program operation. Report steps taken to address challenges. Report steps taken to address challenges. Always have a backup plan (in case your proposed solution doesnt work) Always have a backup plan (in case your proposed solution doesnt work) Bring past challenges to closure Bring past challenges to closure

29 Exercise Review Your Recent Progress Report and Note What You Want to Remember to Upgrade Next Time You Do a Report Review Your Recent Progress Report and Note What You Want to Remember to Upgrade Next Time You Do a Report

30 Additional : Additional Resources: Contact Project STAR Contact Project STAR