Beyond Compliance: Informative Reporting 2008 AmeriCorps Conference

Objectives  Find Out How You’re Doing  Identify reporting challenges  Discuss informative reporting  Review sample report

Why Do Reporting?  Track progress to achieve your results  Identify and address problems  Determine need to make changes  Determine effective activities & processes before committing resources to sustain

How do you use your reports?

What happens from the very beginning of a project determines its eventual impact, long before a final report is issued

Program Stages and Reporting Progress 1 st Period 90 days from Start Date Initial Review 2 nd Period 180 days from Start Date Implementation Review 3 rd Period Implementation Review 4 th Period Results Review Address if components are in place as anticipated/designed for each PMW. Review progress on what needs to be in place for program design to work. Review and report progress or status as appropriate on: Report addressing if components are functioning as anticipated/designed for each PMW. Review progress on each PMW to confirm activities and processes are being implemented as designed. You should report: Address results [outputs & outcomes]. Review progress on PMWs to confirm activities & processes implemented as designed. At this point in the program, there should be evidence of outcomes. You should report: Report addressing final results [outputs and outcomes] of each PMW and the overall program. At this point in the program there should be final outcome data analysis. You should report: a)Pre-service data prep [appropriate instrument(s) in place, or access to existing data]. b)Process to select high need target population is in place. c)Performance Measurement Plan (WBRS) for each instrument complete. d)Status of PM Plan --how & when pre-service data will be/has been collected, analyzed, & reported. e)Report on pre-service data analysis [e.g., # assessed, # in high need group, etc]; or date available. f)Changes: staff, member type, member/staff/site training, dosage, targeted beneficiaries, partner commitments, sites, supervision,… g)Analysis of the initial review process to determine readiness to meet results [outputs, outcomes] h)Training & technical assistance. i)Other issues of concern. a)PM Plan data collection & analysis during this period [pre, mid and/or post]. b)Status & projection on output targets. c)Status of established benchmarks for outcome targets. d)Members’ activities—have they changed, are they on course? e)Status of member training—has it changed, is it on course?. f)Challenges or barriers that effected or may effect ability to meet targets—changes in: staffing, member configuration, training [member/staff/site], dosage, targeted beneficiaries, partner commitments, sites, supervision, etc. g)Plan of action for identified challenges or barriers. h)Training & technical assistance needed. The items in 2 nd Quarter plus: a)Successes and achievements within the targeted areas. b)Program or intervention adjustments based on analysis of pre, mid and/or post data. c)Accuracy of instruments and data. d)Member performance. e)Training & technical assistance needed a.Final result of the targets as stated in the PMW. Results should address the exact target. b.Outcome targets include percent of people or things that changed, what changed and the amount of change c.Highlights of program partner meeting to discuss program outcomes and continuous improvement strategies. d.Plan of action for identified challenges or barriers. e.Training & technical assistance needed

Are components in place and started? Talk about: Selection, training, placement for members, staff, & site. Amount & findings of pre-service data [& outcomes if known]. Beneficiary selection. Member enrollment. Partners. Problems to address. Is it starting as planned? Initial Review- 1st Period

Are things being implemented as designed? Talk about progress/changes in: activities, data collection & analysis [pre, mid, post], outcome benchmarks, staffing, member configuration, beneficiaries, training [member/staff/site], sites & partner commitments. Outcomes should be reported in this stage. Your plan for challenges or barriers. Implementation Review 2 nd & 3 rd period

How did it all turn out? Report final results [outputs & outcomes] and overall program findings. Address the exact target. Talk about partner meeting to discuss outcomes & continuous improvement strategies. Results Review 4 th period

How’s reporting going for your program?

Typical Reporting Problems 1.Faulty Logic on PMW  Unclear/unfocused results  Activity doesn’t produce result  Targets not specific [amount of change]  Measures inappropriate

Typical Reporting Problems 2.Data Issues  Inadequate data collected No baseline data—or too late No data collected or not enough  Inappropriate Instrument  Data doesn’t address Target

Typical Reporting Problems 3.What’s Actually Reported  Processes not assessed to avoid problems  Data not analyzed &/or summarized No discussion/breakout of data [pre] Data dump—too much to decipher  Results/findings not summarized Not concise, unfocused—no analysis

What Makes a Strong Report?  Conveys the status of the program  Reflects program planning and refining  Based on a strong PMW--logic model  Clear & focused on relevant information  Paints a picture—says something  Reflects a plan for data collection & reporting

 Due dates and timelines  Adequate and sufficient data collection  Timely data collection  Who will aggregate & analyze data & when  You have a Plan! Data collection and analysis has to be seen as important to program operation A Data Plan Establishes:

highneedtargetgroup  Identify a high need target group. Who really benefits from your service?  How much service is needed to achieve the results? What’s the dosage?  Importance of periodic assessment  Importance of being curious about the impact of your services  Develop a Data Plan Areas of Concern

Reporting Challenges to Avoid 1.Losing the purpose of the report. It’s not just a funding requirement 2.Uninformative reporting about the process 3.Report doesn’t really reflect the program 4.Weak/no findings about results 5.Weak information about outcomes

Small Group 1.Review the sample report 2.Compare your report to the sample 3.Discuss how you might improve reporting  Do you have a data plan?  Are reports informative to you?  Do you report on program promises? 4.Be prepared to report to large group

Assessment Sayings  What gets measured gets done.  If you can demonstrate results, you can win public support  If you can’t recognize what’s not working, you can’t correct it. SOURCE: Osborne & Gaebler: Results Oriented Government, 1992.

Tips  Don’t delay analysis of pre-service data actualanticipated  Compare actual to anticipated  Discuss comparison. What does it mean? you  Interpret results so you know what to do  Be sure you can really get the data  Results should not be a surprise.  Say something!