Presentation is loading. Please wait.

Presentation is loading. Please wait.

Introduction to Performance Measurement for AmeriCorps*VISTA 1-800-548-3646 www.projectstar.org Washington Dulles Hilton September 12-14, 2005.

Similar presentations


Presentation on theme: "Introduction to Performance Measurement for AmeriCorps*VISTA 1-800-548-3646 www.projectstar.org Washington Dulles Hilton September 12-14, 2005."— Presentation transcript:

1 Introduction to Performance Measurement for AmeriCorps*VISTA Washington Dulles Hilton September 12-14, 2005

2 Day One - Afternoon Session This session will cover: Six Steps of Performance Measurement (PM) Types of Results: Outputs, Intermediate and End Outcomes Use of the Logic Model as a Planning Tool

3 Performance measurement is a six-step process for regularly measuring your project results. Performance Measurement: The Six Steps

4 Performance Measurement Six Steps The logic model describes your AmeriCorps*VISTA project. Use it to identify your project activities and what you will measure to gauge the success of your project. 1. Develop Logic Model & Project Plan

5 Performance Measurement Six Steps 1. Develop Logic Model & Project Plan 2. Develop PM Plan Identifying your data sources and data collectors. Determine a timeline for data collection, analysis, and reporting

6 Performance Measurement Six Steps 1. Develop Logic Model & Project Plan 2. Develop PM Plan 3. Develop and Test Instruments Adapt existing instruments or develop your own; pilot test to make sure instruments collect valid, appropriate information

7 Performance Measurement Six Steps 1. Develop Logic Model & Project Plan 2. Develop PM Plan 3. Develop and Test Instruments 4. Collect Data Go out and collect your data (e.g., fill out attendance sheets, administer surveys, conduct interviews)

8 Performance Measurement Six Steps 1. Develop Logic Model & Project Plan 2. Develop PM Plan 3. Develop and Test Instruments 4. Collect Data 5. Analyze Data Aggregate and analyze data; generate summary info (totals, averages, etc)

9 Performance Measurement Six Steps 1. Develop Logic Model & Project Plan 2. Develop PM Plan 3. Develop and Test Instruments 4. Collect Data 5. Analyze Data 6. Report Results Write progress report; include your performance measurement results

10 Performance Measurement Six Steps 1. Develop Logic Model & Project Plan 2. Develop PM Plan 3. Develop and Test Instruments 4. Collect Data 5. Analyze Data 6. Report Results Continuous Program Improvement

11 What Are Results? Outputs and outcomes your AmeriCorps*VISTA project hopes to achieve by engaging in service activities Outputs: Services delivered and products completed by VISTA members Outcomes: Changes or benefits that occur as a result of services delivered Intermediate Outcomes: Changes that are short of the final result you hope to achieve each year End Outcomes: The most significant changes that your AmeriCorps*VISTA project hopes to achieve each year

12 Sample Scenario Community-based organization (CBO) has existing mentoring program, but faces challenges in recruiting and retaining volunteer mentors. Community-based organization (CBO) has existing mentoring program, but faces challenges in recruiting and retaining volunteer mentors. CBO unable to deliver high-quality service to youth. CBO unable to deliver high-quality service to youth. VISTA members will serve with CBO to develop system to recruit, train and support volunteer mentoring program. VISTA members will serve with CBO to develop system to recruit, train and support volunteer mentoring program.

13 Position descriptions for volunteer mentors Written procedures for screening volunteer mentors Tracking database Training curriculum for new volunteer mentors Written procedures to provide ongoing support to volunteer mentors Outputs answer the question, How much service did we do? They do NOT answer the question, What changed as a result of the service? Outputs

14 Trained agency personnel demonstrate understanding of how to use new system and procedures (new knowledge and skills). Agency personnel use new systems and procedures to recruit, train, and support volunteer mentors (changes in behavior). Intermediate outcomes are important milestones on the way to achieving end outcomes. Achieving intermediate outcomes can lead to attaining end outcomes. Intermediate Outcomes

15 More youth are served by volunteer mentors under the new system than were served under the previous system (increased beneficiaries served). More youth experience a close and stable relationship with a caring adult, because a greater proportion of volunteer mentors complete their service commitments (improved quality of service). End outcomes are changes that reflect the overarching goals of your AmeriCorps*VISTA project. End Outcomes

16 16 Examples of Results OUTPUTSINTERMEDIATE OUTCOME(S) END OUTCOMES Individual Development Account trainings provided Participants IDA program participants accumulate savings to use for purchase of assets IDA program participants use accumulated savings to purchase assets they had planned to purchase Training sessions provided Parent Network Collaborative meetings Network referral system developed Network agencies expand quality and range of services

17 17 Exercise Identifying Types of Results

18 18 Benefits of Developing a Logic Model Helps to build consensus about program goals and desired results Clarifies relationship between key program elements (inputs, activities, outputs, and outcomes) Brings fuzzy goals down to earth Begins to identify things to measure Provides solid foundation for developing project plan

19 19 AmeriCorps*VISTA Logic Model A Logic Model = a planning tool that diagrams how you intend to achieve your AmeriCorps*VISTA project goals by combining resources (inputs) and service activities to produce specific results (outputs and outcomes) that address poverty-related community needs.

20 20 AmeriCorps*VISTA Logic Model Community Need Community Need = Unresolved poverty-related problem or issue in the community that your VISTA project will address

21 21 AmeriCorps*VISTA Logic Model Community Need Goals InputsActivitiesOutputs Intermediate Outcomes End Outcomes End Outcomes= The most significant changes achieved by your project Answer the questions: So what? What changed?

22 22 AmeriCorps*VISTA Logic Model Goals = Broad purposes toward which your VISTA project is directed Community Need Goals

23 23 AmeriCorps*VISTA Logic Model Inputs= Resources your VISTA project will need to engage in the identified activities and produce outputs and outcomes (human, financial, organizational, community) Community Need Goals Inputs

24 24 AmeriCorps*VISTA Logic Model Activities= What VISTA members do to address the community need and achieve desired results Community Need Goals InputsActivities

25 25 AmeriCorps*VISTA Logic Model Outputs: Reflect the amount of service completed by VISTA members. Outputs do not answer the questions: what changed as a result of the work? or how did people or the organization benefit? Community Need Goals InputsActivitiesOutputs

26 26 AmeriCorps*VISTA Logic Model Intermediate Outcomes= Initial changes achieved by your project; milestones on the way to achieving end outcomes Community Need Goals InputsActivitiesOutputs Intermediate Outcomes

27 27 AmeriCorps*VISTA Logic Model Community Need Goals InputsActivitiesOutputs Intermediate Outcomes End Outcomes End Outcomes= The most significant changes achieved by your project Answer the questions: So what? What changed?

28 28 AmeriCorps*VISTA Logic Model Community Need Goals InputsActivitiesOutputs Intermediate Outcomes End Outcomes Program Planning Intended Results

29 29 Community Need Goals InputsActivitiesOutputs Intermediate Outcomes End Outcomes Logic Model for a Mentoring Program (Year 3) Children of incarcerated parents lack social and emotional support for positive youth development that can help to break the cycle of poverty. Develop, enhance systems to recruit, train, and support volunteer mentors. 2 VISTAs Supervisor Program staff Money Partners Assist with implementa- tion, assess performance of new systems, identify promising practices. Policies and guidance revised for volunteer recruitment and sustain- ability systems. Improved recruitment: more youth are served by more volunteer mentors. Improved retention: more volunteer mentors complete service. Improved program quality: more volunteer mentors form closer bonds with youth; mentoring relationships last longer.

30 30 Exercise The Logic Model Puzzle

31 31 Exercise Develop a Logic Model for Your Project

32 32 End of Day Review + What Worked Well About Today? How Could Tomorrow Be Made Even Better?

33 33 Day Two – Morning Session This session will cover: 1. Moving from a Logic Model to the Project Plan 2. CNCS Requirements for VISTA 3. Components of a Project Plan 4. Indicators, Targets, Instruments

34 34 AmeriCorps*VISTA Performance Measurement Requirements One project plan containing one or more goals One project plan containing one or more goals Identify activities and results for each goal Identify activities and results for each goal Results are written as outputs, intermediate outcomes, or end outcomes Results are written as outputs, intermediate outcomes, or end outcomes Label 3-5 of these results as performance measures Label 3-5 of these results as performance measures –One Output –One Intermediate Outcome –One End Outcome Identify indicators, targets, and instruments for each performance measure Identify indicators, targets, and instruments for each performance measure

35 35 VISTA Project Plan Components Goals (Same) Activities (formerly Objectives) Results (New) Output(s) Intermediate Outcome(s) End Outcome(s)

36 36 AmeriCorps*VISTA Project Plan Column A Goals, Activities, and Results Column B Date(s) Column C Project Reporting GOAL 1 Activities Results Outputs Intermediate Outcomes End Outcomes GOAL 2 Activities Results Outputs Intermediate Outcomes End Outcomes GOAL 3 Etc.

37 37 AmeriCorps*VISTA Project Plan Column A Goals, Activities, and Results Column B Date(s) Column C Project Reporting GOAL 1 Activities Results Outputs Intermediate Outcomes End Outcomes GOAL 2 Activities Results Outputs Intermediate Outcomes End Outcomes GOAL 3 Etc.

38 38 AmeriCorps*VISTA Project Plan Column A Goals, Activities, and Results Column B Date(s) Column C Project Reporting GOAL 1 Activities Results Outputs Intermediate Outcomes End Outcomes GOAL 2 Activities Results Outputs Intermediate Outcomes End Outcomes GOAL 3 Etc.

39 39 From Logic Model to Project Plan Column A Goals, Activities, and Results Column B Date(s) Column C Project Reporting Goal(s) Activities Results Outputs Intermediate Outcomes End Outcomes Community Need Goals InputsActivitiesOutputs Intermediate Outcomes End Outcomes

40 40 Multi-Year Goals, Annual Activities and Results Goals are multi-year. Goals are multi-year. Activities and results may vary from one year to the next. Activities and results may vary from one year to the next. Each year the project builds on the previous years activities and results. Each year the project builds on the previous years activities and results. Only list activities and results for current year. Only list activities and results for current year.

41 41 Exercise Identify One Goal for Your Project Identify One Goal for Your Project For that Goal, identify the Activities for Year One and Possible Results: For that Goal, identify the Activities for Year One and Possible Results: –Outputs –Intermediate outcome(s) –End outcome(s)

42 42 AmeriCorps*VISTA Project Plan Column A Goals, Activities, and Results Column B Date(s) Column C Project Reporting GOAL 1 GOAL 2 Activities Results Output Performance Measure Indicator Target Instrument Intermediate Outcome Performance Measure Indicator Target Instrument End Outcome Performance Measure Indicator Target Instrument

43 43 AmeriCorps*VISTA Project Plan Column A Goals, Activities, and Results Column B Date(s) Column C Project Reporting GOAL 1 GOAL 2 Activities Results Output Performance Measure Indicator Target Instrument Intermediate Outcome Performance Measure Indicator Target Instrument End Outcome Performance Measure Indicator Target Instrument

44 44 What is an Indicator? Information you collect to determine progress toward achieving results Information you collect to determine progress toward achieving results Concrete and measurable Concrete and measurable Answers the question, What is my that something has occurred? Answers the question, What is my evidence that something has occurred? Output indicator usually begins with Number of... Output indicator usually begins with Number of... Outcome indicator usually begins with Percent of... Outcome indicator usually begins with Percent of...

45 45 Example of Result and Indicators Desired Result: Increase service to clients Desired Result: Increase service to clients Possible Indicators: Possible Indicators: -Greater number of clients served (number of clients served, compared to last year) -Clients receive more services (number of services provided, per client, compared to last year) -Clients receive services of greater value (dollar value of services provided, per client, compared to last year)

46 46 What is a Target? Describes amount of change you anticipate achieving each year due to efforts of VISTA members. Describes amount of change you anticipate achieving each year due to efforts of VISTA members. Based on indicator – specific, measurable amount of change that will signify Based on indicator – specific, measurable amount of change that will signify success Outcome Target: Outcome Target: – will experience change? –Who will experience change? – people will experience change? –How many people will experience change? – change will occur (compared to baseline)? –How much change will occur (compared to baseline)? –Over will change occur (one year)? –Over what time period will change occur (one year)?

47 47 Target describes amount/level of change you anticipate achieving in one year due to efforts of VISTA members. Target describes amount/level of change you anticipate achieving in one year due to efforts of VISTA members. Based on indicator (concrete, measurable) Based on indicator (concrete, measurable) Rely on experience and knowledge of community to judge how much change to expect in one year. Rely on experience and knowledge of community to judge how much change to expect in one year. Keep in mind your resources. Keep in mind your resources. Be realistic! Be realistic! Formulating Annual Targets

48 48 Indicator and Target Examples OutputOutcome Result Volunteer database developed. Improve volunteer base. Indicator Volunteer database developed. Number of new volunteers recruited. Target One volunteer database developed. 300 new volunteers recruited.

49 49 What is an Instrument? Document or form used to collect information from data source Document or form used to collect information from data source Data source: Person, place or group that provides you with information to measure indicators Data source: Person, place or group that provides you with information to measure indicators Examples: Attendance rosters, tally sheets, questionnaires, checklists Examples: Attendance rosters, tally sheets, questionnaires, checklists Adapt existing instruments or develop your own Adapt existing instruments or develop your own

50 50 Advice about Instruments Make sure instrument will measure indicators Make sure instrument will measure indicators Consider who will administer instrument Consider who will administer instrument Consider who will complete instrument Consider who will complete instrument Consider when to administer instrument Consider when to administer instrument Consider data accessibility Consider data accessibility

51 51 Project Plan: Performance Measures Column A Goals, Activities, and Results Column B Date(s) Column C Project Reporting GOAL ACTIVITIES (Year 1) RESULTS (Year 1) Outputs: 1.Volunteer mentor recruitment systems are developed. 2.Agency personnel receive orientation in new systems Performance Measure Indicator: # agency personnel oriented Target: 2 F/T and 4 P/T agency personnel oriented Instrument: Orientation Log Qtr 1–3 Qtr 4

52 52 Project Plan: Performance Measures Column A Goals, Activities, and Results Column B Date(s) Column C Project Reporting GOAL ACTIVITIES (Year 1) RESULTS (Year 1) Outputs: 1.Volunteer mentor recruitment systems are developed. 2.Agency personnel receive orientation in new systems Performance Measure Indicator: # agency personnel oriented Target: 2 F/T and 4 P/T agency personnel oriented Instrument: Orientation Log Intermediate Outcome: Agency personnel understand how to use new systems. Performance Measure Indicator: # personnel who correctly identify staff roles and who can find info in new database Target: 5 personnel can recruit, screen and train mentors Instrument: Mentoring System User Checklist Qtr 1–3 Qtr 4

53 53 Project Plan: Performance Measures Column A Goals, Activities, and Results Column B Date(s) Column C Project Reporting GOAL ACTIVITIES (Year 1) RESULTS (Year 1) Outputs: 1.Volunteer mentor recruitment systems are developed. 2.Agency personnel receive orientation in new systems Performance Measure Indicator: # agency personnel oriented Target: 2 F/T and 4 P/T agency personnel oriented Instrument: Orientation Log Intermediate Outcome: Agency personnel understand how to use new systems. Performance Measure Indicator: # personnel who correctly identify staff roles and who can find info in new database Target: 5 personnel can recruit, screen and train mentors Instrument: Mentoring System User Checklist End Outcome: Agency personnel use new systems. Performance Measure Indicator: # agency personnel using # new systems Target: Four agency personnel, two systems Instrument: Year-end Agency Interview Qtr 1–3 Qtr 4

54 54 Exercise Identifying Indicators, Targets and How Measured for Outcomes

55 55 Exercise Add Enhancements to Your Project Plan Results

56 56 Day Two – Afternoon Session This session will cover: 1. Selecting Instruments 2. Data Collection Strategies 3. Data Aggregation and Analysis 4. Reporting

57 57 Strategies to Ensure Successful Reporting From the beginning… Align work plan and instruments Plan for collecting the information Understand how to analyze data Anticipate how you will report

58 58 What Do I Report On? Actual Activities Actual Output – –Actual target as compared to anticipated target – –How data collected Actual Intermediate Outcome – –Actual target as compared to anticipated target – –How data collected Actual End Outcome – –Actual target as compared to anticipated target – –How data collected

59 59 Data Collection Issues Can you access the data source (beneficiaries, organizations)? Does an instrument (site or project) exist? Are they willing to give you data? (burden, capacity) Can you get the data when you need it? Are your helpers (e.g. VISTAs) trained? Do you have a mechanism to collect summarized data from sites?

60 60 Methods Logs, Tally Sheets, Attendance Rosters Survey, Questionnaire (Check list, Index, Scale) Interview (Interview Guide) Service Recipients Interview Expert/Provider/Key Informant Observation (Observation Guide) Case Study (Measurable change of a case over time) Focus Group (Focus Group Guide) Secondary Data (Existing records and information) Test / Experimental Designs

61 61 Instrument Considerations Connect to Project Plan results – –Measures the identified PM Aim for appropriate method – –Measures the correct population (data source) Lower burden of use – –e.g. Captive audience, (part of project) – –Balance between best method and manageable method Assure that sites have instruments or summary tool

62 62 Exercise Instrument and Data Collection Review and Discussion

63 63 Data Analysis Quantitative Frequencies Frequencies Ranges Ranges Percents Percents Means Means Pre-post comparison Pre-post comparison Qualitative Content analysis Content analysis –Pre-existing themes –Emergent themes

64 64 Exercise Data Aggregation and Analysis

65 65 Reporting Tips Provide evidence. Avoid hearsay and generalizations. Be candid about your efforts. Dont claim too little. Dont claim too much. Tie report to Project Plan. State results in same format as original target in Project Plan. Pay attention to enhancements: indicator, target, and instruments Pay attention to enhancements: indicator, target, and instruments

66 66 Report Template Reporting Format for Each Performance Measure Actual Target: 1) Compare the actual to the anticipated target stated in Column A. 1) Compare the actual to the anticipated target stated in Column A. 2) Met target? If target not met, explain. 2) Met target? If target not met, explain. 3) Include any additional information about your result. 3) Include any additional information about your result. How Data Collected: 1) Describe how the result was measured. 1) Describe how the result was measured.

67 67 Reporting Each Result Actual Target: 1) Compare the actual target to the anticipated target stated in Column A. Interpret results; explain what it means 2) Met target? If target not met, explain. Include numbers not just percentages. Explicitly state that you did (not) meet target 3) Include any additional information about your result. Describes challenges, next steps

68 68 Describing Your Results Actual Target: 1) Compare the actual target to the anticipated target stated in Column A. [Always include an actual number] 1) Compare the actual target to the anticipated target stated in Column A. [Always include an actual number] 10 interview respondents (86% of collaborative) identified at least three beneficial practices resulting from the partner collaboration. This exceeds our target of one project improvement. The top two included an increase in clients served and…

69 69 Reporting Format for Each Result How Data Collected: 1) Describe how the result was measured. What was used to track the result? If applicable, how was the instrument administered? (When was it administered? How often?) If applicable, include total number approached as well as those who responded.

70 70 Describing Your Results How Data Collected: 1) Describe how the result was measured During the 2nd and 3rd weeks of April, the Project Supervisor, and two VISTAs conducted telephone interviews during the afternoon and evening…...Of the 12 collaborative partners called, 10 completed the interview that assessed their practices resulting from the collaborative…

71 71 Describing Your Work Before Results Are Available...We dont have any results yet.

72 72 What do I Report When I Have Nothing to Report Yet? We are….. Development/piloting of instruments Introducing instrument to sites Gaining access to data Training data collectors Challenges and proposed solutions Administering pre-survey data. (If you have baseline data, report it early!) Say something! Dont wait.

73 73 Describing Your Work Before Results Are Available (Improved) How Data Collected: 1) Describe how the result was measured...In preparation for implementing performance measurement, we drafted a telephone interview protocol containing six questions designed to assess partner collaborative practices. During the first week of March, two collaborative members reviewed the draft and six VISTAs were trained to conduct interviews...

74 74 Day Three – Morning Session This session will cover: 1. Reporting 2. Using the Results of Performance Measurement 3. eGrants and VISTA 4. Closing - Receive box lunch and departure

75 75 Exercise Progress Report Critique Using Checklist

76 76 Exercise Communicating Success Ways to Use PM Results to Strengthen and Promote Your Project

77 Additional Resources: Contact your CNCS State Office Contact your CNCS State Office Contact Project STAR Contact Project STAR


Download ppt "Introduction to Performance Measurement for AmeriCorps*VISTA 1-800-548-3646 www.projectstar.org Washington Dulles Hilton September 12-14, 2005."

Similar presentations


Ads by Google