1. Introduction to Performance Measurement for AmeriCorps*VISTA
Washington Dulles Hilton
September 12-14, 2005

2. Day One - Afternoon Session
This session will cover:
Six Steps of Performance Measurement (PM)
Types of Results: Outputs, Intermediate and End Outcomes
Use of the Logic Model as a Planning Tool

3. Performance Measurement:
The Six Steps
Performance measurement is a six-step process for regularly measuring your project results.

4. 1. Develop Logic Model & Project Plan
Performance Measurement Six Steps
1. Develop Logic Model & Project Plan
The logic model describes your AmeriCorps*VISTA project. Use it to identify your project activities and what you will measure to gauge the success of your project.

5. 1. Develop Logic Model & Project Plan
Performance Measurement Six Steps
1. Develop Logic Model & Project Plan
Identifying your data sources and data collectors. Determine a timeline for data collection, analysis, and reporting
2. Develop PM Plan

6. 1. Develop Logic Model & Project Plan
Performance Measurement Six Steps
1. Develop Logic Model & Project Plan
Adapt existing instruments or develop your own; pilot test to make sure instruments collect valid, appropriate information
2. Develop PM Plan
3. Develop and Test Instruments

7. 1. Develop Logic Model & Project Plan
Performance Measurement Six Steps
1. Develop Logic Model & Project Plan
Go out and collect your data (e.g., fill out attendance sheets, administer surveys, conduct interviews)
2. Develop PM Plan
3. Develop and Test Instruments
4. Collect Data

8. 1. Develop Logic Model & Project Plan
Performance Measurement Six Steps
1. Develop Logic Model & Project Plan
Aggregate and analyze data; generate summary info (totals, averages, etc)
2. Develop PM Plan
5. Analyze Data
3. Develop and Test Instruments
4. Collect Data

9. Write progress report; include your performance measurement results
Performance Measurement Six Steps
6. Report Results
1. Develop Logic Model & Project Plan
Write progress report; include your performance measurement results
2. Develop PM Plan
5. Analyze Data
3. Develop and Test Instruments
4. Collect Data

10. 1. Develop Logic Model & Project Plan
Performance Measurement Six Steps
1. Develop Logic Model & Project Plan
6. Report Results
Continuous Program Improvement
2. Develop PM Plan
5. Analyze Data
3. Develop and Test Instruments
4. Collect Data

11. What Are Results?
Outputs and outcomes your AmeriCorps*VISTA project hopes to achieve by engaging in service activities
Outputs: Services delivered and products completed by VISTA members
Outcomes: Changes or benefits that occur as a result of services delivered
Intermediate Outcomes: Changes that are short of the “final” result you hope to achieve each year
End Outcomes: The most significant changes that your AmeriCorps*VISTA project hopes to achieve each year

12. Sample Scenario
Community-based organization (CBO) has existing mentoring program, but faces challenges in recruiting and retaining volunteer mentors.
CBO unable to deliver high-quality service to youth.
VISTA members will serve with CBO to develop system to recruit, train and support volunteer mentoring program.

13. Outputs Position descriptions for volunteer mentors
Written procedures for screening volunteer mentors
Tracking database
Training curriculum for new volunteer mentors
Written procedures to provide ongoing support to volunteer mentors
Outputs answer the question, “How much service did we do?” They do NOT answer the question, “What changed as a result of the service?”

14. Intermediate Outcomes
Trained agency personnel demonstrate understanding of how to use new system and procedures (new knowledge and skills).
Agency personnel use new systems and procedures to recruit, train, and support volunteer mentors (changes in behavior).
Intermediate outcomes are important milestones on the way to achieving end outcomes. Achieving intermediate outcomes can lead to attaining end outcomes.

15. End Outcomes
More youth are served by volunteer mentors under the new system than were served under the previous system (increased beneficiaries served).
More youth experience a close and stable relationship with a caring adult, because a greater proportion of volunteer mentors complete their service commitments (improved quality of service).
End outcomes are changes that reflect the overarching goals of your AmeriCorps*VISTA project.

16. INTERMEDIATE OUTCOME(S)
Examples of Results
OUTPUTS
INTERMEDIATE OUTCOME(S)
END OUTCOMES
Individual Development Account trainings provided
Participants
IDA program participants accumulate savings to use for purchase of assets
IDA program participants use accumulated savings to purchase assets they had planned to purchase
Training sessions provided
Parent Network Collaborative meetings
Network referral system developed
Network agencies expand quality and range of services

17. Identifying Types of Results
Exercise
Identifying Types of Results

18. Benefits of Developing a Logic Model
Helps to build consensus about program goals and desired results
Clarifies relationship between key program elements (inputs, activities, outputs, and outcomes)
Brings “fuzzy goals” down to earth
Begins to identify things to measure
Provides solid foundation for developing project plan

19. AmeriCorps*VISTA Logic Model
a planning tool that diagrams how you intend to achieve your AmeriCorps*VISTA project goals by combining resources (inputs) and service activities to produce specific results (outputs and outcomes) that address poverty-related community needs.

20. AmeriCorps*VISTA Logic Model
Community Need
Community Need =
Unresolved poverty-related problem or issue in
the community that your VISTA project will address

21. AmeriCorps*VISTA Logic Model
End Outcomes
Community Need
Intermediate Outcomes
Goals
Inputs
Activities
Outputs
End Outcomes=
The most significant changes achieved by your project
Answer the questions:
“So what”? “What changed?”

22. AmeriCorps*VISTA Logic Model
Community Need
Goals
Goals =
Broad purposes toward which your VISTA project is directed

23. AmeriCorps*VISTA Logic Model
Community Need
Goals
Inputs
Inputs=
Resources your VISTA project will need to engage in the identified activities and produce outputs and outcomes
(human, financial, organizational, community)

24. AmeriCorps*VISTA Logic Model
Community Need
Goals
Inputs
Activities
Activities=
What VISTA members do to address the community need and achieve desired results

25. AmeriCorps*VISTA Logic Model
Community Need
Goals
Inputs
Activities
Outputs
Outputs:
Reflect the amount of service completed by VISTA members.
Outputs do not answer the questions:
“what changed as a result of the work?” or “how did people or the organization benefit?”

26. AmeriCorps*VISTA Logic Model
Community Need
Goals
Inputs
Activities
Outputs
Intermediate Outcomes
Intermediate Outcomes=
Initial changes achieved by your project; milestones on the way to achieving end outcomes

27. AmeriCorps*VISTA Logic Model
End Outcomes
Community Need
Intermediate Outcomes
Goals
Inputs
Activities
Outputs
End Outcomes=
The most significant changes achieved by your project
Answer the questions:
“So what”? “What changed?”

28. AmeriCorps*VISTA Logic Model
Community Need
Intermediate Outcomes
End Outcomes
Goals
Inputs
Activities
Outputs
Program Planning
Intended Results

29. Logic Model for a Mentoring Program (Year 3)
Community Need
Intermediate Outcomes
End Outcomes
Goals
Inputs
Activities
Outputs
Children of incarcerated parents lack social and emotional support for positive youth development that can help to break the cycle of poverty.
Improved recruitment: more youth are served by more volunteer mentors.
Improved retention: more volunteer mentors complete service.
Improved program quality: more volunteer mentors form closer bonds with youth; mentoring relationships last longer.
Develop, enhance systems to recruit, train, and support volunteer mentors.
2 VISTAs
Supervisor
Program staff
Money
Partners
Assist with implementa-tion, assess performance of new systems, identify promising practices.
Policies and guidance revised for volunteer recruitment and sustain-ability systems.

30. Exercise
The Logic Model Puzzle

31. Exercise
Develop a Logic Model
for Your Project

32. End of Day Review + ∆ What Worked Well About Today?
How Could Tomorrow Be Made Even Better?

33. Day Two – Morning Session
This session will cover:
Moving from a Logic Model to the Project Plan
CNCS Requirements for VISTA
Components of a Project Plan
Indicators, Targets, Instruments

34. AmeriCorps*VISTA Performance Measurement Requirements
One project plan containing one or more goals
Identify activities and results for each goal
Results are written as outputs, intermediate outcomes, or end outcomes
Label 3-5 of these results as “performance measures”
One Output
One Intermediate Outcome
One End Outcome
Identify indicators, targets, and instruments for each performance measure

35. VISTA Project Plan Components
Goals (Same)
Activities (formerly “Objectives”)
Results (New)
Output(s)
Intermediate Outcome(s)
End Outcome(s)

36. AmeriCorps*VISTA Project Plan
Column A
Goals, Activities, and Results
Column B
Date(s)
Column C Project Reporting
GOAL 1
Activities
Results
Outputs
Intermediate Outcomes
End Outcomes
GOAL 2
 Activities
 Results
GOAL Etc.

37. AmeriCorps*VISTA Project Plan
Column A
Goals, Activities, and Results
Column B
Date(s)
Column C Project Reporting
GOAL 1
Activities
Results
Outputs
Intermediate Outcomes
End Outcomes
GOAL 2
 Activities
 Results
GOAL Etc.

38. AmeriCorps*VISTA Project Plan
Column A
Goals, Activities, and Results
Column B
Date(s)
Column C Project Reporting
GOAL 1
Activities
Results
Outputs
Intermediate Outcomes
End Outcomes
GOAL 2
 Activities
 Results
GOAL Etc.

39. From Logic Model to Project Plan
Community Need
Goals
Inputs
Activities
Outputs
Intermediate Outcomes
End Outcomes
Column A
Goals, Activities, and Results
Column B
Date(s)
Column C Project Reporting
Goal(s)
Activities
Results
Outputs
Intermediate Outcomes
End Outcomes

40. Multi-Year Goals, Annual Activities and Results
Goals are multi-year.
Activities and results may vary from one year to the next.
Each year the project builds on the previous year’s activities and results.
Only list activities and results for current year.

41. Exercise Identify One Goal for Your Project
For that Goal, identify the Activities for Year One and Possible Results:
Outputs
Intermediate outcome(s)
End outcome(s)

42. AmeriCorps*VISTA Project Plan
Column A
Goals, Activities, and Results
Column B
Date(s)
Column C Project Reporting
GOAL 1
GOAL 2
Activities
Results
Output “Performance Measure”
Indicator
Target
Instrument
Intermediate Outcome “Performance Measure”
End Outcome “Performance Measure”

43. AmeriCorps*VISTA Project Plan
Column A
Goals, Activities, and Results
Column B
Date(s)
Column C Project Reporting
GOAL 1
GOAL 2
Activities
Results
Output “Performance Measure”
Indicator
Target
Instrument
Intermediate Outcome “Performance Measure”
End Outcome “Performance Measure”

44. What is an Indicator?
Information you collect to determine progress toward achieving results
Concrete and measurable
Answers the question, “What is my evidence that something has occurred?”
Output indicator usually begins with “Number of...”
Outcome indicator usually begins with “Percent of...”

45. Example of Result and Indicators
Desired Result: Increase service to clients
Possible Indicators:
Greater number of clients served (number of clients served, compared to last year)
Clients receive more services (number of services provided, per client, compared to last year)
Clients receive services of greater value (dollar value of services provided, per client, compared to last year)

46. What is a Target?
Describes amount of change you anticipate achieving each year due to efforts of VISTA members.
Based on indicator – specific, measurable amount of change that will signify success
Outcome Target:
Who will experience change?
How many people will experience change?
How much change will occur (compared to baseline)?
Over what time period will change occur (one year)?

47. Formulating Annual Targets
Target describes amount/level of change you anticipate achieving in one year due to efforts of VISTA members.
Based on indicator (concrete, measurable)
Rely on experience and knowledge of community to judge how much change to expect in one year.
Keep in mind your resources.
Be realistic!

48. Indicator and Target Examples
Output
Outcome
Result
Volunteer database developed.
Improve volunteer base.
Indicator
Number of new volunteers recruited.
Target
One volunteer database developed.
300 new volunteers recruited.

49. What is an Instrument?
Document or form used to collect information from data source
Data source: Person, place or group that provides you with information to measure indicators
Examples: Attendance rosters, tally sheets, questionnaires, checklists
Adapt existing instruments or develop your own

50. Advice about Instruments
Make sure instrument will measure indicators
Consider who will administer instrument
Consider who will complete instrument
Consider when to administer instrument
Consider data accessibility

51. Project Plan: Performance Measures
Column A
Goals, Activities, and Results
Column B
Date(s)
Column C Project Reporting
GOAL
ACTIVITIES (Year 1)
RESULTS (Year 1)
Outputs:
Volunteer mentor recruitment systems are developed.
Agency personnel receive orientation in new systems “Performance Measure” Indicator: # agency personnel oriented Target: 2 F/T and 4 P/T agency personnel oriented Instrument: Orientation Log
Qtr 1–3
Qtr 4

52. Project Plan: Performance Measures
Column A
Goals, Activities, and Results
Column B
Date(s)
Column C Project Reporting
GOAL
ACTIVITIES (Year 1)
RESULTS (Year 1)
Outputs:
Volunteer mentor recruitment systems are developed.
Agency personnel receive orientation in new systems “Performance Measure” Indicator: # agency personnel oriented Target: 2 F/T and 4 P/T agency personnel oriented Instrument: Orientation Log
Intermediate Outcome: Agency personnel understand how to use new systems. “Performance Measure” Indicator: # personnel who correctly identify staff roles and who can find info in new database Target: 5 personnel can recruit, screen and train mentors Instrument: Mentoring System User Checklist
Qtr 1–3
Qtr 4

53. Project Plan: Performance Measures
Column A
Goals, Activities, and Results
Column B
Date(s)
Column C Project Reporting
GOAL
ACTIVITIES (Year 1)
RESULTS (Year 1)
Outputs:
Volunteer mentor recruitment systems are developed.
Agency personnel receive orientation in new systems “Performance Measure” Indicator: # agency personnel oriented Target: 2 F/T and 4 P/T agency personnel oriented Instrument: Orientation Log
Intermediate Outcome: Agency personnel understand how to use new systems. “Performance Measure” Indicator: # personnel who correctly identify staff roles and who can find info in new database Target: 5 personnel can recruit, screen and train mentors Instrument: Mentoring System User Checklist
End Outcome: Agency personnel use new systems. “Performance Measure” Indicator: # agency personnel using # new systems Target: Four agency personnel, two systems Instrument: Year-end Agency Interview
Qtr 1–3
Qtr 4

54. Identifying Indicators, Targets and How Measured for Outcomes
Exercise
Identifying Indicators, Targets and How Measured for Outcomes

55. Add Enhancements to Your Project Plan Results
Exercise
Add Enhancements to Your Project Plan Results

56. Day Two – Afternoon Session
This session will cover:
Selecting Instruments
Data Collection Strategies
Data Aggregation and Analysis
Reporting

57. Strategies to Ensure Successful Reporting
From the beginning…
Align work plan and instruments
Plan for collecting the information
Understand how to analyze data
Anticipate how you will report

58. What Do I Report On? Actual Activities Actual Output
Actual target as compared to anticipated target
How data collected
Actual Intermediate Outcome
Actual End Outcome

59. Data Collection Issues
Can you access the data source (beneficiaries, organizations)?
Does an instrument (site or project) exist?
Are they willing to give you data? (burden, capacity)
Can you get the data when you need it?
Are your helpers (e.g. VISTAs) trained?
Do you have a mechanism to collect summarized data from sites?

60. Methods Logs, Tally Sheets, Attendance Rosters
Survey, Questionnaire (Check list, Index, Scale)
Interview (Interview Guide)
Service Recipients Interview
Expert/Provider/Key Informant
Observation (Observation Guide)
Case Study (Measurable change of a case over time)
Focus Group (Focus Group Guide)
Secondary Data (Existing records and information)
Test / Experimental Designs

61. Instrument Considerations
Connect to Project Plan results
Measures the identified PM
Aim for appropriate method
Measures the correct population (data source)
Lower burden of use
e.g. Captive audience, (part of project)
Balance between “best” method and “manageable” method
Assure that sites have instruments or summary tool
Connect to work plan results
Does instrument have anything to do with work plan. Is it a great instrument about tutoring but the volunteers are working on reading not math
Measure the correct population (data source)
Measure the identified PM
Lower burden of use. Do stations/volunteers have capacity, interest, time to complete?
Aim for appropriate method
Assure that instrument exists. Or instrument may exist but measure the wrong thing or states have instrument but don’t use them. The exercise we are going to do next is from one of the packet that tries to address a few of these concerns. The packets are…

62. Instrument and Data Collection Review and Discussion
Exercise
Instrument and Data Collection Review and Discussion

63. Data Analysis Quantitative Qualitative Frequencies Ranges Percents
Means
Pre-post comparison
Qualitative
Content analysis
Pre-existing themes
Emergent themes

64. Data Aggregation and Analysis
Exercise
Data Aggregation and Analysis

65. Reporting Tips Provide evidence. Avoid hearsay and generalizations.
Be candid about your efforts. Don’t claim too little. Don’t claim too much.
Tie report to Project Plan. State results in same format as original target in Project Plan.
Pay attention to enhancements: indicator, target, and instruments

66. Report Template Reporting Format for Each Performance Measure
Actual Target:
   1) Compare the actual to the anticipated target stated in Column A.
   2) Met target? If target not met, explain.
   3) Include any additional information about your result.
How Data Collected:
   1) Describe how the result was measured.

67. Reporting Each Result Actual Target:
   1) Compare the actual target to the anticipated target stated in Column A.
Interpret results; explain what it means
2) Met target? If target not met, explain.
Include numbers not just percentages.
Explicitly state that you did (not) meet target
   3) Include any additional information about your result.
Describes challenges, next steps

68. Describing Your Results
Actual Target:
 1) Compare the actual target to the anticipated target stated in Column A. [Always include an actual number]
“10 interview respondents (86% of collaborative) identified at least three beneficial practices resulting from the partner collaboration. This exceeds our target of one project improvement. The top two included an increase in clients served and…”

69. Reporting Format for Each Result
How Data Collected:
   1) Describe how the result was measured.
What was used to track the result?
If applicable, how was the instrument administered? (When was it administered? How often?)
If applicable, include total number approached as well as those who responded.

70. Describing Your Results
How Data Collected:
1) Describe how the result was measured
“During the 2nd and 3rd weeks of April, the Project Supervisor, and two VISTAs conducted telephone interviews during the afternoon and evening…”
“...Of the 12 collaborative partners called, 10 completed the interview that assessed their practices resulting from the collaborative…”

71. Describing Your Work Before Results Are Available
“...We don’t have any results yet.”
What if you don’t have any data available yet?

72. What do I Report When I Have “Nothing to Report” Yet?
We are…..
Development/piloting of instruments
Introducing instrument to sites
Gaining access to data
Training data collectors
Challenges and proposed solutions
Administering pre-survey data. (If you have baseline data, report it early!)
Say something! Don’t wait.

73. Describing Your Work Before Results Are Available (Improved)
  How Data Collected:
   1) Describe how the result was measured
“...In preparation for implementing performance measurement, we drafted a telephone interview protocol containing six questions designed to assess partner collaborative practices. During the first week of March, two collaborative members reviewed the draft and six VISTAs were trained to conduct interviews...”

74. Day Three – Morning Session
This session will cover:
Reporting
Using the Results of Performance Measurement
eGrants and VISTA
Closing - Receive box lunch and departure

75. Progress Report Critique Using Checklist
Exercise
Progress Report Critique Using Checklist

76. Communicating Success
Exercise
Communicating Success
Ways to Use PM Results to Strengthen and Promote Your Project

77. Additional Resources:
Contact your CNCS State Office
Contact Project STAR