1 Instruments and Data Collection New Mexico AmeriCorps April 20, 2006 Sue Hyatt, Project STAR Coach

2 Agenda What are instruments? What are instruments? Measuring outputs and outcomes Measuring outputs and outcomes Linking indicators and targets to instruments Linking indicators and targets to instruments Data Collection Issues Data Collection Issues PM Plan PM Plan

3 Performance Measurement Six Steps 1. Develop Logic Model & Project Plan 2. Develop PM Plan 3. Develop and Test Instruments 4. Collect Data 5. Analyze Data 6. Report Results Continuous Program Improvement

4 Types of Performance Measures Outputs Outputs Intermediate Outcomes Intermediate Outcomes End Outcomes End Outcomes

5 What is an Instrument? Document or form used to collect information from data source Document or form used to collect information from data source Data source: Person, place or group that provides you with information to measure indicators Data source: Person, place or group that provides you with information to measure indicators Examples: Attendance rosters, tally sheets, questionnaires, checklists Examples: Attendance rosters, tally sheets, questionnaires, checklists Adapt existing instruments or develop your own Adapt existing instruments or develop your own

6 Advice about Instruments Make sure instrument will measure indicators Make sure instrument will measure indicators Consider who will administer instrument Consider who will administer instrument Consider who will complete instrument Consider who will complete instrument Consider when to administer instrument Consider when to administer instrument Consider data accessibility Consider data accessibility

7 Instrument Considerations Connect to results –Measures the identified PM Aim for appropriate method –Measures the correct population (data source) Lower burden of use –e.g. Captive audience, (part of project) –Balance between best method and manageable method Assure that sites have instruments or summary tool

8 Exercise What Does the Instrument Measure?

9 Indicators and Targets For each result (output, intermediate outcome and end outcome), you need to specify: Indicator Target Instrument COMMON ISSUE: Indicators and targets are NOT tied to the specific instrument being used!

10 What is an Indicator? Information you collect to determine progress toward achieving results Information you collect to determine progress toward achieving results Concrete and measurable Concrete and measurable Answers the question, What is my that something has occurred? Answers the question, What is my evidence that something has occurred? Output indicator usually begins with Number of... Output indicator usually begins with Number of... Outcome indicator usually begins with Percent of... Outcome indicator usually begins with Percent of...

11 What is a Target? Describes of change you anticipate achieving each year Describes amount of change you anticipate achieving each year Based on indicator – specific, measurable amount of change that will signify Based on indicator – specific, measurable amount of change that will signify success Outcome Target: Outcome Target: will experience change? –Who will experience change? people will experience change? –How many people will experience change? change will occur (compared to baseline)? –How much change will occur (compared to baseline)? –Over will change occur (one year)? –Over what time period will change occur (one year)?

12 Advice about Targets Consider available program resources. Consider available program resources. Consider community context. Consider community context. First year targets may be educated guesses. First year targets may be educated guesses. Be prepared to adjust targets based on experience. Be prepared to adjust targets based on experience. Targets should be ambitious but also realistic. Targets should be ambitious but also realistic. Targets are based on indicators and the content of the instrument. Targets are based on indicators and the content of the instrument.

13 Exercise How Targets are Built from Instruments: Tutoring Data Aggregation Exercise

14 Data Collection Issues Can you access the data source (beneficiaries, organizations)? Does an instrument (site or project) exist? Are they willing to give you data? (burden, capacity) Can you get the data when you need it? Are your helpers (e.g. members) trained? Do you have a mechanism to collect summarized data from sites?

15 Methods Logs, Tally Sheets, Attendance Rosters Survey, Questionnaire (Check list, Index, Scale) Interview (Interview Guide) Service Recipients Interview Expert/Provider/Key Informant Observation (Observation Guide) Case Study (Measurable change of a case over time) Focus Group (Focus Group Guide) Secondary Data (Existing records and information) Test / Experimental Designs

16 Exercise Program Instrument and Data Collection Review and Discussion

17 Additional : Additional Resources: Contact Project STAR Contact Project STAR