1. Education Performance Measures Session

2. Session Overview Combination of presentation and interactive components Time at the end of the session for Q&A Use notecards on table to write questions

3. Objectives Strengthen understanding of the three elements integral to a strong performance measurement plan: –Data Collection –Data Management –Data Reporting Experience planning for the data collection process

4. Agenda

5. How does my program demonstrate impact? What evidence do I need to collect to show I achieved my selected outcome ?

6. 6 Amount of service provided (i.e. people served). Reflect the changes or benefits that occur. How does my program demonstrate impact?

7. Work Plan Refresher The Service Activity Description should explain what the Senior Corps volunteers are doing in a way that shows how they will achieve an output and outcome. It should include: who the beneficiaries are; what the volunteers will be doing with the beneficiaries; how often volunteers will provide the service; for how long, and; where the service will take place.

8. Work Plan Refresher 25 Senior volunteers will mentor 75 children eligible for free or reduced lunch for at least 1 hour a week for at least 6 months of the school year. Mentoring will including one-on-one help with homework, reading together, relationship building, and providing encouragement and attention in a supervised junior high or middle school setting.

9. Work Plan Refresher The Instrument Description should: give the name of the instrument; briefly describe who will collect the data; from whom the data will be collected, and; when it will be collected.

10. Work Plan Refresher An Academic Engagement Survey will be given to teachers by station supervisors at the beginning and end of the school year. If the mentoring relationship ends prior to the end of the school year but lasts 6 months or longer, the post survey will also be given. The survey measures improvement in social and academic engagement.

11. Five Data Quality Elements A Standard Process to Consider Data Validity Completeness Consistency Accuracy Verifiability

12. Validity: Data Relevance Ask yourself: –Is the data relevant? –Are your reported items consistent with the approved goals of the current grant? –Are you measuring what you intended to measure? Example ED27 Outcome: Number of students in grades K-12 that participated in the mentoring or tutoring or other education program, including CNCS-supported service learning, who demonstrated improved academic engagement. Data Instrument: Teacher Pre/Post Survey Questions in Survey: Did you have a long wait for a volunteer? Are you happy with the service?

13. Data Completeness Ask yourself: –Does the reported data contain enough information to represent performance measure activities? –Did the reported data come from all stations and/or a random sampling of volunteers/ service recipients? Example ED27 Outcome: Number of students in grades K-12 that participated in the mentoring or tutoring or other education program, including CNCS-supported service learning, who demonstrated improved academic engagement. Data Instrument: Teacher Pre/Post Survey Survey Distribution: Jane provides the survey to 7 of 12 stations because she is running behind on her schedule

14. Consistency: Standard Protocols and Procedures Ask yourself: –Do you clearly document data collection methods? –Do you use the same data collection method for all sites? –Do you train data collectors? –Have you taken steps to eliminate bias? Example ED27 Outcome: Number of students in grades K-12 that participated in the mentoring or tutoring or other education program, including CNCS-supported service learning, who demonstrated improved academic engagement. Data Instrument: Teacher Pre/Post Survey Survey Distribution: Jane provides the same survey to all stations at the same station training

15. Accuracy: Data Free From Error Ask yourself: –Do you have a plan to collect and review data? –Do you have documentation on your data collection plan? –Does the data vary significantly in unexpected ways? Example Data Collection Approach: Collect data from all 12 stations Data Received: 150 surveys Data Check: Jane’s colleague reviews the data to check her calculations, as per her organization's Data Collection Handbook

16. Verifiability: Data Collected According to Plan Ask yourself: –Did the grantee double check that the data collected followed the plans and procedures? –Did the grantee use controls throughout data collection? –Does the grantee regularly review and update its data quality control procedures? Example Situation: Jane submits her PPR and leaves for vacation; her colleagues cannot validate the data she input into it.

17. Doing Data Collection: Identifying Your Source List pros and cons of possible data sources Identify a preferred data source Consider back-ups

18. Doing Data Collection: Considering Methods Survey Interviews Observation Standardized tests Tracking sheets Focus groups Diaries or journals Secondary data

19. Method Requirements Guidance on how to measure and collect data is provided

20. Doing Data Collection: Creating a Schedule Identifies who will collect data, using which instrument, and when –Include stakeholders in planning –Include time for collecting, analyzing, and reporting data Share with team to keep everyone informed

21. Doing Data Collection: Selecting a Final Measurement Instrument For CNCS priorities and performance measures, look for instruments by program: –Go to http://www.nationalservice.gov/resources/performance- measurement/senior-corps http://www.nationalservice.gov/resources/performance- measurement/senior-corps –Programs and projects can also look anywhere they like to find instruments: Use Internet search engines Leverage experience and known best practices in your field Talk to others within you professional network to find out what they are using

22. Doing Data Collection: Selecting a Final Measurement Instrument You may have a pre-existing tool or want to modify a resource found on the Knowledge Network Consider: How else might you use this data? What may other stakeholders want? How will this affect your data collection and instrument design? Keep in mind the 5 data quality elements when developing or modifying a measurement instrument.

23. Learning Activity

24. Data Management Now what? Consider: –Potential usage –Access –Storage and maintenance

25. Data Management Data collection procedures and guidelines Source documents for data (e.g. sign-in sheets, activity logs, attendance logs, client tracking database, tracking forms, surveys, etc.) Formalize data sharing agreements in MOUs

26. Data Reporting

28. Q&A

29. Thank you Stay current by visiting the Knowledge Network! http://www.nationalservice.gov/resources/performanc e-measurement/senior-corps http://www.nationalservice.gov/resources/performanc e-measurement/senior-corps