1. Data Analysis for Evaluation Eric Graig, Ph.D.

2. Slide 2 Innovation Network, Inc. Purpose of this Training To increase your skills in analysis and interpretation of the information you collect To increase your ability to report your findings to a variety of audiences To learn how to make evaluation results actionable

3. Slide 3 Innovation Network, Inc. Three Steps to Data Analysis Analyze Results Communicate Findings Use Findings for Program Improvement

4. Slide 4 Innovation Network, Inc. Keep it Simple Aim for a systematic effort Keep your audience in mind Pay attention to the usability of your evaluation report

5. Slide 5 Innovation Network, Inc. Two Kinds of Data Quantitative Qualitative

6. Slide 6 Innovation Network, Inc. Quantitative Data Data that is numerical, counted, or compared on a scale Demographic data Answers to closed-ended survey items Attendance data Scores on standardized instruments

7. Slide 7 Innovation Network, Inc. Qualitative Data Narratives, logs, experience Focus groups Interviews Open-ended survey items Diaries and journals Notes from observations

8. Slide 8 Innovation Network, Inc. Some Examples Quantitative Qualitative

9. Slide 9 Innovation Network, Inc. Summary Statistics Measures of Central Tendency Mean Median Range Frequency distributions Percentage distributions

10. Slide 10 Innovation Network, Inc. Frequency Distributions To what extent did you increase your skills in putting together a household budget? A lotSomeA littleNot at all Women (N=30)14952

11. Slide 11 Innovation Network, Inc. Percentage Distributions To what extent did you increase your skills in putting together a household budget? A lotSomeA littleNot at all Women (N=30)46%30%17%7%

12. Slide 12 Innovation Network, Inc. Graphing Frequency Data How did you first hear about the web-site? Court Referral Social Worker Friend or Acquaintance Web Search Engine Librarian Newspaper Story Other

13. Slide 13 Innovation Network, Inc. Means and Medians History95 English96 Biology93 Latin92 Math98 Music94 Gym40 Mean = 87 Median = 94 Math98 English96 History95 Music94 Biology93 Latin92 Gym40

14. Slide 14 Innovation Network, Inc. Some Subtleties 40505594100 40929394959698 Mean = 81 Mean = 87

15. Slide 15 Innovation Network, Inc. Histograms

16. Slide 16 Innovation Network, Inc. Cross Tabulations

17. Slide 17 Innovation Network, Inc. Graphing comparisons Satisfaction with Services 0 5 10 15 20 25 30 35 40 ABCDE Clinic Name Satisfaction Score

18. Slide 18 Innovation Network, Inc. Some subtleties Satisfaction with Services 0 2 4 6 8 10 12 14 16 ABCDE Clinic Satisfaction Score Staff Advice Facility

19. Slide 19 Innovation Network, Inc. Some more subtleties Satisfaction with Services 0 2 4 6 8 10 12 14 16 StaffAdviceFacility Satisfaction Component Satisfaction Score A B C D E

20. Slide 20 Innovation Network, Inc. A Note on Quantitative Data Entry Several approaches Paper and pencil tally Word processing table Spreadsheet Custom database

21. Slide 21 Innovation Network, Inc. Qualitative Data Textual data Interview transcripts Case notes/ clinical notes Open-ended survey questions Photographs Video recordings

22. Slide 22 Innovation Network, Inc. Qualitative Analysis Identify themes or patterns Pre-set themes Determine as you go Tag text segments with themes

23. Slide 23 Innovation Network, Inc. Sample data sheet Site Interviewee Date Tape number Length Question number

24. Slide 24 Innovation Network, Inc. Exercise Break into small groups Using the sheet provided search for snippets of text that relate to the codes provided Identify any other themes that may be important that are not provided on the sheet

25. Slide 25 Innovation Network, Inc. Communicating Findings “Information that is not effectively shared with others will not be effectively used.” Source: Building a Successful Evaluation Center for Substance Abuse Prevention

26. Slide 26 Innovation Network, Inc. Audience Staff Board Funders Partners Other agencies Public

27. Slide 27 Innovation Network, Inc. Reporting Formats Written report Data book Flyer Formal oral presentation Round table with partners/ stakeholders Poster Web site

28. Slide 28 Innovation Network, Inc. Whatever strategy you choose: Link findings to the program’s outputs and desired outcomes Include successes and challenges Support your claims with data Acknowledge knowledge gaps

29. Slide 29 Innovation Network, Inc. Organizing Findings Overview Describe program/ describe evaluation Program Implementation What did you do, and how well did you do it? Program Outcomes: What were the changes that occurred? Lessons learned Sub-group differences

30. Slide 30 Innovation Network, Inc. Overview Section Your organization Your program Purpose of the evaluation Evaluation design Structure of the report

31. Slide 31 Innovation Network, Inc. Program Implementation Characteristics of program participants Demographics Outputs Number of sessions held Number of participants served Partnerships formed Satisfaction with program

32. Slide 32 Innovation Network, Inc. Program Outcomes What has changed? What do participants now know? What are they now able to do? What do they actually do? How are there lives different?

33. Slide 33 Innovation Network, Inc. Lessons Learned Which of the program’s assumptions were validated, which need to be reconsidered? Does the program affect different kinds of people differently? Are results better with some groups? How will findings be addressed? What new hypotheses did the evaluation generate? How did the evaluation work?

34. Slide 34 Innovation Network, Inc. Additional Resources One-on-one technical assistance Synchronous course Asynchronous material InnoNet website www.innonet.orgwww.innonet.org