1. Association for the Study and Development of Community Maximizing Impact: Helping Funders Get Smart About Evaluation Kien S. Lee Association for the Study and Development of Community 438 N. Frederick Avenue, Suite 315, Gaithersburg, MD 20877 (301) 519-0722, ext. 108  kien@capablecommunity.com Maine Philanthropy Center October 8, 2008

2. Association for the Study and Development of Community I have to rewrite the evaluator’s report because it is too jargony and not useful. After reading it, I still have no idea how we are doing. I’m told an internal evaluation is not objective The evaluator told us that our program was not ready for evaluation We submit all this information to the funder, I’m not sure anyone actually reads them The funder had unrealistic expectations. There is no way we can measure these outcomes given the constraints I want to evaluate a public education campaign, how much will it cost?

3. Association for the Study and Development of Community American Evaluation Association Guiding Principles www.eval.org A.Systematic Inquiry B.Competence C.Integrity/Honesty D.Respect for People E.Responsibilities for General and Public Welfare

4. Association for the Study and Development of Community Useful Terms to Know About Evaluability Assessments Prospective Evaluations Retrospective Evaluations Formative, Process, Implementation Evaluations Summative, Outcome, Impact Evaluations Meta-Evaluations

5. Association for the Study and Development of Community Readiness Clear and realistic goals, objectives, and strategies Clear understanding of how strategies will lead to anticipated outcomes Capacity to collect the data needed Commitment to learning Resources to hire an evaluator

6. Association for the Study and Development of Community Internal or External Evaluator? Purpose Roles in relation to other parts of the organization Knowledge and skills Existing systems Resources

7. Association for the Study and Development of Community What Do You Look For in An Evaluator? Knowledge of Evaluation Philosophical and Methodological Orientations Commitment to Professional Development Cultural Competency Special Skills and Experiences Required for the Evaluation

8. Association for the Study and Development of Community Multiple Hats Worn by An Evaluator Educator Conflict Resolution Manager Learning Facilitator Evaluator Advocate for Social Justice

9. Association for the Study and Development of Community What if you can’t afford a full- fledged evaluation? Identify and monitor only those outcomes and indicators that are critical Engage external evaluator to review your logic model, measures, and/or instruments Engage external evaluator to double- check quality of data and analyses Engage external evaluator to develop monitoring system and train staff

10. Association for the Study and Development of Community Ask the following questions over and over again: What do I want to know? Who else needs to be involved? Strengths and limitations? How will the findings be used? When will the findings be needed? What is the best way to communicate the findings?

11. Association for the Study and Development of Community Merit or worth? Ways to improve our work? Compliance of grantees? Contribute to the field? Inform policies? Patton, M. (1997). Utilization-focused Evaluation. Thousand Oaks, CA: SAGE Publications Mark, M., Henry, G., & Julnes, G. (2001). Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Program. San Francisco, CA: Jossey-Bass. What do I want to know from the evaluation?

12. Association for the Study and Development of Community Who else needs to be involved? Internal Stakeholders Board and Executive Leadership Staff External Stakeholders Grantees Community Leaders Partners

13. Association for the Study and Development of Community What are the strengths and limitations of the evaluation? What are the theories and assumptions underlying the evaluation? What are the pros and cons of the evaluation design? What can I say or not say about the findings and their implications?

14. Association for the Study and Development of Community How do I plan to use the findings? What decisions do I need to make based on the results? What type of changes am I prepared to make? What changes, if any, can be shared with some people and not with others?

15. Association for the Study and Development of Community When do I need the findings? What is the timeframe for making decisions? When do my supervisor, board members, community leaders, policymakers, or others want to know about the evaluation findings?

16. Association for the Study and Development of Community Where will the evaluation findings be shared? Staff meetings, press conference, board meetings, public hearings, etc. Who is the most effective messenger for the findings? Evaluator, you, your supervisor, someone else What is the best way to communicate the findings?

17. Association for the Study and Development of Community Examples of different forms of communicating evaluation findings: Short, written communications Reports Executive summaries Newsletters, bulletins, briefs, and brochures Verbal presentations Video presentations Posters Working sessions Photography Cartoons Poetry Drama Preskill, H., Torres, R., & Piontek, M. (2004). Evaluation Strategies for Communication and Reporting. Thousand Oaks, CA: SAGE Publications

18. Association for the Study and Development of Community Logic Modeling Is a graphic illustration of your theory of change Makes explicit underlying assumptions about the change process Specifies connections between inputs, strategies, activities, immediate outcomes, intermediate outcomes, and long-term outcomes Provides a process for reflection and improvement

19. Association for the Study and Development of Community The logic model in non-jargon terms! Provides the story line for your grantmaking and evaluation Pushes you to ask and answer the question, “If you do A, then what do you expect to happen? If your answer is B, tell me why you think B will occur.” A picture that lays out your investment hypothesis

20. Association for the Study and Development of Community Assessing the Foundation’s Effectiveness and Impact Start with the overall theory of change and logic model for the entire foundation Then, develop logic models for each program priority area Then, develop logic models for each program or initiative within priority area Check to ensure that each logic model supports the one “above” and “below” it

21. Association for the Study and Development of Community Inputs Immediate Outcomes Long-term Outcomes Contextual Conditions Example of A Simple Logic Model Strategy #1 Strategy #2 Activities Intermediate Outcomes

22. Association for the Study and Development of Community Indicators and Benchmarks Road signs that you are moving in the right direction Shows the change or progress over a defined time period Directly related to your anticipated outcome Can be in the form of a number or statistic where you can see an increase or decrease or a first- time practice or procedure that marks a significant change

23. Association for the Study and Development of Community Develop uniform indicators across grantees that demonstrates funder’s effectiveness Establish these uniform indicators ahead of time before grant is awarded Build data requirements into progress reporting template Provide technical assistance to grantees to track the uniform indicators and indicators specific to their effort In An Initiative with Multiple Grantees and Sites….

24. Association for the Study and Development of Community Measuring Context Rival explanations or counterfactuals Qualitative description (converging evidence) Statistical methods (multiple regression, hierarchical linear modeling)

25. Association for the Study and Development of Community How can we best learn from evaluation? Conduct reflection meetings regularly Be prepared for different scenarios Set up appropriate structures and processes for engaging all the stakeholders (be creative!) Communicate mid-course adjustments that resulted from learnings Identify promising practices and learning assets

26. Association for the Study and Development of Community Why do some evaluation reports sit on the shelf? The evaluation purpose is not connected to learning The report is too technical and jargon-filled The report does not address the learning questions The analysis is weak, confusing, or inappropriate The evaluation report is too late to be useful The findings are deemed too controversial to discuss openly

27. Association for the Study and Development of Community What should the evaluation report look like? Things to consider based on audience needs:  Knowledge of technical language  Format  Length  Style  Number and type of reports (e.g., interim and final; internal and external)

28. Association for the Study and Development of Community Characteristics of a Useful Evaluation Report Flows like the logic model Jargon-free, well-written Findings first, evidence second Uses a variety of illustrations Answers key questions Provides recommendations for mid-course adjustments or future improvements Includes an executive summary

29. Association for the Study and Development of Community Don’t forget – part of the reflection should focus on the evaluation itself!  What would you have done differently given data and resource constraints?  How could the reflection and learning process be improved?  Did we give appropriate feedback to the evaluator?

30. Association for the Study and Development of Community Key Take-aways Determine the purpose of your evaluation and the best evaluator for the job Engage the appropriate stakeholders early on Plan for the reporting, communication about the findings, and learning process at the beginning of the evaluation Assess the context for presenting the findings Know and apply AEA’s guiding principles Hire an external evaluator for certain tasks Anything else?