1. Evaluation in Foundations
Welcome!
Evaluation in Foundations
October 26, 2012
AEA 2012

2. Panel
Ruth Bowman: Ruth is an independent consultant and an affiliate of the Minnesota Evaluation Studies Institute at the University of Minnesota. Most recently, Ruth has held leadership positions within the group of Margaret A. Cargill Philanthropies. She has also served as adjunct faculty at the Hubert H. Humphrey School of Public Affairs (U of M). She served as the Assistant Editor of New Directions for Evaluation from 2004 – 2006.
Becky Stewart: Becky has been Research and Evaluation Director with the Improve Group for eight years, a consulting firm in the Twin Cities, MN. Her client history involves programs ranging from small, local pilots, to statewide projects, to national initiatives and international programs.
Randi Ilyse Roth: Randi Ilyse Roth has been the executive director of the Otto Bremer Foundation since May Before joining the Foundation, Roth served as the independent, court-appointed monitor in a civil rights lawsuit brought by African-American farmers against the U.S. Department of Agriculture. Prior to that, she served as a staff attorney and then as executive director of the Farmers’ Legal Action Group.

3. Foundation Evaluation Systems: A Case Study
Designing Evaluation Systems for a Small Family Foundation

4. Case Study: The Setting
Family foundation with about $3m in annual grant-making
Hired to help the Foundation determine its approach to evaluating its impact
Products:
Grantee report
Foundation “metrics”
Framing questions guided the process
Mission is to supporting efforts that address the unmet human and social needs of individuals, families, and communities that have the least access to resources.
Additional context: Foundation had recently been through a planning period; strategies were pre-determined

5. Grantee Reports and Foundation Metrics
Framing Questions
Values
Grantee Reports and Foundation Metrics
Scope
Resources
Goal

6. What are key VALUES for the foundation?
Why is the foundation doing evaluation? What is the relationship with grantees? What is the foundation’s desired role in the community?

7. What are the GOALS of evaluation efforts?
Accountability: To whom? What are their mandates and expectations? Learning: By whom? About what? Planning: Over what time frame? What decisions need to be supported with data? Funding decisions: To inform ongoing funding decision? Streamlining grantee obligations?
Accountability: To whom? What is mandated? Expected? Needed?
Learning: By whom? What is the learning agenda for the Foundation? TO what extent does the Foundation desire to share learning?
Planning: What information does the Foundation need to do planning? Annual? Longer term? What information is needed?
Continuing support: Can/should the Foundation use information to determine ongoing support? Can the information help the Foundation understand when to STOP funding? Should application/reporting systems be streamlined/coordinated?

8. What are the RESOURCES for evaluation efforts?
What data collection points or systems are available? What is staff capacity to gather, analyze and report on findings? What available data is relevant and accessible? What would be an acceptable burden on grantees?

9. What is the SCOPE of evaluation efforts?
Where does the foundation expect to see change because of their support? What impact is realistic to expect, given grant amount and term? What level of evidence is needed? Do different aspects of the portfolio demand different levels of evaluation? What, if anything, makes sense to measure across different programs in the portfolio?
Unit of analysis – Where do you expect to see change because of your funding? Community? Organization? Program? Participant? What is the Foundation’s theory of change?
Funding/Expected Impact – What can realistically be achieved with the Foundation’s level of support/the scope of the program?
Evidence needed – Is the Foundation support seeking to establish best practices? Conclusively attribute impact to a program model?
Program portfolio - What kind of evaluation questions are appropriate for the kinds of program the Foundation funds?
Confounding factors, environmental trends

10. Grantee Report Content
Monitoring Milestones/Elements Organization Information Financial strength Partnerships Environmental scanning/Trends Needs
Monitoring Milestones/Elements
From this section, suggested elements to inform a board report are:
Percent of programs meeting targets
Average scores for program monitoring items
Highlight informative, interesting select success stories
Highlight major challenges to inform (1) environmental review and (2) funding strategy discussions
Organization Information
Financial strength:
Average scores for financial health item to provide a summary of how organizations are faring in their finances
Provide some examples of how Phillips Family Foundation funding allows grantees to leverage further support; will help Phillips Family Foundation to (1) clarify its understand of impact and (2) understand the role the Foundation plays in the program life cycle
Percent of organizations ranking funding items as (1) hinder ability to work towards goals and (2) accelerating ability to work towards goals
Partnerships:
Average score on the partnership item
Select examples from top and bottom categories to illustrate differences
Note any staff observations on themes of partnership impacts on program effectiveness
Percent of partnerships rated at “accelerating ability to work toward goals”
Use overall results to determine if there if Phillips interest in convening could help strengthen partnerships
Environmental Scan:
Average score on the item regarding emerging trends to describe organizations’ strength in environmental scans
Foundation staff selection of interesting/commonly described trends
Suggestions to inform work planning:
Compilation of ways that the Foundation can be helpful aside from funding (list of interesting ideas and themes)
Extent to which the Foundation is already doing some of the suggested items
Alignment of suggestions to Foundation resources and strengths
Program Model Development
Notes about program model adjustments, best practices and target populations for the Foundation’s ongoing learning
Average score for this section
Note any trends in populations for whom program models are not working well
Common themes of responses to the items around sustainability
Alignment with Foundation resources and strengths
Extent to which these needs call for the participation of other partners and consideration of Foundation role in supporting that involvement

11. Foundation Metrics (excerpt)
The Foundation will spark innovation in mission-aligned programs
Strategy
Support programs that are trying promising new approaches
Indicators
Programs are trying new ideas based on strategic thinking
Foundation is one of few, early funders
Programs show signs of successful implementation
Foundation shares lessons learned and new ideas

12. Randi Ilyse Roth Executive Director
Otto Bremer Foundation