1. Using Evaluation Training to Create Change: The Influence of the Evaluation Fellows Program Amelia E. Maynard Jean A. King

2. Using Evaluation Training to Create Change: The Potential of the Evaluation Fellows Program Jean A. King Amelia E. Maynard

3. EFP partners at the U of MN
Evaluation Fellows Program
EFP partners at the U of MN
The Evaluation Studies track in the Department of Organizational Leadership, Policy, and Development in the College of Education and Human Development
The Consortium on Children, Youth, and
Families (CCYF)
King & Meyer 2011 AEA Presentation

4. Overview
Brief description of the Evaluation Fellows Program (EFP) at the University of MN
Analysis using three frameworks
Evaluation influence
Evaluation capacity building (ECB)
Evaluation field building (EFB)
Questions remaining/next steps

5. The evaluation fellows program (EFP)
An innovative training experience in Minnesota’s Twin Cities
The evaluation fellows program (EFP)

6. Innovative EFP features
Focus on evaluation AND a specific field of practice
Involvement of people from a variety of roles:
Practitioners
Evaluators
Funders
Policy makers

7. EFP operates in the “performance sweet spot”
Evaluation
knowledge
Program
knowledge
PERFORMANCE SWEET SPOT
D. Blyth

8. The intersection of people
in four roles
Practitioners
Evaluators
Policy Makers
Funders

9. Overall EFP outcomes
1. Build the capacity of individual leaders to assess the impact of reform efforts
Generate and apply knowledge about the special evaluation challenges inherent in attempting change in a specific field
3. Create knowledge about how to engage people in evaluating reform efforts in ways that will advance public dialogue, policy change, and improvement
4. Apply participants’ new knowledge to reform and policy development

10. EFP cohorts to date
Cohort 1 ( ) - Youth development program evaluation (out of school time)
Cohort 2 ( ) - Evaluation of educational reform
Cohort 3 (proposed) - Evaluation of a public health program area

11. Frameworks for analy-zing the impact of efp
Different ways to think about this innovative training
Frameworks for analy-zing the impact of efp

12. Frameworks to shape our thinking about the EFP
Evaluation influence
Evaluation capacity building (ECB)
Evaluation field building (EFB)

13. Evaluation influence
Evaluation “influence” – integrated understandings of the broader consequences and impact of evaluations
Kirkhart (2000)
Henry & Mark (2003), Mark & Henry (2004)
Influence is “the capacity or power of persons or things to produce effects on others by intangible or indirect means” (Kirkhart, 2000)

14. “Integrated theory” of influence
Dimensions:
Intention
Unintentional
Intentional
Source
Process
Results
Time
Immediate
End-of-cycle
Long-term
Kirkhart, 2000

15. Schematic “theory” of evaluation influence
Intermediate and long-term evaluation outcomes leading to social betterment (Mark & Henry, 2004)
Cognitive/affective
Motivational
Behavioral
Salience
Personal goals
Individual practice
Opinion valence
Social reward
Collaborative practice
Descriptive norms
Incentives
Program future
Agenda setting
Market forces
Policy adoption
Program future is continuation, termination or expansion
King & Meyer 2011 AEA Presentation

16. Applying influence to the EFP
Traditional notions of influence are not quite right when applied to the EFP
It’s not about the process of conducting an evaluation (Evaluation 101 or 304)
It’s not (only) about wanting key people to use the evaluation’s results
It IS about evaluation understanding (could lead to changes in thinking, motivation, behavior)

17. For thinking about the EFP
Influence of the evaluation enterprise broadly framed
Our question:
Does collaborative training and understanding of the evaluation process and the use of evaluation lead to improvement in an area (both content and geography)?

18. Frameworks to shape our thinking about the EFP
Evaluation influence
Evaluation capacity building (ECB)
Evaluation field building (EFB)

19. Evaluation capacity building
“Intentional work to constantly co-create and co-sustain an overall process that makes quality evaluation and its uses routine in organizations and systems”
Stockdill, Baizerman, & Compton (2002)

20. What, exactly, is evaluation capacity building (ECB)?
Using an evaluation for
(1) its results
AND
(2) the explicit purpose of building people’s capacity to evaluate again

21. For ECB practitioners, the focus
“. . .[is] on responding to requests for evaluation services while simultaneously considering how today’s work will contribute to sustaining the unit in the longer term”
(Compton, Glover-Kudon, Smith, & Avery, 2002, p. 55)
King & Meyer 2011 AEA Presentation

22. Volkov & King (2007) Checklist for ECB
Purpose- “To provide a set of guidelines for organizational evaluation capacity building (ECB), i.e., for incorporating evaluation routinely into the life of an organization”
Available at

23. Broad categories of ECB content
Category
Overarching Activity
Organizational context
Be aware of the internal and external organizational context, power hierarchies, administrative culture, and decision-making processes
ECB structures
Purposefully create structures--mechanisms within the organization--that enable the development of evaluation capacity
Resources
Make evaluation resources available and use them

24. The EFP trainings are ECB
Organizational context- Bringing together people from a variety of settings
ECB structures- Developing capacity within the group to move a broader evaluation agenda forward
Resources- Creating collective power so that they can garner resources for evaluation collectively

25. Adapting ECB structures for EFP
Develop and implement a purposeful long- term ECB plan for the Twin Cities area
Build and reinforce infrastructure to support continued training (MESI)
Foster purposeful socialization into a growing evaluation network
Build and expand a network of EFP alumni (peer learning structure)

26. Frameworks to shape our thinking about the EFP
Evaluation influence
Evaluation capacity building (ECB)
Evaluation field building (EFB)

27. The concept of evaluation field building (EFB)
Three phases of EFB in development evaluation in South Asia (Hay, 2010)
Creating a centralized planning system (served planners)
Improving evaluation quality, pushed by northern-based organizations, e.g., funding agencies (served donors)
Increasing use (to serve government professionals, donors, and their constituents)

28. EFB applied to an urban center
Program evaluation is alive and well in the Twin Cities, MN
Many forms of evaluation practice
Evaluation training opportunities (local, national)

29. EF [P +B]
The Evaluation Fellows Program is a coordinated effort to build the field of evaluation in Minnesota
Increasing evaluation capacity in people working in local agencies and organizations
“Increasing use (to serve government professionals, donors, and their constituents)”

30. EFP’s field building intent
Attending to overlapping roles
Attending to the content “sweet spot”

31. Thoughts on future developments
Where do we go from here?

32. Questions remaining
Can EFP-style training actually lead to increased influence/evaluation capacity/a more developed field in a geographic area?
What would be the outcomes and benefits of building geographic vs. individual/ organizational capacity?

33. Next steps
Get the EFP re-funded- Why are funders reluctant to fund this work?
Continue to build the informal network of EFP alumni
Conduct research on the challenges/ benefits of geographic ECB
King & Meyer 2011 AEA Presentation