0. Anita M. Baker, Ed.D. Evaluation Services
Understanding and Utilizing Evaluative Thinking An Approach for Evaluators, Funders and Third-Sector Professionals
Anita M. Baker, Ed.D. Evaluation Services
Bruner Foundation
Rochester, New York

1. Working Definition of Program Evaluation
The practice of evaluation involves thoughtful, systematic collection and analysis of information about the activities, characteristics, and outcomes of programs, for use by specific to reduce uncertainties, improve effectiveness, and make decisions.
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

2. Working Definition of Program Evaluation
The practice of evaluation involves thoughtful, systematic collection and analysis of information
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

3. Working Definition of Program Evaluation
The practice of evaluation involves thoughtful, systematic collection and analysis of information
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

4. Working Definition of Program Evaluation
The practice of evaluation involves thoughtful, systematic collection and analysis of information about the activities, characteristics, and outcomes of programs,
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

5. Working Definition of Program Evaluation
The practice of evaluation involves thoughtful, systematic collection and analysis of information about the activities, characteristics, and outcomes of programs, for use by specific people,
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

6. Working Definition of Program Evaluation
The practice of evaluation involves thoughtful, systematic collection and analysis of information about the activities, characteristics, and outcomes of programs, for use by specific people, to reduce uncertainties, improve effectiveness, and make decisions.
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

7. Evaluation Capacity The ability to do evaluation.
Involves proficiency in:
asking questions
determining data required to answer questions
collecting data using appropriate strategies
analyzing collected data and summarizing findings
using and sharing findings
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

8. Evaluative Thinking
Evaluative Thinking is a type of reflective practice that uses the five key evaluation skills in areas other than programs or initiatives.
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

9. What Are Key Components of Evaluative Thinking?
1. Asking questions of substance
2. Determining data needed to address questions
3. Gathering appropriate data in systematic ways
4. Analyzing data and sharing results
5. Developing strategies to act on findings
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

10. What Are Key Components of Evaluative Thinking?
1. Asking questions of substance
2. Determining data needed to address questions
3. Gathering appropriate data in systematic ways
4. Analyzing data and sharing results
5. Developing strategies to act on findings
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

11. What Are Key Components of Evaluative Thinking?
1. Asking questions of substance
2. Determining data needed to address questions
3. Gathering appropriate data in systematic ways
4. Analyzing data and sharing results
5. Developing strategies to act on findings
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

12. What Are Key Components of Evaluative Thinking?
1. Asking questions of substance
2. Determining data needed to address questions
3. Gathering appropriate data in systematic ways
4. Analyzing data and sharing results
5. Developing strategies to act on findings
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

13. What Are Key Components of Evaluative Thinking?
1. Asking questions of substance
2. Determining data needed to address questions
3. Gathering appropriate data in systematic ways
4. Analyzing data and sharing results
5. Developing strategies to act on findings
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

14. What Are Key Components of Evaluative Thinking?
1. Asking questions of substance
2. Determining data needed to address questions
3. Gathering appropriate data in systematic ways
4. Analyzing data and sharing results
5. Developing strategies to act on findings
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

15. Evaluative Capacity
The combination of evaluation skills and evaluative thinking.
Evaluative Capacity requires a commitment to doing and using evaluation in programs, strategies, and initiatives as well as a commitment to using those same skills in other aspects of organization work.
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

16. Key Tenet of Bruner Foundation Investments
Evaluation capacity used well, supports programs, strategies, and initiatives that in turn lead organizations to better deliver on their missions and better meet the needs of those they serve. The Bruner Foundation has invested in Evaluation Capacity Building (ECB) for more than 20 years.
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

17. Organizations that Regularly use Evaluative Thinking Will . . .
Think carefully about developing and assessing programs/other actions.
Incorporate analytical findings into planning.
Involve significant others in planning.
Develop written, logical plans.
Follow plans.
Have strategies in place to modify plans
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

18. Organizations that Regularly use Evaluative Thinking Will Also . . .
Regularly conduct evaluations.
Involve multiple stakeholders in developing evaluation designs, collecting and analyzing data.
Share results of evaluations with multiple stakeholders.
Use results of program evaluation to drive continuous improvement of programs and to modify policies and procedures.
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

19. Lastly, Organizations that Regularly Use Evaluative Thinking Will . . .
Insure that there are key staff with evaluation expertise, whose jobs or components of their jobs are dedicated to evaluation.
Hire evaluation consultants when needed.
Provide or obtain training in evaluation for program staff
current
well-delivered
provided for enough staff members to insure that evaluation use is a standard practice
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

20. Using the Bruner Foundation Evaluative Thinking Assessment Tool
Created in 2005 to assess the extent to which evaluative thinking is present in various organizational capacity areas.
Grew out of the Bruner Foundation’s Evaluative Thinking in Organizations Study (ETHOS).
ETHOS Study
Automated in 2007 and updated in after additional study.
Evaluative Thinking Assessment
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

21. What Organizational Capacity Areas Does the Bruner Foundation Evaluative Thinking Tool Address?
Mission
Strategic Planning
Governance
Finance
Leadership
Fund Development
Evaluation
Client Relationships
Program Development
Communication & Marketing
Technology Acquisition & Training
Staff Development
Human Resources
Alliances/Collaborations
Business Development
Created in 2005 by ETHOS partners
Addresses multiple organizational capacity areas
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

22. The Bruner Foundation Evaluative Thinking Assessment Tool
ORGANIZATION MISSION
Assessment
Priority a.
The mission statement is specific enough to provide a basis for developing goals and objectives 1  b.
The mission is reviewed and revised on a scheduled basis (e.g. annually) with input from key stakeholders as appropriate 0  c.
The organization regularly assesses compatibility between programs and mission 1 d.
The organization acts on the findings of compatibility assessments (in other words, if a program is not compatible with the mission, it is changed or discontinued)
Comments:
#DIV/0!
Please proceed to the next Worksheet
MARCIE
Within my organization/within other NFP organizations
Where is it happening/where should it happen and it isn’t/where might it happen?
Can we identify responsibility centers/people (board expects, but do they evaluate selves?)
The “don’t know” factor – is it a by-product of communication issues? NHS found disparity among managers as to what they were and weren’t aware of! – alters perception
Evaluation beyond performance – using it for budgeting/planning
How do we get more staff involved (capacity-building)
Leadership succession: building capacity positively impacts – more staff/faculty responsible for thinking evaluatively. Can head off issue of what happens when leadership changes.
1. This is a high priority
2. This is a lower priority
3. This is not a priority at all
Agree, Enter 1
Disagree, Enter 0
Don’t know, leave it blank
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

23. Organization Leaders Who Use Evaluative Thinking . . .
Support program evaluation and evaluative thinking.
Educate staff about the value of evaluation and how to participate effectively in evaluation efforts.
Use evaluation findings in decision-making.
Include attention to evaluation as an important part of a succession plan. (New leaders will be expected to value and be knowledgeable about evaluation.)
Foster use of technology to support evaluation and evaluative thinking.
Use data to set staff goals and evaluate staff performance.
Bruner Foundation
Rochester, New York

24. Program Development Organizations That Use Evaluative Thinking . . .
Identify gaps in community services before planning new programs.
Incorporate findings from program evaluation into the program planning process.
Involve multiple stakeholders in developing and revising program plans.
Develop written program plans including a logical formulation of each program.
Follow program plans where possible; insure that there are strategies in place to modify program plans if needed.
Bruner Foundation
Rochester, New York

25. Technology Acquisition and Training In Organizations That Use Evaluative Thinking . . .
Technology systems are able to provide data to evaluate client outcomes.
Technology systems are able to provide data as necessary (e.g., financial information, time tracking) to evaluate organizational outcomes.
Technology systems are regularly assessed to see if they support evaluation.
Staff technology needs are regularly assessed.
Technology systems include software that can be used to manage an analyze evaluation data.
Bruner Foundation
Rochester, New York

26. Staff Development Organizations That Use Evaluative Thinking . . .
Conduct formal staff development needs assessment annually (based on staff and organizational needs).
Develop a comprehensive plan for staff development based on needs assessment data.
Provide opportunities for staff to assess staff development training.
Provide opportunities for organization to assess staff development plan.
Use results of assessments to influence future staff development.
Bruner Foundation
Rochester, New York

27. Human Resources Organizations That Use Evaluative Thinking . . .
Have an established personnel performance review process.
Use performance reviews to provide feedback relative to performance expectations.
Collect and update information on credentials, training and cultural competencies of staff; use the information to recruit, hire and train culturally competent staff.
Conduct regular (e.g., annual or biannual) staff satisfaction surveys, and use the results to inform modification of policies and procedures.
Bruner Foundation
Rochester, New York

28. Evaluation Organizations That Use Evaluative Thinking . . .
Share results of program evaluations with leaders, staff, clients, board members and funders as appropriate.
Use results of program evaluation to drive continuous improvement of programs.
Insure that there are key staff with evaluation expertise and that there are staff members whose jobs or components of whose jobs are dedicated to evaluation.
Provide or obtain current, well-delivered training in evaluation for program staff members.
Provide training for enough staff members to insure that evaluation use is a standard practice.
Bruner Foundation
Rochester, New York

29. Evaluative Thinking Assessment Tool Summary Report, by Capacity Area
Organizational Capacity Area
Capacity Score*
Action Planning** 1
Mission
100
No action required in this area 2
Strategic Planning 40
Action required see priorities 3
Governance 50
Action suggested see priorities 4
Finance 91 5
Leadership 6
Fund Development/Fund Raising 25 7
Evaluation 30 8
Program Development 9
Client Relationships 67 10
Communication and Marketing 11
Technology Acquisition and Planning 12
Staff Development 13
Human Resources 57 14
Business Venture Development 15
Alliances and Collaboration
MARCIE
Within my organization/within other NFP organizations
Where is it happening/where should it happen and it isn’t/where might it happen?
Can we identify responsibility centers/people (board expects, but do they evaluate selves?)
The “don’t know” factor – is it a by-product of communication issues? NHS found disparity among managers as to what they were and weren’t aware of! – alters perception
Evaluation beyond performance – using it for budgeting/planning
How do we get more staff involved (capacity-building)
Leadership succession: building capacity positively impacts – more staff/faculty responsible for thinking evaluatively. Can head off issue of what happens when leadership changes.
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

30. Evaluative Thinking Assessment Tool Summary Report Score Chart
MARCIE
Within my organization/within other NFP organizations
Where is it happening/where should it happen and it isn’t/where might it happen?
Can we identify responsibility centers/people (board expects, but do they evaluate selves?)
The “don’t know” factor – is it a by-product of communication issues? NHS found disparity among managers as to what they were and weren’t aware of! – alters perception
Evaluation beyond performance – using it for budgeting/planning
How do we get more staff involved (capacity-building)
Leadership succession: building capacity positively impacts – more staff/faculty responsible for thinking evaluatively. Can head off issue of what happens when leadership changes.
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

31. Bruner Foundation ECB Clearinghouse
Evaluation Capacity Building Strategies
Bruner Foundation ECB Clearinghouse
Basic evaluation training
Advanced evaluation training
Applied learning
Coaching and T.A.
Networking – peer learning
Alumni study groups
Anchoring
Funding collaboratives/shared governance
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

32. ECB Trainees Use What They Learn
Percent Who Think Participating in ECB Helped Their Organizations . . .
Some
A lot
TOTAL
N=97
Do better evaluations of programs
42%
57%
99%
Use evaluative thinking skills in multiple aspects of work
43%
52%
95%
Commission better evaluation
49%
92%
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

33. Clarification from the Field
I think we are using an evaluative mind to determine direction of the Agency. We continuously ask questions now: What is the information we need? How do we get the information? What is the best method to use? What is the time frame to implement? When can data be available?. . . As Executive Director, I now have a team of "evaluative minds" to strategically look at areas to improve, develop, and prepare for organizational or funding opportunities.
Bruner Foundation
Rochester, New York Anita M. Baker, Evaluation Services

34. Evaluative
Thinking
Bruner Foundation
Rochester, New York