1. Tools for Creating a Culture of Assessment The CIPP Model and Utilization-Focused Evaluation
Yvonne Belanger, Duke University
Library Assessment Conference
September 25-27, 2006
Charlottesville, VA

2. Key Questions for Libraries
How can we build a culture of evaluation, so that many people contribute to evaluation?
How can we provide a context for evaluation strategies and results?
How can we conduct evaluation that helps with decision making?
Overview
Culture, Context, and Conducting evaluation
Varying evaluation resources available at different institutions
Drive toward decision-making is acceptable
Today’s presentation
A framework and key issues to consider in evaluation planning
Specific tools, templates and strategy – focus on the CIPP model and utilization-focused evaluation

3. Culture of Assessment
“…organizational environment in which decisions are based on facts, research, and analysis, and where services are planned and delivered in ways that maximize positive outcomes and impacts for customers and stakeholders.”
Lakos et al,

4. Barriers to a Culture of Assessment
Lack of evaluative thinking (at all levels)
Lack of engagement in evaluation
Pseudoevaluations (Stufflebeam, 1999)
Promote a positive or negative view of a program, irrespective of its actual merit and worth
Lack of evaluative thinking – includes intuition-based rather than data driven decision making, lack of systems thinking – how does what I am actually doing connect back to my goals and forward to my intended outcomes
PR Inspired studies (withholding all negatives)
Politically controlled studies
Other factors discussed by Jim Self and Steve Hiller and others at this conference – need for clear leadership, specific person or group tasked with assessment

5. Building evaluative thinking: CIPP Model
Stufflebeam’s CIPP Model - Context, Input, Process and Product evaluation
Focus: decision-making
Purpose: facilitate rational and continuing decision-making, particularly for programs and services with long-term goals
A comprehensive framework for guiding formative and summative evaluations
Based on a presumption that evaluation’s most important purpose is not to prove but to improve programs

6. Details of the CIPP Model
Context: Environment & Needs
Input: Strategies & Resources
Process: Monitoring implementation
Product: Outcomes - both quality and significance
More information at
The CIPP was developed by D. Stufflebeam (see annotated bibliography for references)
A comprehensive framework for guiding formative and summative evaluations
Based on a presumption that evaluation’s most important purpose is not to prove but to improve programs
Has evolved over 30 years but remained up to date with new ideas from evolving approaches – e.g. Patton’s Utilization-focused evaluation, Guba & Lincoln’s Stakeholder focused evaluation
CIPP adapts well to carrying out evaluations on any scale (projects, programs, organizations)
An organizing framework, not a lockstep linear process
Sensitive to needs of decision makers (more detail on that ahead…)
Systems approach – for that reason, using logic modeling to get a systems view of projects and programs can be a useful first step
Multiple observers and informants
Mining existing information
Multiple procedures for gathering data; cross-check qualitative and quantitative
Independent review by stakeholders and outside groups
Feedback from Stakeholders

7. CIPP approach recognizes…
“All politics are local” – offers a tailored evaluation approach designed to answer locally interesting & useful questions, emphasis is on credibility and usefulness rather than generalizability to other places, times, audiences
Tips taken from Stufflebeam’s recent writings on using the CIPP approach (OPEN, 2003):
Multiple observers and informants
Cross-check often referred to as “triangulating”
Multiple procedures for gathering data
Mining existing information
Independent review
Stakeholder feedback “All politics are local” – a tailored evaluation approach designed to answer locally interesting & useful questions, emphasis is on credibility and usefulness rather than generalizability to other places, times, audiences

8. CIPP View of Institutionalized Evaluation
CIPP provides a systematic way of thinking about how evaluation can contribute to short term and long term organizational planning
CIPP for Decision Makers
C: Define goals and priorities
I: Assess competing proposals in terms of feasibility, alignment with goals
P: Provide context for interpreting outcomes, plan for service improvement
P: Keep organization focused on achieving important outcomes, gauge success of efforts
Connects manager / decision-maker thinking with an evaluation structure that all staff can contribute to and see themselves as a part of
Stufflebeam sees Input as potentially the most neglected type of evaluation (Stufflebeam, OPEN, 2003)
Provides a framework for integrating evaluation as an activity central to achieving broader organizational goals
Illustrates the focus of the model on use of evaluation information to shape goals, plans, and actions
Stufflebeam, OPEN, 2003

9. Advantages of the CIPP Model
Adapts well to carrying out evaluations on any scale (projects, programs, organizations)
An organizing framework, not a lockstep linear process
Sensitive to needs of decision makers
Systems approach – encourages a systems view of projects and programs

10. Taking a utilization-focused approach means asking…
Building evaluative thinking and engagement: Utilization-focused evaluation approach
Taking a utilization-focused approach means asking…
Why is this evaluation being undertaken?
What decisions need to be made with the results?
Who will be most affected by those decisions?
How can we engage those people in the entire evaluation process?
All participants in an evaluation should be clear as to why the evaluation is being conducted, whether or not the results (or all of the results) will be shared publicly, internally only, only with key decision-makers, etc. and how the outcomes of the evaluation might affect them. Failing to follow these procedures will jepoardize your efforts to build a culture of assessment by destroying good will for assessment efforts, contribute to negative view of assessment and increase the paranoia of any staff who already feel threatened by these efforts.

11. Utilization-focused evaluation
Premise – by engaging stakeholders in the entire evaluation process from design to implementation of recommendations
Evaluation addresses questions of greatest importance to those in a position to directly make use of its findings
Reduces the cultural barriers that can inhibit use of results by increasing transparency, empowering stakeholders

12. Another advantage of the Utilization-focused approach
“Process Use” benefits
First described by Patton - ‘ways in which being engaged in the processes of evaluation can be useful quite apart from the findings that may emerge from these processes’
Four types of Process Use
1. Enhancing shared understandings, especially about results;
2. Supporting and reinforcing the object of the evaluation through intervention-oriented evaluation;
3. Increasing participants’ engagement, sense of ownership
4. Organizational development
Patton (1997) Utilization-focused evaluation – New Century Text (3rd ed) pp )
Patton 1997, pp

13. Process Use & Culture of Assessment
Increased capacity to make use of evaluation findings
Know how to use evaluation information – producing better evaluation users in the organization who can effectively “weigh evidence, consider contradictions and inconsistencies, articulate values, and examine assumptions”
through their experiences interpret evidence, draw conclusions, and make judgments
Patton, 2004, “On Evaluation Use: Evaluative Thinking and Process Use”

14. Example
Evaluation of the Duke iPod experiment & Duke Digital Initiative…

15. Summary Foster a culture of assessment by:
Adopting frameworks that support decision-making
Engaging staff as stakeholders in the entire process of evaluation from design to implementation of recommendations
Leverage the opportunity of Process Use to develop staff and make them more saavy evaluation consumers

16. Final Thoughts…
“…evaluation's most important purpose is not to prove, but to improve.”
Daniel Stufflebeam
(CIPP Model)
“Research is aimed at truth. Evaluation is aimed at action.”
Michael Quinn Patton
(Utilization-focused Evaluation)
Michael Patton, Former President of AEA, leader in evaluation
"Research is aimed at truth. Evaluation is aimed at action.”
Research efforts often focus on a particular variable, and is often narrowly focused to answer only question.

17. Thank You! Yvonne Belanger Head, Program Evaluation
Academic Technology & Instructional Services
Perkins Library
Duke University

18. References
Stufflebeam, D. (1999). Foundational models for 21st century program evaluation.
Stufflebeam, D. (2003). The CIPP Model for Evaluation: An update, a review of the model’s development, a checklist to guide implementation. Paper read at Oregon Program Evaluators Network Conference, at Portland, OR.
Patton, M. Q. (2004). "On evaluation use: Evaluative thinking and process use." The Evaluation Exchange IX(4).
Patton, M. Q Utilization-focused evaluation: The new century text (3rd ed.). Thousand Oaks, CA: Sage.