123 E. 8th Street Claremont, CA Valuing Knowledge Flow in the Evaluation Process Evaluation 2011: Values and Valuing Multisession 329 Karen Widmer, M.S. Claremont Graduate University November 3, 2011

Widmer’s Ode to Knowledge  The flow of knowledge is the chief necessary condition for evaluation capacity/building  It mediates or moderates all evaluative activities and learning  It determines how well an organization can self- assess and how well it can use what it finds when it looks.

LEARNING ORGANIZATIONS Valuing Knowledge Flow in the Evaluation Process Evaluative capacity Organizational performance Knowledge platform Moderates Mediates My model suggests: By building the platform on which it stands, evaluation capacity grows and delivers a return on performance. A learning organization does this well.

Follow the flow = better evaluation Polanyi (1966) “Tacit and explicit knowledge” Argyris and Shon ( 1978) “Single- and double-loop learning” Nonaka (1994) “Knowledge creation” Becerra-Fernandez (2010) “Knowledge roles” Davenport & Prusek (1988) “Data Information Knowledge” Knowledge Basics EvaluationPerformance Knowledge

Tacit and Explicit Knowledge 1. Tacit: “Indwells” - understanding that resides in the mind Did observed outcomes match the evaluand’s program theory? Technical know-how Cognitions ~ Relationships and trust Polanyi Apply Ways to elicit tacit knowledge on-the-job Collaborative problem-solving, exit interview, focus group, network, job rotation, facilitate, mentor, apprentice, rounds, coach, cross-train, “master”, lessons learned, case-based reasoning 2. Explicit: “articulated” – written or recorded content

Single- and double-loop learning Underlying assumptions What we do Results Single-Loop Learning – low-level evaluation Double-Loop Learning - continuously create new knowledge by reconstructing the framework Argyris & Shon Apply Do you structure reframing into your evaluations?

Tacit knowledge Explicit knowledge Tacit knowledge SocializationExternalization Explicit knowledge InternalizationCombination Are participants in synch and ready to action-ize the newly internalized? Converting the forms of knowledge Nonaka Apply

Awareness of the purposes that are served by knowledge aids evaluation design and choice of measures Discover o Socialization a o Combination b Capture o Internalization a o Externalization b Share o Socialization a o Exchange b Apply o Direction a, b o Routine a Becerra- Fernandez Apply a =tacit knowledge; b = explicit knowledge Four ways to use tacit explicit flow!

DataInformationKnowledge Discrete objective facts Data with a message  Categorized  Condensed  Corrected  Calculated  Contextualized Actionable insights  Experience  Ground truths  Complexity rules  Involves judgment  Values and beliefs * Davenport & Prusek Distinguishing Data Information Knowledge What do you need to measure (data, information, or knowledge)? And to what end? Apply

More recent streams of knowledge flow Oliver (2009) “Knowledge transfer” Ashley (2009) “Innovation diffusion” Davison (2009) “Knowledge translation” Hawe (2009) “The complexity lens” 2 nd Generation Concepts What other disciplines evaluate by following the knowledge flow? Apply Hint Information systems, business management, instructional design, public policy, healthcare…other?

Widmer’s Ode to Knowledge  The flow of knowledge is the chief necessary condition for evaluation capacity/building  It mediates or moderates all evaluative activities and learning  It determines how well an organization can self- assess and how well it can use what it finds when it looks.

(11:40 AM, today) Session 329: The Value of Knowledge Management in Evaluation: A Research Perspective Knowledge Translation Knowledge Management/Knowledge Exchange Multi-paper Session 329; Laguna A Thursday, Nov 3; 11:40 AM to 12:25 PM Sponsored by the Research on Evaluation TIG For more on knowledge and evaluation? Join Us!

Ashley, S. R. (2009). Innovation diffusion: Implications for evaluation. In J. Ottoson & P. Hawe (Eds.), New Directions for Evaluation, 124, DOI: /ev.310 Argyris, C., & Schön, D. (1978) Organizational learning: A theory of action perspective. Reading, MA: Addison Wesley. Becerra-Fernandez, I., & Sabherwal, R. (2010). Knowledge management: Systems and processes. Armonk, N.Y.: M. E. Sharpe, Inc. Davenport, T. H. & Prusak, L. (1998). Working knowledge. How organizations manage what they know. Boston, MA: Harvard Business School Press. Davison, C. M. (2009). Knowledge translation: Implications for evaluation. In J. Ottoson & P. Hawe (Eds.), New Directions in Evaluation, 124, DOI: /ev.310 DeGroff, A. & Cargo, M. (2009). Policy implementation: Implications for evaluation. In J. Ottoson & P. Hawe (Eds.), New Directions in Evaluation, 124, DOI: /ev.310 References

Hawe, P., Bond, L., Butler, H. (2009). Knowledge theories can inform evaluation practice: What can a complexity lens add? In J. Ottoson & P. Hawe (Eds.), New Directions in Evaluation, 124, DOI: /ev.310 Nonaka, I. (1994). A dynamic theory of organizational knowledge creation. Organization Science, 5 (1), Oliver, M. L. (2009). The transfer process: Implications for evaluation. In J. Ottoson & P. Hawe (Eds.), New Directions in Evaluation, 124, DOI: /ev.310 Polanyi, M. (1966). The tacit dimension. London, England: Routledge and Kegan Paul. Preskill, H. & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29 (4), Online. Retrieved September 11, 2008, from doi: / References

123 E. 8th Street Claremont, CA (707) Contact Information Karen Widmer, M.S. Claremont Graduate University Evaluation 2011: Values and Valuing November 3, 2011 Multisession 329 Valuing Knowledge Flow in the Evaluation Process