1. A Framework for Evaluating Knowledge Transfer and Exchange (KTE) in Public Health
Paper Presented at the Annual Conference of the American Evaluation Association
Minneapolis, Minnesota
October 25, 2012
Marla Steinberg PhD

2. Purpose Use Usefulness Reach
A simple framework for monitoring and evaluating KTE in public health interventions
Reach
Usefulness
Use
KTE is increasingly a requirement or focus for public health interventions
I want to share a simple framework for M&E

3. Outline The complex ecologies of KTE Frameworks for evaluating KTE
Three examples of using the framework
It is many different things
Takes place within complex ecologies

4. Complexity Hundreds of Terms
Encompasses a number of distinct but inter-related processes
Transfer or disseminate the learnings from a project
Scale-up or adoption of a program
Using evidence to inform program planning/design
Knowledge users and knowledge producers working together (translational research)
The science of implementation and dissemination

5. Many Distinct Processes
Knowledge Producers & Users
Knowledge Develop-ment
Knowledge Transfer
Knowledge Translation
Knowledge Utilization
Knowledge Diffusion
Knowledge Dissemin-ation
Knowledge Brokering
Knowledge Synthesis
Lots of terms, many different meanings.
Science of Implementation & Dissemination

6. Want help navigating the complexity?
Issue 124 of New Directions in Evaluation (2009)
examined the same program from different KTE lenses or foci
Illustrates how the KTE evaluation would examine different things depending on the focus of the KTE work

7. Timing of the evaluation in project lifecycle

8. Conceptualizing Use Carol Weiss: Vedung: (2004): Conceptual Conceptual
Instrumental
Symbolic
Vedung: (2004):
Conceptual
Instrumental
Process
Tactical
Ritual
Legitimization

9. Timing since dissemination

10. Gaining the attention of the intended audience or users

11. Complex pathways to use
Damschroder, L.J., Aron, D.S., Keith, R.E., Kirsh, S.R., Alexander, J.A., & Lowery, J.D. (2009). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation Science. Implementation Science, 4:50.
Product is necessary but not sufficient!

12. Multiple Knowledge Products

13. The Solution: Frameworks for Evaluating KTE

14. Simplified Framework Use Usefulness Reach
Can be tailored to the particular evaluand:
Stage of development
KTE processes and products
Unique manifestations of use

15. Example 1: Web-based KT Platform

16. Year 1 – Needs Assessment
Evaluation Plan
Year 1 – Needs Assessment
Year 2 – Build Platform
Year 3 – Promote Use
Reach
Usefulness
Use
EQ1: Did we engage a broad section of intended users?
EQ2: Do user’s like what we have built?
EQ3: What factors facilitate use?
EQ4: How effective are our promotional strategies?
EQ5: How are systems changing?
Event Logs
User Testing
Web Analytics
Case Studies
Pre and Post Tests
Relationship Tracking

17. Example 2: Coalitions Linking Action and Science for Prevention
Funded by the Canadian Partnership Against Cancer
Organizations coming together to develop and deliver chronic disease prevention interventions
KTE was a primary component of the program
Disseminating or scaling-up programs

18. Reach of KTE Work Monitoring
Participation of target audience in the production of the materials
Number disseminated (hard copies)
Web analytics

19. Use: KTE Survey for Target Audience
Conceptual:
Influence Thinking
Instrumental:
Shared resource
Discussed it
Used for practice changes
Program changes
Change resource allocation
Integrated into curriculum
Policy changes

20. Use Questions Awareness of Knowledge Product
Are you aware of
Yes
No (if not aware will be directed out of survey and provided info on how they can obtain information if survey is administered online)
How did you become aware of ______
Have you read or reviewed …….?
No, please explain why you have not reviewed it
Transmission of Knowledge Product
Have you shared the resource or passed it on to others? No
Have you discussed the resource or shared information about it with others?

21. Use Questions Changed my awareness about the issue
Yes No
Not Yet Decided//Not Certain
Changed my awareness about the issue
b) Changed my thinking or attitude about the issue
c) Increased my knowledge
d) Increased my skills
e) Provided me with opportunity to further a professional relationship
f) Provided me with an opportunity to develop a new professional relationship
g) Other, please explain:

22. Use Questions:

23. Example #3: KTE Organization
National Collaborating Centre for the Determinants of Health
Mandate:
To support the synthesis, uptake and use of knowledge on health equity and the social determinants of health

24. Usefulness
Reach
Use

25. Data Sources Reach: Usefulness: Use: Web analytics
Monitoring Form for hard copy dissemination
Usefulness:
Audience survey
Focus groups
Use:

26. Marla Steinberg marlasteinberg@telus.net What do you think?
Will this work for you?
I am happy to share the tools with you
Feel free to use/adapt these tools and let me know how they worked for you or how you changed them!
Marla Steinberg