1. KU07 Conference Stockholm. Sweden August 16 th, 2007 The dependent variable in knowledge translation research: What should we measure? How should we measure it? Carole Estabrooks PhD, RN Janet Squires PhD Student, RN Lars Wallin PhD, RN Petter Gustavsson PhD

2. Outline 1.The need for a measure 2.Measures to date 3.The Canada/Sweden measurement study 4.How should we go about measuring RU? 5.Introduction to group discussions What does the theory tell us to measure?

3. The need for a measure CE

4. Why measure research use instead of patient outcomes when assessing KT interventions? Not instead – in addition However, if we do not measure research use, it is difficult to claim that the intervention tested to increase research use was effective And, if we only measure patient outcomes….it is difficult to determine if an improvement in such outcomes is attributable to the intervention to increase research use Finally, working in organizations, it is we argue useful to develop and use measures of KT that reflect use at unit (possibly team) and facility levels, in addition to individual levels

5. What is Research Utilization ? Types of Research Utilization  Instrumental  Conceptual  Symbolic/Persuasive

6. Nurses & Allied Health Professionals Measures to date JS

7. Nurses and Allied Health Professionals Review of nursing and allied literature Estabrooks, Wallin, & Milner. (2003). ). Measuring knowledge utilization in health care. International Journal of Policy Evaluation & Management, 1(1), 3- 36. Purpose: To assess the state of measurement science specific to the concept of research utilization

8. In that review the group assumed some requirements of robust measurement instruments Substantive (content) theory Measurement theory Minimal self-report & social desirability effects Unequivocal scale Acceptability within the research field

9. The review Search Strategy  Published reports of research ‘use’  CINAHL and Medline electronic databases Search Results  42 publications  3 commonly used multi-item measures  Multiple single item measures JS

10. Findings Existing Measures Common Multi-item Measures: Nursing Practice Questionnaire (N=3)  Nursing Practice Questionnaire (Modified) (N=6) Research Utilization Questionnaire (N=7) Edmonton Research Orientation Survey (N=3) Other multi-item measures (N=3) Single item measures (N=20)

11. Multi-Item measures NPQ (Brett 1987)  Based of Rogers innovation-decision process theory, reliability limited to chronbach alpha, content validity assumed RUQ (Champion & Leach 1989)  reliability limited to chronbach alpha, limited validity assessments, operational defn of use unclear EROS (Pain, Hagler, & Warren 1996)  reliability limited to chronbach alpha, construct validity assessment, operational defn of use unclear, recall bias

12. Single Item Measures 20 studies used single-item measures Similar limitations to multi-item but also:  Different terminology (makes it difficult to compare studies)  Array of different scoring methods and scaling assumptions  Recall bias

13. Medicine Measures to date LW

14. Guideline literature Based on Grimshaw et al (2004). Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technology Assessment, 8, 1-72. 235 studies were reviewed Data extracted on two main measures  Process of care results  Outcome of care results Not described how data was collected, exemplified with charts and self-reports 225/235 studies used process of care measures 50/235 studies used outcome of care measures

15. Guideline literature Examples of measures Process of care  % of patients blood pressure measured (hypertension screen)  % of patients smoking status recorded  % of patients documented nutritionist consult  Rating of time spent counseling (physician questionnaire) Outcome of care  Systolic blood pressure  % of patients stopped smoking at 12 month  % of patients satisfied with care  Sickness Impact Scores at 6 month  % of patients surviving at 1-year

16. General Problems Across Measures Measuring the construct of ‘research use’  Poor construct clarity  Lack of theoretical framing  Presumption of linearity  Common assumptions held about research use Methodological  Lack of measurement theory underpinning  Lack of psychometric assessment  Unit of analysis - ecological fallacies  Retrospective survey design dependent on self- report and recall  Operationalization of ‘use’ unclear  Absence of longitudinal work  Scaling Regardless of the measure…. JS

17. Update of the 2003 measurement review – repairing prior sins Expanded Search  Databases: HAPI, CINAHL, MEDLINE, EMBASE, SCOPUS, OCLC Papers First, OCLC WorldCat, ABI Inform, Web of Science, Sociological Abstracts  Hits: 23,065  Abstracts Screened: 11,810 Articles pulled: n=105 Need more information: n=497  Hand search  Cross references

18. Some notes on Experiences with a single item measure CE

19. Research Utilization Studies (1999-2003) Mean (SD) Unit 1Unit 2Unit 3Unit 4Unit 5Unit 6Unit 7 Overall RU#1 3.94 (1.78) 5.43 (1.50) 4.47 (2.00) 3.59 (1.54) 4.43 (2.00) 5.18 (1.61) 5.16 (1.41) 4.80 (1.75) RU#2 4.67 (1.85) 5.51 (1.61) 5.21 (1.89) 4.12 (1.87) 5.24 (1.81) 5.69 (1.39) 5.59 (1.60) 5.30 (1.68) RU#3 4.83 (1.91) 5.83 (1.25) 5.06 (1.82) 5.19 (1.72) 5.56 (1.78) 5.93 (1.30) 5.59 (1.42) 5.56 (1.57) IRU 4.91 (1.85) 5.60 (1.75) 5.65 (1.50) 5.06 (1.98) 5.6 (1.83) 5.71 (1.44) 5.89 (1.22) 5.51 (1.65) CRU 5.33 (1.75) 5.80 (1.53) 5.59 (1.50) 4.71 (1.83) 6.04 (1.43) 6.03 (1.19) 5.92 (1.38) 5.74 (1.50) PRU 4.00 (1.96) 5.12 (1.63) 5.31 (1.62) 5.13 (1.82) 4.86 (1.91) 4.81 (1.87) 5.00 (1.73) 4.79 (1.84) Range: 1-7

20. Alberta Nurse Survey (2002) Mean (SD) Staff NurseEducatorManager Overall RU #1* Range: 0-5 3.54 (.91) 4.36 (.74) 3.76 (.94) Overall RU #2* Range: 0-5 3.64 (.90) 4.47 (.63) 3.80 (.83) Instrumental RU* Range: 0-5 3.49 (.92) 4.00 (1.01) 3.48 (1.14) Conceptual RU* Range: 0-5 3.59 (.89) 4.14 (.82) 3.78 (.92) Persuasive RU* Range: 0-5 2.60 (1.17) 3.54 (1.10) 3.22 (1.13) * = One-Way ANOVA P-value <0.001

21. CIVMIL Study (1999, 2005) Mean (SD) US Army Setting Canadian Civilian Setting Overall RU 4.53 (1.76) 4.68 (1.72) IRU*4.18 (1.93) 4.47 (1.92) * = Gamma (ordinal test) statistic P-value <0.05

22. AKUTE Study NursesAlliedMDsManagerEducator Overall RU Range: 1-5 2.98 (1.40) 3.11 (1.32) 3.34 (1.33) 2.72 (1.23) 3.26 (1.24) IRU* Range: 1-5 3.20 (1.43) 2.91 (1.36) 2.37 (1.24) 2.68 (1.27) 3.44 (1.28) CRU Range: 1-5 2.67 (1.39) 2.78 (1.31) 2.77 (1.14) 2.84 (1.14) 2.83 (1.39) PRU* Range: 1-5 1.62 (1.09) 1.65 (1.06) 1.58 (0.97) 1.76 (1.00) 2.21 (1.24) * = One-Way ANOVA P-value <0.05

23. Research use in nursing practice in relation to educational, individual and work contextual factors (Sweden) Data collected within the LANE project (Longitudinal analysis of Nursing Education) Nationwide survey, prospective longitudinal design, 2002-2008 Two cohorts of nursing students Instrumental, conceptual and symbolic research use at 1, 2, 3 and 5 years after graduation Aims: To identify predictors and to study the variability over time of nurses’ research use LW

25. The current measurement study Canada/Sweden CE

26. Developing a Valid and Reliable Measure of Research Utilization (Ref: Belfast KU04) Purpose : To develop and assess an instrument that measures research utilization among health professionals The Canada ARM: CIHR (2005-2007) Investigators: Estabrooks, Wallin, Scott-Findlay, Profetto-McGrath, Harley The Swedish ARM: Karolinska Institute supported Investigators: Wallin, Strandberg, Kajermo-Nilsson, Gustavsson

27. Measurement Study Planned outcomes: A set of definitions for research utilization A finalized version of the research utilization concept model previously developed by the team and expert panel A preliminary set of indicators of research utilization from which to develop items for the instrument An instrument

28. The Approach Construct clarity  Series of focus groups with three sets of experts  Local (research team) expert panel  International panel  Managers and educators  Providers (RNs/RPNs, LPNs/aides) Indicator and item development  Series of focus groups with two sets of experts  Managers and educators  Providers of nursing care (RNs/RPNs, LPNs/Aides) Develop and pilot test the draft instrument(s)

29. Measurement Study International Panel: Jo Rycroft-Malone (UK) Carl Thompson (UK) Marita Titler (US) Anne Sales (US) Judith Ritchie (Canada) Jo Logan (Canada) Nancy Edwards (Canada) Donna Ciliska, (Canada)

30. Instrumental Use “Action” Non-use A. Informed (non-defensible) B. Informed (defensible) Non-use Task completion Relational Assessment Screening Research Use Research Non-Use “Thinking” (i.e., Clinical Decision Making Clinical Reasoning ) Conceptual Use Instrumental Use Task completion Action in Practice Uninformed Non-use Conceptual Use Concept Map (Study Team)

31. HI LO HILO Instrumental Conceptual Research utilization

32. Findings: Canadian ARM JS

33. Focus Groups – Round 1: Themes Identified Research Utilization Terminology Definitions Examples Indicators

34. Terminology Majority related to instrumental research use No delineation between low and high conceptual use “Research Utilization” not commonly used Terms commonly used included:  Evidence-based medicine (practice)  Research-based knowledge  Based on research  Based on clinical evidence

35. Definitions Research Use  Putting something into practice  A personal choice or judgement Conceptual Research Use  Thinking, critical thinking Instrumental Research Use  Action, implementing, observable

36. Examples Examples increased in concreteness from manager/educator to RN/RPN to LPN/aide Most examples r/t instrumental research use Gradient from conceptual to instrumental Examples common to groups:  Talking/discussing  Following policies and procedures/protocols  Specific conditions: oe.g. wound care, Braden scale

37. Indicators Following research-based policies and procedures/protocols Asking questions Peer-to-peer interactions Reading research literature Bringing in research articles Self report Observation Chart documents

38. Focus Groups – Round 2 Validation for IRU schema developed based on round 1 analysis:  Self-report oQuestionnaire oInterview (healthcare provider, manager)  Observation  Chart audit

39. Findings: Swedish ARM LW

40. Terminology Research Utilization not a common term (but might work) More accepted than evidence-based practice - “trendy” concept

41. Definitions Research Use  Basing nursing on scientific findings  Working according to appraised and compiled information  Adopting a critical approach to one’s work Conceptual Research Use  learning through critical reflection/thinking Instrumental Research Use  Active process

42. Examples Most examples r/t instrumental research use Adherence to research-based routines/guidelines Using systematic assessment instruments Quality management work All groups mentioned application of concrete findings r/t procedures and treatments:  Insulin Injection technique  S/C injection of sterile water for labour pain

43. Indicators Felt instrumental and conceptual research use were observable Following research-based routine/guidelines Updating routine/guidelines Observation:  of specific practices, using systematic assessment instruments, a structured nursing ‘working method’ Chart documents Using internet or intranet to seek information Providing rationale for one’s actions Peer-to-peer interactions

44. Similarities between Canadian and Swedish findings Instrumental:  Adherence to research-based policies/protocols  Easier to talk about specific practices  Examples increased in concreteness from manager/educator to RN to LPN/Aide Conceptual:  A progression of conceptual to instrumental  Equated with thinking/critical thinking LW

45. Participants not at all accustomed to discussing research utilization Consequently, focus groups not an optimal method Much confusion about conducting research vs. use of research Can we distinguish conceptual RU from instrumental RU? RU often viewed as a process with similarities to the process of EBP. Is conceptual use ‘necessarily’ a precursor to implementation? What about implementation without understanding ? Learnings & Directions

46. Critical reflection viewed as an important attribute to RU. At the same time participants claimed that much of RU was unreflected. Enhanced awareness of the knowledge base for clinical practice - a strategy for RU? How to measure RU if practitioners are not aware of if they are using research? Adaptation to patients’ perceptions and preferences important and contributes to make the concept of RU vague. Consequences for measuring RU?

47. Next Steps Systematic review (update) of instruments to measure research use (in-progress) Manuscript on Canada/Sweden findings re instrumental use (in-progress) Concept analysis (literature based) on conceptual use (in-progress, Canada) Submit grant applications 2008 1. Develop and pilot instrumental measures 2. Construct clarity re conceptual use (concept analysis using ethnoscience) 3. Eventually development of a measure of CRU… CE

48. Psychometric theory PG

49. Sources

50. Latent, Not manifest Manifest Observed responses

51. To a great extent To a moderate extent To a little extent To no extent

52. i1 i2 i3 i4

53. Strength i1 i2

54. Location

55. an instrument is valid for measuring an attribute if and only if  (1) the attribute exists and  (2) variations in the attribute causally produce variations in the outcomes of the measurement procedure

56. Classical Test Theory: Strength i1 i2 i3 i4

57. Item Response Theory: Location

58. Respondents Items Direction of increasing ”X” Direction of decreasing ”X” Respondents with low ”X” Respondents with midrange ”X” Respondents with high ”X” Item indicating lowest level of ”X” Item indicating lower level of ”X” Item indicating higher level of ”X” Item indicating highest level of ”X”

59. The construct The research-practice gap  Barriers to research use oInability to obtain research findings in the area of interest Understanding, Resistance, Relevance, Agreement with conclusions oTime constraints oCosts oRewards

64. 10% to 60%

65. Unidimensional? Measures of model fit  Absolut fit oSRMR=0.09 (0.07)  Parismony fit oRMSEA=0.08 (0.05)  Comparative fit oCFI=0.93 (0.95) = not unidimensional!

69. Conclusions Strength  Variance  Repeated measurement Location  Process  building an hierarchy

70. Respondents Items Direction of increasing ”X” Direction of decreasing ”X” Respondents with low ”X” Respondents with midrange ”X” Respondents with high ”X” Item indicating lowest level of ”X” Item indicating lower level of ”X” Item indicating higher level of ”X” Item indicating highest level of ”X” Construct Map

71. Relevant Theories for Research Utilization Studies

72. Theory of Planned Behaviour Attitudes Subjective Norms Perceived Behavioural Control Behavioural Intention Behaviour Ajzen & Madden, (1986), Journal of Experimental Social Psychology, 22, 453

73. Context Facilitation Evidence Evidence-based practice Promoting Action on Research Implementation in Health Services (PARIHS) Successful implementation is a function of evidence, context and facilitation Kitson et al, 1998 Rycroft-Malone et al, 2002 Rycroft-Malone et al, 2004 Research Clinical experience Patient experience Local data Context Culture Leadership Evaluation Purpose Role Skills Low High

74. Elements The Innovation Communication Channels Time Social System Stages Awareness Persuasion Decision Implementation Adoption 1931 - 2004 Diffusion of Innovation Influencing factors Attributes of the innovation Individual Organizational

76. Group Work: What does the theory tell us to measure? Crucial concepts in each theory? What is the dependent variable in the theory? Focusing on one theory – what concepts are important to measure? How do these concepts relate to research use? Are they a measure of research use? How would you measure research use according to the theory? What measurement approach would you take? Is any one theory adequate to guide the measurement of research use?