2. Pierre Pluye MD PhD Janique Johnson-Lafleur MSc Vera Granikov MLIS Vinita DSouza MSc Information Technology Primary Care Research Group

3. Objectives To share our research findings Study objectives: Validation of IAM in search context (PULL) How do family physicians use (apply) clinical information they retrieve?

4. Information Assessment Method (IAM) http://iam2009.pbworks.com

5. The bottom line ‘Acquisition-Cognition-Application’ model is relevant IAM can reveal the value of clinical information from knowledge resources e.g. InfoPOEMs Using clinical information for a specific patient was highly associated with: i) searches that were done to address a clinical question ii) meeting the objective of that search, and iii) searches where the cognitive impact was reported as "my practice was (will be) changed and improved" or "this information confirmed I did (will do) the right thing."

6. How do family physicians use (apply) clinical information they retrieve? A story of one search for clinical information ‘acquisition-cognition-application-outcomes’ Family physician from Edmonton On May 26, 2008: a young man with knee pain Reason for search: to look up something I forgot

7. Vignette “It was a young patient […] in his early 20’s […] He liked to run, and he had developed knee pain because of running. He didn’t want to stop running, but he was wondering what he could about the knee pain I knew there are specific exercises good for this type of knee pain, and so I was curious to search to see if there were any studies on specific exercises. […] I had forgotten what specific exercises could be done.”

9. IAM Search objectives Cognitive impact Use Patient health outcomes COMMENT (optional) Response format = ‘yes’ or ‘no’

10. Vignette Cognition: Practice improvement “I was able to be more specific about my recommendation. Rather than saying generally to a patient You need to go exercise, I said You need to do this type of exercise.” Application: Information was used as follows: “I advised the patient on the type of exercises that he needed to do. […] I provided a new exercise program.” Outcomes: Regarding patient health, MD16 reported the information would avoid unnecessary or inappropriate treatment, prevent further deterioration, and improve patient health or functioning.

11. Information Assessment Method (IAM) CONCEPTUAL / THEORETICAL FOUNDATION

12. AcquisitionCognitionApplication Saracevic T and Kantor KB. Studying the value of library and information services. Part I. Establishing a theoretical framework. Journal of the American Society of Information Science 1997;48:527.

13. Acquisition “The process of getting information or objects potentially conveying information, as related to some intentions.” e.g. A physician searches for an answer to a clinical question, within an electronic knowledge resource.

14. Cognition “The process of absorbing, understanding, integrating the information.” e.g. Cognitive processes that take place during reading and continue for some time afterward.

15. Application “The process of (potential use) of this newly understood and cognitively processed information.” e.g. Based on the information acquired and cognitively processed, the physician changes the management of a specific patient.

16. The impact of technology-enabled knowledge translation 41 FPs in practice 1-year mixed methods study IAM linked to Essential Evidence Plus on PDA 1,767 rated searches (nested within MDs) 3,300 rated information hits (nested within searches)

17. AcquisitionCognitionApplication Information Assessment Method

18. Searches and hits per MD NMeanRange Rated searches / MD40 MD44.2 (30.8)6 - 148 Rated hits / MD40 MD82.5 (63.7)9 - 294 Rated hits / rated search1,767 rated searches 3,300 rated hits 1.9 (1.7)1 - 22

19. AcquisitionCognitionApplication

20. Reason Address a clinical question / problem / decision about a specific patient1,310 (74%) Look up something I had forgotten672 (38%) Share information with a patient/caregiver624 (35%) Exchange information with other health professionals520 (29%) Search in general or for curiosity496 (28%) Fulfill an educational or research objective434 (25%) Plan, manage, coordinate, delegate or monitor tasks with other health professionals 197 (11%) Reasons for Searches Conducted by Family Physicians

21. Effect of participants age on Reasons for Search: “Forgotten” Random Effects (Logistic) Regression Model Model Look up something I had forgotten ~ Age Parameter estimate (SE) P ValueStandard Error Age (years)0.02670<0.00010.006029

22. AcquisitionCognitionApplication

23. AcquisitionCognitionApplication

24. AcquisitionCognitionApplication Types of Cognitive Impact 7,275 reported cognitive impacts linked to 3,300 rated hits This information confirmed I did (will do) the right thing151646% I was reassured146845% I learned something new124638% I recalled something113634% My practice was (will be) changed and improved96329% No Impact78024% Negative Impact (all four types)1665%

25. AcquisitionCognitionApplication Negative Cognitive Impact: 166 of 3,300 rated hits (5%) I was dissatisfied, as this information had no impact on my practice79 (2.4%) I was dissatisfied, as there was a problem with this information67 (2.0%) I think this information is potentially harmful13 (0.4%) I disagree with this information7 (0.2%)

26. How does achieving the search objective influence cognitive impact? Objective met in 1,336 searches (75.6%) containing 2,482 hits (75.2%) Cognitive impact n=3,300 rated information hits Objective met n= 2,482 (%) Objective not met n= 818 (%) My practice was (will be) changed and improved899 (36.2)64 (7.8) I learned something new1,104 (44.5)142 (17.4) This information confirmed I did (will do) the right thing1,378 (55.5)138 (4.2) I was reassured1,351 (54.4)117 (14.3) I recalled something1,021 (41.1)115 (14.1) I am dissatisfied, as this information has no impact on my practice10 (0.4)69 (8.4) I am dissatisfied, as there is a problem with this information21 (0.9)46 (5.6) I disagree with this information6 (0.2)1 (0.1) I think this information is potentially harmful11 (0.4)2 (0.2) This item of information had no impact at all on me or my practice288 (11.6)492 (60.2)

27. Search variables associated with the use of that information for a specific patient Random Effects (Logistic) Multivariate Regression Final Model Use of information ~ Situational relevance + Five Search Objectives Parameter estimate (SE)P ValueOdds Ratio [95% CI] Search objective 12.5974 (0.1795)<0.000113.4 [9.4 – 19.1] Search objective 2-0.3591 (0.1634)0.02810.7 [ 0.5 – 1] Search objective 3 -1.2161 (0.1566)<0.00010.3 [0.2 – 0.4] Search objective 4 0.3064 (0.1481)0.03881.4 [1 – 1.9] Search objective 5 0.6047 (0.1618)0.00021.8 [1.3 – 2.5] Search objective was met2.3341 (0.1964)<0.000110.3 [7 – 15.2] 1.To address a clinical question 2.To fulfill a research objective 3.Search in general or for curiosity 4.Look up something I had forgotten 5.To share information with a patient / caregiver

28. AcquisitionCognitionApplication

29. AcquisitionCognitionApplication 1,708 rated hits (52%) nested in 1,109 searches were (or will be) used for a specific patient

30. Is ‘Cognitive Impact’ associated with the use of information for a specific patient? Cognitive impact n=3,300 Used for a specific patient n= 1,708 (51.8%) Not used for a specific patient n=1,592 (48.2%) My practice was (will be) changed and improved709 (41.5)254 (16.0) I learned something new756 (44.3)490 (30.8) This information confirmed I did (will do) the right thing 1081 (63.3)435 (27.3) I was reassured1027 (60.1)441 (27.7) I recalled something803 (47.0)333 (20.9) I am dissatisfied, as this information has no impact on my practice 23 (1.4)56 (3.5) I am dissatisfied, as there is a problem with this information 22 (1.3)45 (2.8) I disagree with this information2 (0.1)5 (0.3) I think this information is potentially harmful5 (0.3)8 (0.5) This item of information had no impact at all on me or my practice 106 (6.2)674 (42.3)

31. Association between items of ‘Cognitive Impact’ and the use of information for a specific patient Random Effects (Logistic) Multivariate Regression Final Model Use of information ~ Cognitive Impact Parameter estimate (SE) P ValueOdds Ratio [95% CI] My practice was (will be) changed and improved 1.2136 (0.1346) <0.00013.4 [2.6 – 4.4] I learned something new -.4256 (0.1335) 0.00150.7 [0.5 – 0.9] This information confirmed I did (will do) the right thing 0.8119 (0.1220) <0.00012.3 [1.8 – 2.9] I recalled something 0.3028 (0.1202) 0.01191.4 [1.1 – 1.7]

32. Next steps? IAM in the EMR