1. Net Explorations Finding the Internet Evidence: a program evaluation study http://dfcm.med.utoronto.ca/library/netexp/ http://dfcm.med.utoronto.ca/library/netexp/ Rita Shaughnessy MLS – U of T DFCM Anita Lambert-Lanning MLS – U of T DFCM – CFPC Jennifer McCabe MD, CCFP – U of T DFCM Chris Cavacuiti MD CCFP – U of T DFCM Risa Freeman MD, CCFP - U of T DFCM

2. Goal of Net Explorations Workshop Optimize web-based search skills using two types of tools – Medline d-base and a Web search engine Objectives of Workshop 1.Formulate a clinical scenario then refine into an appropriately worded search question. 2.Identify the question type & the corresponding filtering terms (search strategy) 3.Discuss the validity of information found from a FM research viewpoint. 4.Complete a critical evaluation of one (1) document found during their search using the Educational Prescription (EduRx )

3. critical appraisal, EBM, EBP,knowledge transfer (brief) Teaching ebm skills Teaching critical appraisal skills in health care settings. Parkes J, Hyde C, Deeks J, Milne R. The Cochrane Database of Systematic Reviews 2001, Issue 3. Art. No. CD001270. (sys review) Does having a librarian at PBL sessions improve understanding of the medical lit & info access skills? Koufogiannakis D et. al. UofAlberta 2004 presented at CHLA St. Johns (controlled trial) Did educational EBM initiative with OT students at Queens develop positive perceptions re: EBP? Vileneuve & Maranda JCHLA 2005 26:13-26. (qualitative evaluation-focus groups) Validation of the Fresno test of competence in evidence based medicine – knowledge testing re: EBM skills using open ended ? Ramos et.al. 2003

4. Hypotheses (multiple) CCs at UofT are learning appropriate search techniques Using a MEDLINE vs GOOGLE approach is a reasonable way to teach EBM searching “ Net Explorations ” is an appropriate way to support the academic project in FM “ The EduRx ” is an appropriate way to expose FM research to undergrads at UofT

5. Multi-method evaluation S-D variables of CCs Satisfaction levels with the workshop components Scoring of 136/201 EduRxes Assigned subject headings to questions posed

6. Demographics 05-06 based on 127/201 (63%) who completed workshop evaluation Age Range 20-40 yrs Mean 25 yrs Gender Male 45.7 % Female 52.0% Missing data 2.3% Status 96.9% 3 rd yr CCs Missing data 3.1

7. CC satisfaction workshop (N=127)

8. CCs 1-word descriptors of workshop – 104 positive terms Excellent, useful, informative, helpful, organized, relevant – 5 neutral terms Average, fair, ok – 15 negative terms Could have been compressed into shorter time Redundant ineffective too much in too little time 42 Additional comments/suggestions

9. EduRx scoring template Clinical _Scenario1 point Search_Q1 point Qtype1 point Predict search_Outcome1 point Q1 search strategy explained2 points only Q2 URL or Citation1 point (either point directly to the document found with a URL or enough citation information) Q3 valid Family_Practice 1 point ----------------------------------------------------------------------- Total possible points8 points

10. EduRx Results: Question type identified by CC

11. EduRx Results: Source of Document found N=136

12. EduRx Results: Applicability to FM (N=136)

13. Results: EduRx scores # % %03-04 poor (1-2 pts) 0 0 1.7 fair (3-4 pts)1511.0 16.4 good (5-6 pts)5540.4 48.0 excellent (7-8 pts)6648.6 33.9 Total 136 100100 anita rita chris

14. Most frequently Occurring Subject Headings in the EduRx (n=78 unique subjects/136 search exercises) 8 – Issues in pregnancy 6 - Colorectal Neoplasms 6 - Hypertension 5 - Depression 4 - Breast Neoplasms 4 - Common Cold 4 - Otitis media 4 - Prostatic Neoplasms 3 - Contraceptive devices 3 - Diabetes Melletus, type 2 3 - Headache 3 - Hyperlipidemia 3 - Obesity – Weight Loss 3 - Ovarian neoplasms 3 - Pertussis vaccine 3 - Sexually transmitted diseases 3 - Skin neoplasms 3 - Smoking Cessation

15. Discussion: Are CCs learning appropriate search techniques? Scores on EduRx indicate yes but 64% did not get full marks for search strategy EBM searching using MEDLINE vs GOOGLE Good exposure to advantages vs disadvantages We believe that “ Net Explorations ” is an appropriate way to support the academic project based on continued positive feedback from clerks

16. Limitations: Still in pilot phase of testing the scoring method Scoring takes up way too much time! Only 63% provided evaluations of the workshop Quota sampling method used to randomly select therapy ? Types has some bias (e.g. difficulty determining the amount of sampling error)

17. Conclusions: >% of females to males Increase in average number of hours spent on the internet Consistently high scores for the three major elements of workshop (PubMed, Google, EduRx) The EduRx always gets lower ranking than other parts of workshop Only 36% got full marks for the search strategy part of the EduRx Need to re-examine scoring system Subject headings continue to show wide diversity of FM topics

18. Webcast for rural/remote CCs “ Captivate ” software used Labour intensive process Initial feedback and assessment ’ 05 - 06 –Faculty – favourable to webcast in principle –Students – very amenable to web-based format –Technical problems with remote access –Current webcast quality poor –Continuous updating required Webcast experience

19. Thanks for listening Rita Shaughnessy – r.shaughnessy@utoronto.ca Anita Lambert-Lanning – all@cfpc.ca Net Explorations workshop: http://dfcm.med.utoronto.ca/library/netexp Department of Family & Community Medicine University of Toronto Canada

20. Techniques used in Workshop Conducted by a health librarian (RS or AL-L) in a demo fashion using an EBM approach to searching using Pubmed Clinical Queries and a web search engine (Google or Scirus). Faculty support provided by physicians and HS librarians UofT Practice time of 1 hour provided to complete their own search & critical appraisal (EduRx)

21. SAMPLE EDURX SCENARIO:A 25-year old G1P0 female with a history of asthma presents with cough and shortness of breath. Upon further history, it becomes evident that she has stopped using Flovent or Ventolin to control her symptoms in fear that the medications will impact the development of her baby. QUESTION: How do you manage asthma in pregnancy to adequately control the mother's symptoms without putting the fetal well-being at risk? QTYPE: Guideline HYPOTHESIS STATEMENT: I hope to find evidence showing that by and large the asthma management during pregnancy will be similar in nonpregnant women, with the exception of some medications that may be preferred over others. I also hope to find information outlining an approach to pregnant mothers with asthma in the clinical setting to ensure effective management of their asthma. SEARCH STRATEGY: Using the PubMed Search Engine, I decided to search for a systematic review outlining recent guidelines to the management of asthma in pregnancy, and also to determine whether there is a consensus amongst the different studies done. To do so, I selected "Clinical Queries" on PubMed --> under the heading "Finding Systematic Reviews," I typed "asthma AND pregnancy AND (management OR treatment)." I then limited my search to articles that were English, dealth with Humans, and from 1997-2007. This search resulted in 27 hits. DOCUMENT SELECTED: NAEPP Expert Panel Report: Managing Asthma During Pregnancy: Recommendations for Pharmacologic Treatment — 2004 Update Journal of Allergy and Clinical Immunology, Volume 115, Issue 1, January 2005, Pages 34-46 RELEVANCE TO FM: Yes this document is very relevant to Fm HOW IS IT RELEVANT OR NOT: Family physicians play a major role in prenatal care, which involves ensuring the well-being of both the mother and the fetus throughout the duration of the pregnancy. Asthma is a very common complication that occurs in pregnant women. The information in this article provides the most recent guidelines for managing asthma in pregnancy, including drugs that should be considered, as well as emphasizing the importance of monitoring lung function in expectant mothers throughout the course of their treatment.