2. Levels of Searching ADEPT Glasgow 2nd October 2003

4. Judging the Level of Search

5. Two epidemiologists are leaving a party late one night and one drops their car keys. The other following behind offers to help scrabble around on the ground to look for them. "Where did you drop them?" asks the second. "Over there" says the first, pointing to a spot some feet away. "Then why are we looking over here?" "Because there's a street light on over here" comes the reply.

6. Practical Examples from... Rapid Evidence Retrieval to Systematic Searching NES/SCPMDE and BEME Collaboration (Alex Haig) Beatson (Annette Thain) Others... Balancing sensitivity and specificity

7. Sensitivity & Specificity Sensitivity (recall) - percentage of “gold standard” Specificity (precision) - positive predictive value

8. Searching across the health disciplines Inter-professional direction Search experience is extendible –same core skills for systematic and rapid searches –core search skills are transferable

9. Medical Education Searching No filters Problematic study designs Grey literature in large quantities Dispersed sources –databases –grey literature –web –hand searches –expert opinion

10. the dissemination of information which allows medical teachers, institutions and all concerned with medical education to make decisions on the basis of the best evidence available the production of appropriate systematic reviews of medical education which reflect the best evidence available and meet the needs of the user, and the creation of a culture of best evidence medical education amongst individual teachers, institutions and national bodies.

11. Topic Review Groups High Fidelity Simulation in Medical Education The impact of feedback in formative and summative assessment and its effectiveness on the improvement of knowledge, skills and attitudes of health science students during their learning process and on teaching skills of tutors. What are the features of faculty development that make it effective?

12. Topic Review Groups What conditions are necessary or required for assessment and feedback to enhance physician improvement? A systematic review of educational interventions on the communication skills of medical clinicians Performance-based instruments that measure medical communication competence - an update and expansion of previous reviews.

13. Topic Review Groups Predictive values of assessment measurements obtained in medical schools and future performance in medical practice What does early clinical experience contribute to the basic education of health professionals? The impact of inter-professional education on health and social care practitioners, professional practice, patient/client related health and well being, and public health and social care outcomes.

14. International Educational Catholic inclusion of evidence

15. Psychologists estimate that we each make up to 1,200 decisions every day. Increasingly, professionals are being expected to base their decisions on evidence.

16. “Systematic Searching in Medical Education” September 2001, Berlin

18. Methods Topic - Feedback in assessment (BEME pilot and current topic review group, Barcelona/Tel Aviv) Software - Ovid [CGI version 7.8] Databases - Medline, Embase, ERIC

19. Methods - journal selected limited time and resources required a title that was most comprehensively indexed Academic Medicine 1996-present

20. Methods - strategies Three levels of strategy: ¶ standard (most users; limited search syntax) · enhanced (some use of search syntax) ¸ expert (full use of search syntax) syntax includes: free-text, controlled vocabulary, term explosions, phrase lists, subheadings, sub strings, filters, proximity operators, etc...

21. Methods - handsearching “...refers to the planned searching of a journal page by page (i.e. by hand), including editorials, letters, etc., to identify all relevant items.” time consuming and meticulous produces the “gold standard” by which search efficiency can be measured

22. Sensitivity Sensitivity (recall) - percentage of “gold standard” Sensitivity = total retrieved by search total of the hand-search Gold Standard = 46

23. Embase Sensitivity (n) Basic 4.3% 2 Enhanced 10.7% 5 Expert 15.2% 7 ERIC Sensitivity (n) Basic 0% 0 Enhanced 4.3% 2 Expert 6.5% 3 MedlineSensitivity (n) Basic 0% 0 Enhanced 10.7% 5 Expert 19.6% 9 GS=46

24. Specificity Ù Specificity (precision) - positive predictive value Ù Specificity = relevant records identified total retrieved by search

25. EmbaseSpecificity (n) Basic 40% 5 Enhanced 33% 15 Expert 30.4% 23 ERIC Specificity (n) Basic 0% 0 Enhanced 40% 5 Expert 37.5% 8 MedlineSpecificity (n) Basic 0% 0 Enhanced 31.3% 16 Expert 32.1% 28

26. A Note of Caution Academic MedicineAcademic Medicine is a journal that specialises in medical education : –more likely to be indexed for context –journal presents information for better retrieval Other journals will fare worse Other specificity scores for BEME pilots (not limited to one journal) ranged from 6 to 34%, with feedback in assessment at 17.8%

27. Reasons for shockingly poor performance  Incomplete coverage of journals Á No indexed database for medical education  Existing controlled vocabularies are inadequate for medical education

28. MeSH - Feedback (Mapped Term)

29. 1. Feedback, Biochemical Feedback, Biochemical: A mechanism of communication among life processes to co- ordinate development, reproduction, and homeostasis. In humans, feedback loops are especially important for communication between organs that are spatially separated. Virtually all hormones from the nervous and endocrine systems are under feedback control: by peripheral hormones, cations, metabolites, osmolarity or extracellular fluid volume.

30. 2. Feedback Feedback: A mechanism of communication within a system in that the input signal generates an output response which returns to influence the continued activity or productivity of that system.

31. Feedback, Psychological Feedback, Psychological: A mechanism of information stimulus and response that may control subsequent behaviour, cognition, perception, or performance. (From APA Thesaurus of Psychological Index Terms, 8th Ed.)

32. Knowledge of results (Psychology) A principle that learning is facilitated when the learner receives immediate evaluation of learning performance. The concept also hypothesises that learning is facilitated when the learner is promptly informed whether a response is correct, and, if incorrect, of the direction of error.

33. AND exp education, medical/ Feedback, biochemical - 0 Feedback -385 Feedback, psychological - 10 Knowledge of results (psychology) - 9

34. Results are transferable … to medical education searches and searches in general –lexical ambiguity of terms –inappropriate and/or inconsistent indexing –free text sacrifices specificity

35. Two unsatisfactory options: ignore the evidencebase your decisions on biased, irrelevant, or incorrect information hand search journals for every question 6 journals 10 years 30 minutes/issue = 27 working days

36. Two (slightly less) unsatisfactory options: design extremely broad search strategies time intensive sorting through results rely on partial evidence from searches perhaps the best bet

37. One barrier to systematic searching: Mortality

38. What is to be done?

39. Essentials for any Level of Searching: 1. Query formulation 2. Sources 3. Resources 4. Filters 5. Syntax

40. Define the search query Identify and expand concepts Set the scope of the query An Iterative Process: 1. Query formulation

42. McDonald S, Taylor L, Adams C. Searching the right database: a comparison of four databases for psychiatry journals. Health Libraries Review; 16: 151-6, 1999. 2. Sources

43. 3. Resources Dedicated time Administrative support Professional support

44. 4. Filters Know them Use them

45. 5. Syntax Know it Use it

46. OSCE Objective structure clinical examination Objective structure clinical exam$ Objective structure clinical test$ Multiple station examination… Simulated patient examination... Standardi#ed patient examination... GOSCE, OSLER, etc.

47. Searching at the Correct Level of Detail

48. What has been done? Raised Awareness –Amongst teachers, researchers, students, deans, other practitioners in med ed –Amongst information professionals and education technologists Published study guide for medical education Creation of search filters Completed systematic reviews METRO

49. Medical Education Taxonomy Research Organisation May 2002 UK wide initiative LTSN-01 funding METRO-1 complete; METRO-2 beckons

50. Annette Thain - Clinical Librarian Services Clinical teams had different needs for clinical librarian services Offered weekly search service after meetings SDIs Current Awareness Bulletins

51. Often very specialised questions to help with unusual /rare presentations or complications Want max of 10 papers Prefer easy accessible as need to read paper that week before next clinic

52. Would check – guidelines – Cochrane – Databases – Core/ respected journals – known authors – large trials – reviews – UK papers

53. Types of questions: Often wanted info for the patient, or to answer a question the patient had posed and the consultant needed the evidence. Treatment options often effect of a combination Clinical trials - often heard of trial at a meeting and wanting any papers published and results to know if it might affect their patient. Want patient information on a rare topic usually tried websites of associations if usual sources failed

54. “Rapid” Systematic Reviews Policy Evaluation Unit to provide the evidence base for Ministers to make decisions team of researchers and information professionals 85% of evidence retrieved and synthesised in 6-8 weeks the remaining 15% is added when available

55. fin