1. Connecting with international evidence: Measuring the importance of multiple database searching for BMJ Clinical Evidence Sarah L Greenley, Information Specialist, BMJ Knowledge, London UK sgreenley@bmjgroup.com Introduction Performing a rigorous literature search when conducting a systematic review is one way of attempting to reduce bias. If multiple databases are searched, there could be less risk of missing relevant studies and offering misleading conclusions. However, many systematic reviews still search only two databases 1 and even articles found in multiple databases (such as Medline and Embase) might not be retrieved due to different indexing practices 2. Additionally, literature searching can be time consuming, expensive, and dependent on readily accessible bibliographic databases, appropriate search strategies and even access to translation services to ensure that as many relevant studies as possible are retrieved. BMJ Clinical Evidence systematic reviews summarise the current state of knowledge and uncertainty about the prevention and treatment of clinical conditions, based on thorough searches and appraisal of the literature. A standard BMJCE literature search for new reviews and annual update searches consists of a search for pre-specified study types on Medline, EMBASE, The Cochrane Database of Systematic Reviews, CENTRAL, DARE and HTA databases as a minimum. If the topic is mental health related in addition Psycinfo is searched, along with Cinahl for topics appropriate for nursing. Compared with similar evidence-based resources this is a comprehensive search strategy although information specialists do not hand-search material due to the time-frame to conduct searches across more than 250 conditions on an annual basis. Previous work has looked at impact of single or reduced database searching on diabetes 3, clinical nutrition 4, family medicine 5, dentistry 6 and musculoskeletal conditions 7 but the broad nature of coverage of BMJCE puts it in a good position to look at the impact of different database searches across a range of specialities. In addition, the structure of BMJCE reviews allows new evidence to be quickly identified through the substantive changes section (see Fig 1) which details where new evidence has been added and identifies whether this evidence has led to a change in conclusions about the effectiveness of a particular treatment. This allows us to calculate the impact of a reduced database search on the conclusions of a review by highlighting those studies unique to single databases. For this study we retrospectively studied results included in BMJCE reviews following a standard BMJCE literature search and measured the impact of a reduced Medline and The Cochrane Library only search upon these references. Figure 1: sample substantive changes section from BMJCE As Clinical Evidence online reviews contain links where available to PubMed records, existence of a link adjacent to a reference cited in the reference section was taken as evidence that the reference was indexed on Medline. Where no Pubmed link was displayed, each reference was manually searched for inclusion on Medline. References missing from Medline were noted and the same reference was cross checked for inclusion on Embase, and the relevant issue of The Cochrane Library (including CDSR, DARE, HTA and CENTRAL) used at the time of the search. If the review topic was a mental health one, Psycinfo was additionally cross-checked. The percentage of references that would have been missed with a reduced version of the Clinical Evidence standard search (Medline/Cochrane only) was calculated for references as a whole, update references and update references that led to a change in categorisation. In addition, the number of references that were unique to Embase and Psycinfo at the time of the search were noted. Results The total number of references from the evidence section of 20 selected BMJ Clinical Evidence reviews was calculated as 667. For the 20 reviews studied, 12 of the 20 reviews (60%) would have missed references with a reduced Cochrane and Medline only search. (Fig 2) References 1 Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG Epidemiology and Reporting Characteristics of Systematic Reviews PLoS Medicine 2007 Vol. 4, No. 3, e78 doi:10.1371/journal.pmed.0040078Epidemiology and Reporting Characteristics of Systematic Reviews 2 Royle P, Waugh N. Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system. Health Technol Assess 2003;7(34). 3 Royle PL, Bain L, Waugh NR (2005) Sources of evidence for systematic reviews of interventions in diabetes Diabet Med 22(10):1386-93 4 Avenell A, Handoll HHG, Grant AM (2001) Lessons for search strategies from a systematic review, in The Cochrane Library, of nutritional supplementation trials in patients after hip fracture. Am J Clin Nutr, Vol. 73, No. 3, 505-510, 5 Wilkins T, Gillies RA, Davies K (2005) EMBASE versus MEDLINE for family medicine searches: can MEDLINE searches find the forest or a tree? Can Fam Physician 2005 Jun;51:848-9 6 Bickley SR, Glenny AM, (2003) The cochrane oral health group trials register: electronic searching and beyond. J Dent Educ 67(8):925-30. 7 Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus C. (2000) Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough Control Clin Trials 21(5):476-87. Compared to the standard BMJCE literature search, a reduced Medline and Cochrane only search would have missed 28 of 667 total references cited, (4.19%). Percentage references missing from individual reviews ranged from 0 (8 of the 20 reviews) missing to 14.29% of references missing (Seminoma) (Fig 3) Methods 20 selected BMJ Clinical Evidence reviews across the range of specialities were manually reviewed by an information specialist. The number and percentage of references contained in the evidence (benefits and harms) sections were calculated and references only appearing in the background section (aetiology, risk factors, incidence and prevalence, definition) were excluded as they are outside the scope of the Clinical Evidence Information Specialists systematic search. The final figure gave the total number of references which form the basis of the evidence and subsequent categorisation of interventions. The search date of the review was noted to check which issue number of The Cochrane Library was used for the original search. If a Medline only search had been performed, 50 references (7.5%) of the total would have been missed. 6 from 97 (6.8%) of references added at the last annual update of the review would have been missed from a Medline/Cochrane only search but most significantly, of the 15 references that led to a change in the conclusions about the effectiveness of an intervention, 4 (26.6%) would have been missed with a Medline/Cochrane search in place of the standard Clinical Evidence literature search.(Fig 3) These 4 references were: Diabetic foot ulcers and amputations: an RCT unique to Embase led to pressure offloading with felted foam being re-categorised as unknown effectiveness from beneficial. Tennis elbow:an RCT unique to Embase led to corticosteroids injections re-categorised from beneficial to trade off between benefits and harms. Postnatal depression: a systematic review unique to Psycinfo on psychodynamic therapy led to it being re-categorised from likely to be beneficial to unknown effectivness, Endmetriosis: an RCT unique to Embase on laparoscopic removal of endometriotic deposits led to the categorisation being changed from unknown effectiveness to likely to be beneficial. Discussion This research adds to the body of evidence recommending a search of multiple databases in order to minimise bias when searching for evidence. Although the implications of not searching databases such as Embase or Psycinfo might appear minimal in terms of the amount of references missed (4.19% of total references), their addition to the total body of evidence could have practice-changing results as highlighted here with four examples. Although this study attempted to choose example reviews across the range of specialties, it was a small subsection of the 250+ reviews in BMJCE and is therefore limited. Additionally, references classed as unique to EMBASE may have been subsequently added to CENTRAL for example – the objective of the study was to check what was available on databases at the time of the original search date and study the implications on the conclusions of a regularly updated compendium of evidence which bases conclusions on available evidence at the time of literature search.