1. Searching beyond the RCT - looking for sibling studies on qualitative, economic and process research Faten Hamad and Christine Urquhart

2. Introduction Systematic reviews – producing the evidence …We have the evidence on what works (or doesn’t) – what happens next? Finding qualitative, economic and process research evidence – preliminary findings of research to identify sibling studies associated with particular randomised controlled trials.

3. Aims and objectives of presentation are to: Discuss searching strategies to find evidence beyond the randomised controlled trial Present preliminary findings Assess your reactions to the findings!

4. Background Sibling studies is the name which has been chosen to indicate the relationship that groups a set of related studies (randomised control trial, qualitative, process and economic evaluations). An intervention may be viewed as: “a complex system where intervention itself is a fragile creature that is delivered in a social system of interacting elements, such as an individual’s capacity, interpersonal relationships, institutional setting and infrastructure.”

5. Search strategies and search filters Existing work by Cochrane Information Retrieval Methods Group and others (e.g. Hedges team) Aim to maximise recall (sensitivity), keeping precision reasonable, and ensuring specificity Search strategies vary with the database – depends on index terms, which terms have to be added as free text terms… Relevance judgements

6. Seed studies selection We chose a range of topic areas, and dates for the large RCTs and chose five seed studies with different characteristics. We will discuss the result of the following two of seed studies: -Telemedicine and diabetes (a known RCT, with many known direct siblings, that could be used to validate and checking the search strategies for their sensitivity). - Tamoxifen for breast cancer prevention (known qualitative sibling, two RCTs involved).

7. Search strategies and databases under investigation Different search strategies and different databases were explored for the reason of exploring the differences in search performance in regard to different study areas. - Author-Subject search (using a very simple subject term combination) with each of the author names in the seed article in turn, in MEDLINE on PubMed. -Related article search in MEDLINE on PubMed (for the seed article). - E-library search (with a combination of ISI (WoS), OCLC WorldCat, OCLC Articles First, EBSCO Business Complete, and EBSCO International Bibliography) - simple subject term combination only, and limited to the first 300 documents retrieved.

8. Search strategies and databases under investigation-Cont. - SCOPUS search (author-subject search as in MEDLINE on PubMed). - CINAHL (author-subject search as in MEDLINE on PubMed). - Cited reference in ISI, Web of Science (with the seed article as the reference). - Cited reference search in SCOPUS. - Cited reference search in CINAHL.

9. Results Table 1 IDEATeL study: search strategy and retrieval performance search strategy Relevant/RTotal retrieved Unique Relevant/R Odds Ratio Related Search(PubMed)2518651.01219 Author +Subject(PubMed)2315741.09189 Citation(Web of Science)145711.86855 Subject search(e-library)39296331.19589 SCOPUS Author subject3952523.64179 SCOPUS citation3264127.13514 CINAHL Author subject17180100.85393 CINAHL citation4706.90196 Total relevant retrieved without duplicates106 Total retrieved without duplicate634 Total non-relevant retrieved without duplicates528

10. Table 2 IDEATeL study search strategies retrieval per study type and odds ratio calculations. search strategyRCTsQualitativeEconomical evaluation Process evaluation R1R1 R/N 2 OR 3 R1R1 R/N 2 OR 3 R1R1 R/N 2 OR 3 R1R1 R/N 2 OR 3 Related Search(PubMed)8390.103210210.2396380.18874130.1548 Author-Subject(PubMed)8390.12719220.2534290.13774130.1906 Citation(Web of Science)4430.19908230.7442290.47550170 Subject search(e-library)25220.203420110.3254920.80556110.0976 SCOPUS Author - subject17302.920512193.2551291.1453894.5812 SCOPUS citation19281.56928230.8044290.51393140.4955 CINAHL Author subject103724.05442713.1851108.921511.867 CINAHL citation1460.73913283.642901100170 1 : Relevant retrieved. 2: Relevant not retrieved. 3: Odds Ratios.

11. Table 3 Tamoxifen study: search strategy and retrieval performance search strategy Relevant/RTotalUnique Relevant/R Odds Ratio Related Search(PubMed)17200130.58277 Author+Subject(PubMed)1745170.24573 Citation(Web of Science)55320.60155 Subject search (e-library)59288542.316121 SCOPUS Author subject72229474.76069 SCOPUS citation45920.41722 CINAHL Author subject194175.49701 CINAHL citation1401.87527 Total relevant R without duplicates156 Total retrieved without duplicate1028 Total non-relevant R without duplicates872

12. Table 4 Tamoxifen study search strategies retrieval per study type and odds ratio calculations. search strategyRCTsQualitativeEconomicProcess evaluation R1R1 R/N 2 OR 3 R1R1 R/N 2 OR 3 R1R1 R/N 2 OR 3 R1R1 R/N 2 OR 3 Related Search(PubMed)12830.10983310.073510DIV/ 0 1250.0304 Author-Subject(PubMed)12830.04631330.00970104220.0582 Citation(Web of Science)2930.06772320.19660101250.1258 Subject search(e-library)26690.162918160.486401013 0.4323 SCOPUS Author-subject52430.647010240.222901010160.3344 SCOPUS citation09504300.36850100260 CINAHL Author subject15801.16761330.18870103230.8123 CINAHL citation1940.075003400100260 1: Relevant retrieved. 2: Relevant not retrieved. 3: Odds Ratios.

13. Discussion CINAHL author-subject search was the most effective search that can precisely retrieve direct and indirect siblings of a certain seed study in general with higher score in the case of the IDEATeL study in specific. - IDEATeL, the number of retrieved records was 41 (with 19 relevant); - Tamoxifen, the number retrieved was 18 (with 17 relevant); CINAHL author- subject odds ratios were quite different, but this does not affect the fact this search strategy was the best search strategy, in terms of the chances of finding relevant material from a search, with proportionally fewer irrelevant items retrieved.

14. The SCOPUS author–subject strategy was the second best search performance. For both the seed studies, the e-library subject search retrieved most of the unique studies. Simple subject terms and / or the combination of simple subject terms and author names for each seed study, appeared to be the most effective method of retrieving most of the siblings, outperforming citation searching (apart from SCOPUS citation with the IDEATeL seed study).

15. Conclusions and future works The analysis indicates that there is neither a winner in the search strategies nor for the databases. The CINAHL author-subject performed well (in terms of precision). The SCOPUS author-subject search performed the next best, (higher recall for the relevant studies for both seed studies). The e-library author-subject search produced a good number of relevant studies (and a high proportion of unique items as well).

16. Conclusions and future works Further research will examine how expanding the number of databases and changing the selection will affect the relative performance of the e-library Meta-lib search. Further work is needed to identify how grey literature, conference proceedings and thesis and dissertation material can be obtained efficiently, as Web of Knowledge found some of the direct sibling publications for the IDEATel studies that could not be obtained on any of the other databases used.

17. Thank you for Listening