Presentation on theme: "Rapid Response Health Technology Assessments Speed Versus Systematic Searching Hayley Fitzsimmons firstname.lastname@example.org January 30, 2009 OLA Super Conference."— Presentation transcript:
1 Rapid Response Health Technology Assessments Speed Versus Systematic Searching Hayley FitzsimmonsJanuary 30, OLA Super Conference 2009
2 Agenda Background Evidence-based medicine movement Role of the Canadian Agency for Drugs and Technologies in Health (CADTH)CADTH’s rapid response HTIS serviceSpeed versus systematic searching: Problems & issuesTechniques used by the rapid response Health Technology Inquiry Service (HTIS) to ensure focused yet comprehensive searchingLimitsDatabase selectionFiltering for study types (i.e. Systematic reviews or randomized controlled trials)Selective grey literature searchConclusion
3 Evidence-Based Medicine To make decisions about:what interventions to prescribe to a patient?what interventions to fund with public money?what diagnostic tool to purchase for your health care institution?..base your decision upon the best evidence available.
4 Evidence-Based Medicine “Over two million articles are published annually in the biomedical literature in over journals”Ad Hoc Working Group for Critical Appraisal of the Medical Literature. Academia and clinic: a proposal for more informative abstracts of clinical articles. Ann Intern Med 1987;106:So do you want to wade through all the evidence or would you rather have it done for you by trained researchers?
5 Evidence-Based Medicine Best evidence comes from:systematic review (SR)meta-analysis (MA)health technology assessment (HTA)All three types of publications synthesize the relevant studies (usually but not exclusively RCTs) and apply strategies to help limit the bias which may occur in an individual study.These 3 types of reports do the dirty work of wading through all the available evidence and synthesizing the findings in one place
6 Evidence-Based Medicine SR: A review of studies in which evidence has been systematically searched for, studied, assessed, and summarised according to predetermined criteria. (NLM)MA: Systematic methods that use statistical techniques for combining results from different studies to obtain a quantitative estimate of the overall effect of a particular intervention or variable on a defined outcome. (NLM)HTA: Technology assessment in health care is a multidisciplinary field of policy analysis. It studies the medical, social, ethical, and economic implications of development, diffusion, and use of health technology. (INAHTA)Metaanalysis is often part of a Systematic review but not necessarily. A systematic review does not necessarily include a quantitative summary of the results AlWAYS found in a MA.HTAs are the most valuable to decision makersNot only do HTAs address clinical benefit and safety, but they also look at the economic and/or social impact of the intervention in questionHTA Can be just one health technology or comparison of manyIf find good/recent SR, MA, or HTA on exactly your question – often the evidence contained within is good enough for you to base you health care decisions – especially if it is Canadian and has economic component
7 Evidence-Based Medicine Health Economic Reviews important because:Systematic reviews, meta-analysis, Clinical trials and guidelines are concerned with clinical benefits and harm and do not necessarily address the cost issue that most decision makers need in order to make an informed decision on spending limited public health care funds.High up on our checklistHealth economic reviews not part of clinical evidence literature but VERY useful to decision makersReviews or structured abstracts of economic studies can be found in a couple of specialized databases such as NHS EED (in handout)
8 Hierarchy of Evidence When no SRs, MAs, or HTAs available: Use hierarchy of evidence chart to find the best evidence possible as high up the ladder as possible.For example, if we find well designed RCTs on a topic, it is not normally necessary to look at the observational studies in order to make the most informed decision possible.Occasionally RCTs are not an ethical or practical means of studying a topic. A good example of this is the famous cohort study done by Hill, Doll, and Peto in the fifties which followed for many years, 40,000 male doctors divided into four cohorts: Non smokingLight smokersModerate smokersHeavy smokers
9 Evidence-Based Medicine Where do clinical practice guidelines fit on the hierarchy of evidence ladder?Old Style Guidelines or Consensus StatementsGOBSAT (Good old boys sitting around a table)Not necessarily based upon clinical evidenceOpinions of clinical expertsBottom rung of hierarchy of evidence ladder “Case Studies, anecdote and personal opinion”Many recommendations made in guidelines were later disproved in clinical trialsLignocaine (in MI for preventing arrhythmia) example of clinical practice that was used even after research proved it to be ineffective or even harmful70’s advocated Lignocaine for this use74 research showed no benefit and in some cases caused fatal arrhythmiasLignocaine continued to be routinely used for this indication until mid-80sHowever sometimes all you have is the opinions of the clinical experts !!
10 Evidence-Based Medicine New Style Guidelines or Consensus StatementsGuidelines produced today have been heavily influenced by the evidence-based medicine movementGuidelines are beginning to mimic systematic reviewsEach guideline needs to be individually assessed as to the thoroughness in which it reviews the available evidence.MAY fall on top rung of hierarchy of evidence ladder!Guidelines now tend to rely upon the available research evidence as well as expert opinionsGood example of this are the guidelines from NICE (Nat’l Institute for Clinical Evidence) UKThe National Guidelines Clearinghouse gives summaries of guidelines and reviews the evidence contained within (in grey literature checklist)
11 Evidence-Based Medicine Summary “Health care providers, researchers, and policy makers are inundated with unmanageable amounts of information; they need systematic reviews to efficiently integrate existing information and provide data for rational decision making.”BMJ 1994;309(6954):597 (3 September)
12 Canadian Agency for Drugs and Technologies in Health CADTH MandateThe Canadian Agency for Drugs and Technologies in Health (CADTH) provides Canada’s federal, provincial and territorial health care decision makers with credible, impartial advice and evidence-based information about the effectiveness and efficiency of drugs and other health technologiesTo achieve its mandate CADTH produces a wide variety of publications from comprehensive systematic reviews and economic evaluations to more concise bulletins and rapid response updates.Stakeholders ie Canadian publicly funded health-care system – have policy decisions for which they need questions answeredCADTH services are available to Canadian health care decision makers in the federal government, provincial health ministries, Local Health Integration Networks, regional health authorities, hospitals, and national and federal health care programs in CADTH-area jurisdictions.We CADTH do the research in order to find the best evidence to answer their questions
13 CADTH Technology Reports: Pros & Cons ComprehensiveExpensive to produceExtensive quality review both internal and externalSlow: takes an average of months to completeReviews all aspects of a health technology including clinical effectiveness, cost-effectiveness, social, ethical, and economic issuesNot feasible to produce if there is not enough published evidence available (i.e. a new technology)CADTH’s Technology Reports:represent CADTH’s most comprehensive assessments of health care technologies and servicesexamine more mature technologies, for which there is a larger and higher quality body of evidence availableclinical effectiveness and cost-effectiveness are examined, as well as the social, ethical, and economic implications of its use, compared with alternative treatmentsextensively peer reviewed by internal staff, and by external clinical and methodological experts.
14 CADTH’s Rapid Review Health Technology Inquiry Service (HTIS) ProsConsFastNot as comprehensiveAble to handle new /rare technologies without much published evidenceLess rigorous quality review (depending on the report level)InexpensiveLess detailed reportsShorter reportsLevel one reports do not include a summary/synthesis of the results (bibliography only)February 2005:In response to a growing demand from Canadian health-care administrators and decision makers to support quick decision making and tight timelines CADTH launched its’ rapid response, Health Technology Inquiry Service (HTIS)The goal of the HTIS is to provide Canadian health care decision makers with HTA information, based on the best available evidence, in a quick and efficient manner
16 Types of HTIS RequestsHTIS responds to inquiries on the assessment of health care technologies including drugs, devices, diagnostic tests, and medical and surgical procedures.
17 CADTH’s Information Services department Our job:ensure researchers have all the available evidence on a topicfailure to retrieve and analyze relevant materials can seriously compromise the quality of our reportstime constraints of HTIS rapid reviews makes comprehensive retrieval challengingThe IS department at CADTH has 17 employees - literature searching and article retrieval for all CADTH reports. 4.5 employees are devoted exclusively to THIS.Canadian health care decision makers rely upon CADTH reports to make informed decisionsCrucial that reports are based upon sound evidence and if the evidence is unavailable that it is communicated.
18 Speed versus systematic searching: Problems & issues HTIS Rapid Response SearchesHTA Systematic SearchesLess time to search, especially for lower level reports which takes an average of two daysTakes a long time: Need to be comprehensive, pressure to retrieve everything of relevanceCannot give research assistants/officers thousands of references for broad search requests, as they do not have the time to screen themNormal to retrieve thousands of articles for research assistants/officers to screenVery little or no time allotted for hand searching relevant studies (report dependent)Time allotted for hand searching bibliographies of relevant studiesDue to tight timelines HTIS only searches major grey literature sitesHTA searches an extensive list of major and minor HTA sites as well as other relevant grey literature websitesHTAs are not usually indexed in databases so it is important to search the internet grey literature for this crucial type of evidence
19 Speed versus systematic searching: Problems & issues Searching for rapid review HTIS programMain problemTo find all the relevant evidence on subject in a timely manner, without giving the researchers so much information that they cannot meet tight HTIS rapid review timelines
20 Disclaimer & Methods Section Applying Limits Database Selection Techniques used by HTIS Searchers to Provide Quick, Focused Results with Minimal ErrorDisclaimer & Methods SectionApplying LimitsDatabase SelectionFiltering Searches for Study TypeSelective/Focussed Grey Literature Search
21 Disclaimer Section of HTIS Disclaimer on searching: … HTIS responses are based on a limited literature search and are not comprehensive, systematic reviews. The intent is to provide a list of sources of the best evidence on the topic that CADTH could identify using all reasonable efforts within the time allowed…..
22 Methods SectionHTIS methods example: A limited literature search was conducted on key health technology assessment resources, including PubMed, the Cochrane Library (Issue 4, 2008), University of York Centre for Reviews and Dissemination (CRD) databases, ECRI, EuroScan, international health technology agencies, and a focused Internet search. Results include articles published between 2003 and November 2008, and are limited to English language publications only. Filters were applied to limit the retrieval to systematic reviews, health technology assessments, meta-analyses and guidelines.
23 Applying Limits Guidelines 5 years English Normally the searches that we do are very broad for example a level 1 we recently did “chronic pain management for outpatients.”We limited the search by:Guidelines5 yearsEnglishWithout limits we would retrieve thousands of articles and researchers would spend weeks weeding through all the literature and miss deadlines!
24 Applying Limits - Human limits Rapid review HTIS requests typically have the following limits:5 yearsEnglish languageHumanStudy type (topic dependent)CADTH Systematic review Technology Reports generally do not have date or language limits:- Human limitsStudy type (occasional - topic dependent)
25 Applying Limits If there is too much information on the topic; if lower-evidence study types such as observational studies have been requested negotiate to see if they may be removed (especially if there are a lot of higher quality study types available)consider removing articles of the following document types:CommentsLettersCase reportsconsider shortening the date range (negotiated with the requestor)discuss narrowing the topic with the requestor
27 Databases searched for HTIS reports Level 1 & 1.5:Medline and/or PubMed (98% overlap)Cochrane Database of SRs (unless SRs not required)Centre for Reviews and Dissemination (CRD) – University of YorkOther major databases are only searched if little or nothing is found in MedlineLevel 2:If retrieval is small in Medline up to two other relevant databasesLevel 3 & 4:Medline and PubMed (98% overlap)EmbaseCochrane LibraryTwo or more relevant databasesLower level reports typically do not search as many databases as the higher level HTIS reports, in order to limit retrieval and meet timelines
28 Filtering for Study Type Searching for most CADTH reports is limited by study design, some of the most common include:Systematic Reviews/HTAs/Meta-analysisRandomized clinical trialsControlled trialsObservational trialsClinical practice guidelinesEconomic studiesIn HTIS we add filters to limit to higher evidence only (systematic reviews, health technology assessments etc).
29 Limiting by Study Design using Filters Filters are primarily used to limit searches to a certain study design for example the following PubMed filter to restrict searches to systematic reviews and/or HTAs.:Systematic Reviews and HTAs Filter:systematic[sb] OR (meta-analysis[pt] OR meta-analysis[mh] OR meta-analysis[tw] OR metanalysis[tw]) OR meta analy*[Title/Abstract] OR metaanaly*[Title/Abstract] OR met analy*[Title/Abstract] OR metanaly*[Title/Abstract] OR integrative research[Title/Abstract] OR integrative review*[Title/Abstract] OR integrative overview*[Title/Abstract] OR research integration*[Title/Abstract] OR research overview*[Title/Abstract] OR collaborative review*[Title/Abstract] OR collaborative overview*[Title/Abstract] OR systematic review*[Title/Abstract] OR health technology assessment*[tiab] OR "Technology Assessment, Biomedical"[mh] OR HTA*[tiab] OR "Cochrane Database Syst Rev"[Journal:__jrid21711] OR "health technology assessment winchester, england"[Journal]
30 Filtering for Study Type No filter is infallible!We tested/validated our SR/HTA filter with the help from the Hedges group at McMaster (PubMed filters are based upon Hedges filters) and found it was approximately 90% precise and had a 46.5% sensitivity score. This translates to:retrieving about 90% of the SRs/HTAs available in PubMedapproximately 46% of the articles we retrieve will NOT be SRs or HTAs.the more precise the filter the less sensitive it is (more junk!)Filters modified for HTIS use tend to be more precise but less accurate
31 Filtering for Study Type There are three places to search when looking for study design, our filters attempt to cover all three possibilities:Title/abstract fieldsPublication type fieldSubject heading field (e.g. MESH, or Emtree)
32 Example #1 study design found in all three fields:
34 Finding Validated Filters InterTASC Information Specialists' Sub-Group: Search filter resourceProvides an extensive bibliography of published filters available by type (i.e. RCT), database (i.e. Embase) and search platform (i.e. PubMed, Ovid, etc.)PubMed also has a few pre-loaded filters:Search of the literatureInterTASC is a subgroup of HTAI - Health Technology Assessment International Association
35 Observational Study Filters Listed in InterTASC
36 What is Grey Literature ConferencesThesesProposalsDraftsStatisticsInternal ReportsOpen access journals (un-indexed)Government documents (including HTAs)Press ReleasesHow do we ensure that we find all the evidence out there? This has become increasingly difficult with the exponential growth of the Internet and the emergence of online publishing and open-access repositories
37 Grey Literature Matters Because….. Health technology assessment (HTA) reports are peer reviewed and highly useful to health-care decision makers and are NOT usually indexed in PubMed !Example: CADTH’s own Technology reports and overviews are not indexed in PubMed“Unpublished trials contribute about 20% of the weight in individual meta-analysis."Krishnan, R R. Evidence-based practice: how to read what you read. Psychopharmacology Bulletin 2004; 37(4):HTA reports are peer reviewed and highly useful to health-care decision makers as they seek to answer two main questions:1. Is the health care technology clinically beneficial and relatively free of harm?2 Does the cost of the technology warrant the clinical benefit?HTAs often government funded reports NOT in PubMed
38 Grey literature Matters Because….. Another way of finding articles that were missed in the original database search due to:bad or no indexinginconsistent languageincomplete database search strategyChallenging:Buried in Deep Web (cannot Google!)Copious!
39 CADTH Grey Literature Checklist “Grey Matters: A Practical Search Tool for Evidence-based Medicine”In order to use the tool:save fileopen file“Open as read only?” will appearclick “No”At CADTH the Information Services team has developed a grey literature search tool AKA checklist to ensure that nothing of importance is missed in the grey literature and to standardize our search processesWe have had many requests from researchers globally for copies of our checklist, leading to our decision to post “Grey matters: a practical search tool for evidence-based medicine” on our webpage for all to use and benefit from.This is the complete checklist searched by CADTH Technology reports, HTIS uses a small subset of the most important sites within this checklist
40 Grey Literature Searching for HTIS PROBLEM:Complete CADTH grey literature checklist used for Technology Assessments is almost 30 pages full of websites to search and can take up to a week to search comprehensively!Solution:Abridged version (7 pages - some of which is optional) of the complete CADTH checklist including Canadian and major evidence-based medicine sites is used to search lower level HTIS reports in order to save time while making sure most important grey literature is retrieved
41 Grey Literature Checklists used for HTIS Level 1 & 1.5 and 2 reportsAbridged version of complete HTA Checklist including:Canadian and major international HTA websitesCanadian and major international guideline sitesSafety information from Canadian and a couple of other major regulatory agenciesFocused Internet search
42 Grey Literature Checklists used for HTIS Level 3 & 4:The complete HTA Checklist but judgement is used as to which sites need to be searched depending on the question (i.e. a drug search will typically skip all sites dealing exclusively with devices and vice versa for a device search)The sites chosen for level 3 and 4 reports normally include all HTA sites both Canadian and International as well as safety and, if required, economic sites
43 ConclusionThe techniques used by HTIS to limit search results to reasonable numbers;limitsstudy type filtersfocused database and grey literature searchesallows us to capture most of the major evidence-based literature available in an efficient manner.These same techniques can be useful for others requiring quick and efficient ways for searching evidence-based medicine.
44 HTIS Testimonial“The information you provided is informing both our drug policy and funding people on discussions that are occurring for this entire class of drugs. I've been pleased with the report that was provided, and hope to see this service continued in the future … Very few health ministries in the country can possibly retain the 'in-house' capacity to do this type of review in this short a turnaround, so it is very good to know that CADTH had the foresight to develop this service.”(Mr. Sean Delaney BSc, MPH, Manager, Province Wide Services, Alberta Health and Wellness, Edmonton AB)