1. Critical Appraisal: An Introduction
Melanie Browne HBHSc, MLIS
Information Specialist

3. Agenda Introduction Define Evidence-based clinical practice
How to read a research article
Search for best evidence
Apply critical appraisal in your library
Our new role
Case presentation
Resources

4. Evidence-Based Medicine
"...the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.“
from Sackett, DL, et al. "Evidence based medicine: What it is and what it isn't." (BMJ 1996; 312: 71-2)
EBHC is a deceptively simple concept but the devil is in the detail. EBHC is an expansion of the concept developed for medicine in the early 1980s. It means integrating individual clinical expertise with the best available external; clinical evidence from systematic research". This approach has migrated from clinical medicine to other health care disciplines variously being defined as "evidence based healthcare" (Gray 1997) and "evidence based practice" (McKibbon 1998).

5. "Evidence based medicine (EBM) is an approach to health care that promotes the collection, interpretation, and integration of valid, important and applicable patient-reported, clinician-observed and research-derived evidence. The best available evidence, moderated by patient circumstances and preferences, is applied to improve the quality of clinical judgements" (McKibbon et al 1995).
Ann McKibbon , a health sciences librarian from McMaster University in Ontario who redefined this approach to encompass the patient's perspective.
McKibbon, KA, Wilczynski, N, Hayward, RS, Walker Dilks, CK & Haynes, RB (1995) The medical literature as a resource for health care practice. Journal of the American Society of Information Science 46(1)

6. Why EBM?
A clinician needs to read 17 peer reviewed articles per day, every day of the year, to stay current (Haynes 1993).
Haynes, R. (1993) Where's the meat in clinical journals?
ACP Journal Club, 119: A23-4.
Simple, is it not. Just apply the research evidence in the context of your professional judgement and your patient's choices. What happens though when there are 20 million pieces of 'evidence' of varying quality and sometime of contradictory conclusions. Your average consultation time is 7-10 minutes. Practically, how can you access, distil and apply this research? To address these issues a number of approaches have developed. One is to develop summaries of the best available research evidence and another is to train clinicians and other health care decision makers to find and appraise relevant evidence. Other approaches have been to imbed knowledge prompts in software for drug and test orders and for electronic patient records. These are generically called decision support systems. Another story entirely.
There are 20 million pieces of 'evidence' of varying quality and sometime of contradictory conclusions.

7. How Can You Access, Distil and Apply Research/Evidence?
Develop summaries
Train clinicians and other health care decision makers to find and appraise relevant evidence
Decision support systems
To address these issues a number of approaches have been developed. One is to develop summaries of the best available research evidence and another is to train clinicians and other health care decision makers to find and appraise relevant evidence. Other approaches have been to imbed knowledge prompts in software for drug and test orders and for electronic patient records.

8. Summaries With Rigorous Methodologies
Cochrane Collaboration
National Centre for Clinical Excellence
Centre for Clinical Effectiveness at Monash University
As you know any research process requires a review of past studies to provide context. Some research consciously reviews significant past research to provide a useful summary. This type of narrative review is open to a number of sources of bias. The author's personal beliefs, the quality and availability of research papers all bias such summaries of research evidence. To overcome these limitations groups such as the Cochrane Collaboration, the National Centre for Clinical Excellence in the UK. In Australia, the Centre for Clinical Effectiveness at Monash University have developed more rigorous methodologies.
Imiquimod: A Review Imiquimod: A Review Journal of Cutaneous Medicine and Surgery: Incorporating Medical and Surgical Dermatology Volume 6, Number 6 / December, 2002 Background and Objective: Imiquimod (Aldara) is an immune response modifier used primarily to treat anogenital warts. Imiquimod induces cytokines and enhances cell-mediated cytolytic antiviral activity in vivo. It exhibits antiviral and antitumor effects; however, it does not exhibit direct antiviral effects in vitro. By inducing cytokines, such as interferon a, imiquimod stimulates both the innate immune response and the cellular arm of acquired immunity. Currently, imiquimod is approved for use in the treatment of external genital and perianal warts in adults. Conclusion: Beyond its established use, there are case reports and preliminary studies suggesting the effectiveness of imiquimod 5% cream in the treatment of nongenital human papillomavirus warts, molluscum contagiosum, actinic keratosis, basal cell carcinoma, squamous cell carcinoma, Bowen's disease, nongenital human papillomavirus infection, and vulvar intraepithelial neoplasia.

9. The Argument for Evidence-Based Medicine
Stay up to date with the current literature
Communicate effectively
Make the best use of information from the history, physical examination, and diagnostic testing
Avoid common pitfalls of clinical decision-making
Why EBM?
Practicing evidence based medicine allows clinicians to keep up with the rapidly growing body of medical literature.
Evidence based medicine improves clinicians' skills in asking answerable questions and finding the best evidence to answer these questions.
Evidence based medicine can provide a framework for critically appraising evidence.
Practicing evidence based medicine encourages clinicians to integrate valid and useful evidence with clinical expertise and each patient's unique features, and enables clinicians to apply evidence to the treatment of patients.
adapted from Straus, SE and Sackett, DL. "Getting research findings into practice: Using research findings in clinical practice." (BMJ 1998; 317: )

10. Critical Appraisal
The assessment of evidence by systematically reviewing its relevance, validity and results to specific situations. - Chambers, R. (1998).

11. What is ‘best’ evidence?
Using critical appraisal skills you can understand the methods and results of the research and then be able to assess the quality of the research.

12. Critical appraisal Exact science Critical appraisal “right” answer

13. Information Mastery Slawson and Shaughnessy Formula
Usefulness of Medical Information =
Relevance x Validity ____________________
Work to Access

14. Smith R. What clinical information do doctors need
Smith R. What clinical information do doctors need? Br Med J 1996; 313:
Summarize this study here..

15. When doctors see patients they usually generate at least one question
Most of the questions concern treatment
Many of the questions are highly complex, simultaneously asking about individual patients and particular areas of medical knowledge

16. Often doctors are asking not simply for information but for support, guidance, affirmation, and feedback
Doctors are most likely to seek answers to these questions from other doctors
The best information sources provide relevant, valid material that can be accessed quickly and with minimal effort

17. Basic elements of clinical decision making
Haynes RB. Loose connections between peer-reviewed clinical journals and clinical practice. Ann Intern Med 1990;113:724-8.

18. Why Should we Critically Appraise?
Published research is not always reliable
Published research is not always relevant
To improve clinical effectiveness, we need a systematic framework to interpret research

19. Difficulties with Critical Appraisal
Can be time consuming initially
Doesn’t provide an “easy” answer
It could show a lack of good evidence in a particular topic

20. Key Steps to Effective Critical Appraisal
1. Are the results valid?
2. What are the results?
3. How will these results be relevant to
the patient?

21. Validity and Reliability
A test is valid when it measures what it’s supposed to.
If a test is reliable, it yields consistent results.
A test can be both reliable and valid, one or the other, or neither.
Reliability is a prerequisite for measurement validity. 
How valid a test is depends on its purpose—for example, a ruler may be a valid measuring device for length, but isn’t very valid for measuring volume.

22. Reliable, but Not Valid!

23. Not Reliable, Not Valid!

24. Reliable and Valid

25. Mark Newman - Middlesex University updated 04/2001

26. What a Physician can be Faced With on a Daily Basis?
~1 - 2 questions per patient (clinics)
15/patient/day (wards)
~30% of questions are followed
People
Books
Electronic resources
40% easy to answer
30% tough to answer
30% cannot answer
Source: Dawes M, Sampson U. Knowledge management in clinical practice: a systematic review of information seeking behaviour in physicians. Int J Med Inform Aug; 71(1):9-15. Review. PMID:

27. Steps to EBM 1. Formulate a clear, focused clinical question
2. PICO model
3. Search the literature for the best external
evidence
4. Critically appraise the evidence for its validity
and usefulness
5. Implement the useful evidence in clinical
practice
6. Evaluate the results

28. Question Formulation Not easy but EXTREMELY important
Good questions will
Focus/clarify your information need
Give you some idea of where to look for information
Give you searching concepts and terms

29. Background Questions …Often broad in nature
…Often not patient-specific but fact based
…May not need to integrate knowledge
…More common early in training/new situations

30. Foreground Questions
…Often for a specific patient or clinical situation
…Narrow in focus
…Need to integrate external information with clinical/situational data

31. PICO Structure
Patients P
Intervention I
Comparison C
Outcome O

32. Clinical Scenario
What therapeutic agents can be used for rate control of atrial fibrillation (AF) in a patient with congestive heart failure (CHF)?

33. Starting Point Department: Emergency
Population: Patients with atrial fibrillation and congestive heart failure
Intervention: Rate control
Comparison: N/A
Outcome: Mortality, effectiveness of rate control

34. PICO Schematic Model Rate control Patient w/ Atrial Fibrillation in
congestive heart
failure
Effectiveness
of rate
control
Comparison
Rhythm control /
No treatment

35. General Search Strategy
Clinical problem
Define the search question
Try another relevant resource
Choose a resource/database
Create a search strategy
Create a search strategy
Summarize the evidence
Poor
yield
Summarize the evidence
Try another relevant resource
Apply the evidence
Adapted from: Sackett, D. et al Evidence-Based Medicine: How to Practice and Teach EBM. 2nd Edition. Toronto: Churchill Livingstone.

36. Study Types for Question Types
Diagnosis
Prospective cohort study with good quality validation against “Gold Standard”
Therapy
Randomized controlled clinical trial (RCT)
Etiology/
Harm
RCT, cohort or case-control study (probably retrospective)
Prognosis
Prospective cohort study

37. Does this treatment work?
systematic review, RCT
How good is a diagnostic test?
(prospective) cohort study
Should we screen?
RCT
What causes this disease?
RCT, prospective cohort study, case control study (rare diseases)
What did people think or do?
cohort study, cross-sectional survey, qualitative study

38. This is our flowchart – the decision tree (hand out the hand out)
This is our flowchart – the decision tree (hand out the hand out). Access to our resources through here. (guide for you)

39. How can we do Critical Appraisal?
Use common sense
Use simple checklists
Use different checklists depending on the different types of studies (i.e., RCTS, systematic reviews etc)
Checklists help you focus on the important parts of the article

40. Research Methodology Who were the participants of the study?
How were they recruited?
Was there bias in the recruiting methods?
How was the data collected?
What statistical tests were used?
Where the data collection methods accurate?

41. Critical appraisal questions
What is the paper about?
Why was the study done?
What type of study was done?
Was it primary research (experiment, RCT, cohort, case-control, cross-sectional, longitudinal, case report/series)?

42. Critical appraisal questions
Was it secondary research (overview, systematic review, meta-analysis, decision analysis, guidelines development, economic analysis)?
Was the design appropriate (for study on treatment, diagnosis, screening, prognosis, or causation)?

43. Critical appraisal questions
Was the study ethical?
Is the design right?
(BMJ Editor's checklists)

44. How to Read a Research Article? – First Glance.
Purpose of reading the paper.
Do not read the abstract.
Read the title, find out who the authors are and where they work, look for sources of funding and conflicts of interest
Look at the tables and figures
Is there a diagram to show the flow of participants through each stage of the study
Why was the study done and what hypothesis were the authors testing?
What is broadly the topic of research?
Therapy, Diagnosis, Screening, Prognosis, Causation?
What type of study was done?
Is this a primary (experimental, clinical trial, survey) or a secondary paper (review, meta-analysis, guideline, economic analysis)?
Was the study design appropriate?
Now proceed to a critical appraisal of the paper
1. Are you trying to answer a clinical question? Will the paper help in your clinical practice? Is it a communication from one group of researchers to another (in which case ignore it unless you are yourself engaged in this area of research)?

45. Case Presentation
Read the Ray et al(2008) paper and divide the group into 2 – for a debate on the paper “Breast size and risk of type 2 diabetes mellitus.” CMAJ

46. Ray and colleagues studied data from women in conjunction with the Nurses' Health Study II and found that breast size at age 20, assessed by recall of bra cup size, correlated positively with the incidence of type 2 diabetes

47. After adjustment for relevant factors, such as body mass index, waist circumference and family history of diabetes, the hazard ratio dropped to 1.58 but remained significant.

48. Nurses' Health Study II population data emanates from women who were mainly of white ancestry, and that their analysis is based on recall and self-report.

49. New Role of Health Sciences Librarians
Teaching access to the literature and other information resources
Teaching use of technology as a means to access and manage information
Teaching skills in information organization and critical appraisal

50. Role of Librarians as Information Clinicians
Medical Informatics Tutor
EBM Educator

51. Informationist
A discipline requiring a combination of the skills of a librarian, a clinical epidemiologist and a medical scientist
Davidoff, F. & Florance,V. (2000) The Informationist: A New Health Profession? Annals of Internal Medicine, 132:

52. Effective INFORMAtician
Knows categories of information resources
…therapy, diagnosis, prognosis
…synthesized resources vs original studies
Knows strengths/weaknesses of information resources
Knows when to use each category
Knows that some resources are better than others in certain situations
…Cochrane diagnostic accuracy data?
Knows what to do with the results

53. Practical solutions - Librarian as a partner on the teaching team; teachable moments
Formal hospital library educational programs have focused on orientation and instruction in the use of bibliographic tools. i.e.- LATCH (Literature Attached to Charts) and clinical medical librarianship
The educational programs which seem most effective are those taught at the point of need; that is, they capitalize on the "teachable moment." This teachable moment almost always occurs within the context of the curriculum. The trend in library education programs toward greater collaboration with discipline experts is a promising one. Taken to the extreme, the librarian trains the discipline expert who then conveys the library educational content to the students. This, however, is a costly solution unlikely to be adopted by many institutions. A more practical approach places the librarian as a partner on the teaching team, an effective position from which to convey the value of information services in the health sciences.

54. Critical Appraisal Skills Training
Research methodology
Statistical techniques

55. Courses for Librarians Out There
CILIP: the Chartered Institute of Library and Information Professionals
Critical appraisal skills for healthcare librarians: building on the basics
As well as designing and delivering education and training programs in both academic and clinical setting, librarians are involved in designing developing interactive web sites and software tools. Some Cochrane Review groups include librarians as research partners, while other group are merely demanding customers of their local hospital libraries. Librarians are also involved in EBHC at policy and administrative levels. Examples are the appointment of an executive Library Adviser to the U.K. National Health Services and a recent Deputy Director of the Australasian Cochrane Centre being a former public librarian.

56. General Resources:
CASP (Critical Appraisal Skills Programme) - part of the Public Health Resource Unit based at Oxford, CASP runs training workshops on critical appraisal skills.  
Evidence-Based Medicine Toolkit - hosted by the University of Alberta, this is an online "box" of handy tools to help you find, appraise, and apply in practice, evidence-based research
How to read a paper - a set of ten guides from the BMJ (individual links given in the relevant section below).
Levels of Evidence - a ranking system used to rank various study designs in order of evidence-based merit: systematic reviews/meta-analyses and well conducted randomised controlled trials (RCTs) are usually seen as the best form of "evidence", with research based on the outcome of a case series placed somewhere near the bottom
Netting the Evidence - search for the keyword "appraisal" to find a quality assessed list of appraisal resources
User's guides to evidence based practice - based on a series of articles published in JAMA, these guides give comprehensive advice on how to find, appraise and apply research in practice

57. Keeping Current Daily InfoPOEMs http://www.infopoems.com/ bmjupdates+
Tables of contents of journals
(e.g. JAMA)
From journal itself
My NCBI from PUBMED

58. Predictive Value of tests
Confidence Intervals
Relative Risk Reduction
Intention to treat
Relative Risk
P Value
Absolute Risk Difference
Sensitivity
Absolute Risk Reduction
Absolute Benefit Increase
Specificity
Likelihood Ratio
Numbers Needed to Treat (NNT)
Odds Ratio