1. Writing Reviews Worth Reading
Tina Jones
Gavin Leslie
Andrea Marshall

2. The Traditional Literature review versus a Systematic Review
Dr Tina Jones
Manager, Australian Centre for Evidence Based Clinical Practice, FMC
Senior Lecturer, Flinders University

3. An Evidence Based Approach to Clinical Practice
The Evidence Based Practice (EBP) framework emerged in the early 1970s as a means of improving clinical practice
Shifted decision-making from a culture of delivering care based on tradition, intuition and authority
To a framework where clinical decisions are based on best available evidence

4. Where do nurses find the evidence?
38.7% nursing journals (Estabrooks 1998)
> 400 nursing journals listed
Access barriers
Lack of time and easy access
Lack of evidence related to nursing
Lack confidence in ability to locate, understand and properly evaluate research
Lack of organisational infrastructure support (Thompson et al 2001, Nagy et al 2001)

5. A Traditional Literature Review
Narrative in style
Written by clinical experts or individuals who have read widely on a particular topic
Provide a qualitative summary of the literature
Use an informal and subjective approach to identifying and appraising the literature

6. Davies H. & Leslie G Maintaining the CRRT circuit: non-anticoagulant alternatives, ACC; 19(4):

7. Limitations of the Traditional Review
Traditional reviews have been criticised as haphazard and biased, subject to the idiosyncratic impressions of the individual reviewer (Mulrow 1987)
Potential sources of bias
Lack of explicit inclusion or exclusion criteria for the studies cited
Reviewer bias towards particular views/practices
No mention of methods used to assess studies, or their variable quality
Lack rigor in both synthesis of findings and data extraction

8. Limitations of the Traditional Review
Mulrow (1987) from June identified 50 reviews in 4 major medical journals
(Ann of Int Med, Arch of Int Med, JAMA, NEJM)
Criteria:
Purpose of review, search methods and sources, reasons for inclusion & exclusions, methodological assessment, explication of data inconsistencies, integration of findings, summary of pertinent findings, directives for new research
‘Current medical reviews do not routinely use (such) scientific methods to identify, assess, and synthesise information’ (Mulrow 1987)
The more authoritative the expert writing the review, the lower the quality of the review (Oxman & Guyatt 1995)

9. Systematic Reviews
A review of the relevant literature on a focused clinical question that employs explicit methods to minimise bias in:
the conduct of the literature search
the appraisal of the individual studies
the methods for pooling and summarising data

10. Gardner A et al 2005 Best practice in stabilisation of oral endotracheal tubes: a systematic review ACC; 18(4):

11. Purpose of Systematic Reviews
“The purpose of a systematic literature review is to evaluate and interpret all available research evidence relevant to a particular question”
(NHMRC, 2000)

12. Why do we need Systematic Reviews?
Invaluable to health care providers, researchers and policy makers
Inundated with unmanageable amounts of information
Powerful means of integrating the best research evidence
Rational basis for health care decisions
Potential to improve healthcare outcomes

13. Why do we need Systematic Reviews?
To avoid the personal bias of an author
‘How to live longer and feel better’ – Linus Pauling (1986)
large doses of vitamin C prevent colds
Systematic review showed Vitamin C in doses as high as 1 g/day for several winter months, had no consistent beneficial effect on incidence of the common cold (Douglas et al 2003)

14. Why Systematic Reviews Matter…

15. Why Systematic Reviews Matter…

16. Why Systematic Reviews Matter…
Allow data to be ‘pooled’ and summarised, increasing statistical ‘power’
A meta-analysis is a statistical technique for combining the findings of two or more studies in order to provide a single quantitative estimate of the overall treatment effect

17. The Power of Systematic Reviews
1972 the first RCT of prenatal corticosteroids to women at risk of preterm birth
Within 10 years, 7 more trials had been completed, all showing benefit
Because no-one had ‘collated’ these findings until 1989, there was limited use of prenatal corticosteroids until early 1990s

19. The Power of Meta-analysis

20. NHMRC Levels of Evidence
Level I Systematic Review of Level II studies
Level II Randomised Controlled Trial
Level III-1 Pseudo Randomised Controlled Trial
Level III-2 Comparative Study with concurrent controls
Level III-3 Comparative Study without concurrent controls
Level IV Case Series
(NHMRC, 1999)

21. Assoc Prof Gavin D Leslie Editor - Australian Critical Care
Who conducts systematic reviews & where to find them? Basic steps in systematic review process – question identification, literature review, and compilation of data.
Assoc Prof Gavin D Leslie
Editor - Australian Critical Care

22. Who conducts systematic reviews & where to find them?
Anyone can conduct a review
Numerous EBP “centres” specialising in reviews - e.g.
Cochrane Collaboration (medical)
Joanna Briggs institute (nursing, midwifery)
Pedro (physiotherapy)
Remember - not all reviews are the same!

24. Cochrane Collaboration http://www.cochrane.org.au/
Best known (est 1996) international collaboration
Over 2000 reviews available
Predominantly based on RCT
Medically orientated
Strict criteria and process

26. Joanna Briggs Institute http://www.joannabriggs.edu.au/about/home.php
Australian based international group
Nursing & midwifery focussed
different criteria and approach to Cochrane, although principles are the same
Well known for “Practice Summaries”

28. Where to find them? Numerous search routes (web based)-
original site - limited to own reviews
Specialised search facilities e.g.
TRIP
Generic search facilities e.g.
OVID
Your librarian!

29. Where to find them? EBP “promoters” NICS ACEBP
ACEBP
scharr - netting the evidence

30. Basic steps in systematic review process – question identification.
A clearly identified question - PICO
Patient
Intervention
Comparison
Outcome

31. Framing the Question Question
What is the preferred skin site preparation for venous cannulation in adult patients?

33. Basic steps in systematic review process – literature review
Search Strategy
Use a thorough search strategy
Describe your search strategy, list key words
List databases accessed, reference lists, other lit sources, language
Eliminate inclusion bias
Devise a selection process prospectively with inclusion and exclusion criteria
Study designs, levels of evidence (RCTs etc)

35. Basic steps in systematic review process –literature review
Process of Critical Appraisal
Data Extraction & Synthesis. Errors minimised (use checklists), tools identified (objective criteria or measures), use 2 independent reviewers (and a third arbiter)
Provide details of all studies reviewed, both included and not included
Important in really deciding the integrity of the review.
Provided in full review - not usually in “article” size version.

36. Basic steps in systematic review process – compilation of data
Evaluation of studies cumulatively using statistical assessment.
Forrest plots, risk ratios, CIs

37. New developments in systematic reviews
Andrea Marshall
Sesqui Senior Lecturer Critical Care
The University of Sydney
Associate Editor, Australian Critical Care

38. Advantages of systematic reviews
Limit bias
Conclusions more reliable and accurate
Assimilate large amounts of information
Formally compared studies
generalisability of findings and consistency of results
Identify inconsistency in results and generate new hypotheses
Increase the precision of the overall result

39. Limitations of Systematic Reviews
Meta-analyses are often used to recover something from poorly designed studies, studies of insignificant statistical power, studies that give erratic results and those resulting in apparent contradictions.
If a medical treatment has an effect so recondite and obscure as to require meta-analysis to establish it, I would not be happy to have it used on me. It would seem better to improve treatment, and the theory underlying the treatment".
HJ Eysenck 1995
Systematic Reviews
Chalmers I, Altman DG
BMJ Publishing Group, London. p73

40. Limitations of systematic reviews
Combines studies of varying quality, from the excellent to the appalling
May be inconsistent with high quality RCTs
Important distinctions between primary studies may be lost (inclusion/exclusion criteria or intervention)
Dissimilar conclusions depending the "review question"
Over-generalisation may make it difficult for practitioners to apply the results

41. Bias in systematic reviews
Publication bias
Negative studies tend not to get published
Data from one study can be available in multiple formats
Selection bias
inclusion criteria
Language bias
predominantly English language studies included

42. When systematic reviews disagree with each other
Aim
Methods
Judgement of quality of studies
Summing up evidence
Systematic reviews include an element of judgement, whatever method is used

43. Quality Of Reporting Of Meta-analyses (QOROM)
Encourages authors to provide readers with information regarding
searches
selection
validity assessment
data abstraction
study characteristics
quantitative data synthesis
trial flow

45. Qualitative research & SRs
Cochrane Qualitative Research Methods Group
Aims to provide guidance on methodological standards
Aims to publish a protocol for conducting SR of qualitative evidence
There is a place for qualitative research in reviews of evidence, but how can it be incorporated?

46. Qualitative Systematic Reviews
Searching the qualitative literature
Appraising the qualitative literature
Synthesizing the qualitative evidence
Linking with existing quantitative evidence

47. Appraising qualitative research evidence
Should we?
How is it done?
Criteria – or is this imposing a positivist framework?
Ultimately a means of determining the quality of qualitative research is needed

48. Quality criteria Many criteria but little agreement
Some criteria enforce values of the positivist paradigm (Kappa statistics in coding) eg. Engel & Kuzel, 1992; Strauss & Corbin, 1998)
Is this meaningful? (Morse, 1994; Yardley, 2000)
What grounds are the criteria based?
Quality criteria
100s of quality criteria exist but there is huge variation within them (Dixon-Woods et al, 2004; NHS CRD, 2001)
Seale & Silverman (1997) emphasise detailed transcription, support of data with counts of events (quasi-statistics) & computer software
But Popay et al (1998) prioritise subjectivity, flexibility & adequate description arguing that computer software & quasi-statistics are neither necessary not sufficient for rigorous qualitative research
Some criteria argue qualitative research should aim to be reproducible & multiple coding is a good way of assessing its quality (Engel & Kuzel, 1992; Strauss & Corbin, 1998) While others deem such criteria meaningless (Morse, 1994; Yardley, 2000) On what grounds are these criteria based?

49. Quality
Philosophical underpinnings impact on criteria (some are incompatible)
Procedural aspects should be considered

50. Good quality Clear reporting of methods Transparency Sampling
Data collection
Analysis
Transparency
Adequate presentation of the data
Indicators of good quality based on our findings
Clear reporting of methods - sampling, data collection & analysis - transparency Adequate presentation of data – grounded in the data In-depth analysis of data - interpretative
Clarity throughout: integration of research questions, methods, data, results & conclusions drawn – trustworthiness

51. Good quality Analysis Trustworthiness Grounded in the data
Integration of research question, methods, data, results, and conclusions

52. Further work to develop SRs of Qualitative studies
Need to consider diverse study designs
Theoretical perspectives are hugely influential
Identify fatal flaws
Further development needed:
to devise criteria that recognise diversity of study designs & theoretical perspective in qualitative research
to distinguish between fatal flaws & minor errors
to enable the application of high quality qualitative research in teaching & practice

54. Cochrane or cock-eyed? how should we conduct systematic reviews of qualitative research? Andrew Booth Senior Lecturer in Evidence Based Healthcare Information, School of Health and Paper presented at the Qualitative Evidence-based Practice Conference, Taking a Critical Stance. Coventry University, May
Abstract
The quantitative versus qualitative debate has taken significant steps towards reconciliation within the wider field of evidence based practice. Nevertheless, far more insidious discrimination remains. Systematic review methodology exhibits all the characteristics of "institutionalised quantitativism" in that criteria for a "good" review are almost entirely determined by the quantitative methods promoted and perpetuated by the Cochrane Collaboration. Nobody who understands qualitative research would insist that its primary studies demonstrate alien concepts such as "sample size" or "statistical power". Yet comparably fundamental absurdities persist with regard to qualitative syntheses. Why should systematic reviewers of qualitative research pursue a "gold standard" comprehensive literature search when concepts such as "data saturation" have an established pedigree? Why should they apologise for an absence of meta-analysis when little-known techniques such as meta-ethnography could be included in a reviewer's toolbox? Why shouldn't they apply systematic, explicit and reproducible principles of thematic or concept analysis to create syntheses that advance our understanding of qualitative issues and highlight research gaps? The author draws on experience of a dozen systematic reviews, a third qualitative, to suggest how systematic reviews of qualitative research might acquire a methodology that is more sympathetic to the paradigm within which they are conducted.

55. Synthesised