1. Systematic reviews of health promotion and public health interventions
Rebecca Armstrong
Elizabeth Waters
Cochrane Health Promotion & Public Health Field

2. Overview Overview of systematic reviews
Outline of The Cochrane Collaboration
Role of the HPPH Field
Function of systematic reviews in informing policy and practice
Key elements of systematic reviews
Asking answerable questions
Searching for evidence
Assessing quality
Synthesising results
Applicability and transferability

3. Types of reviews Reviews Systematic reviews Meta-analysis
(narrative/literature/
traditional)
Systematic reviews
Meta-analysis

4. Narrative reviews Usually written by experts in the field
Use informal and subjective methods to collect and interpret information
Usually narrative summaries of the evidence
Do not confuse with narrative systematic reviews, which describes the type of synthesis of data.
Read: Klassen et al. Guides for Reading and Interpreting Systematic Reviews. Arch Pediatr Adolesc Med 1998;152:

5. What is a systematic review?
A review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyse data from the studies that are included in the review*
*Undertaking Systematic Reviews of Research on Effectiveness. CRD’s Guidance for those Carrying Out or Commissioning Reviews. CRD Report Number 4 (2nd Edition). NHS Centre for Reviews and Dissemination, University of York. March 2001.

6. Key elements of a systematic review
Structured, systematic process involving several steps :
Formulate the question
Plan the review
Comprehensive search
Unbiased selection and abstraction process
Critical appraisal of data
Synthesis of data (may include meta-analysis)
Interpretation of results
All steps described explicitly in the review

7. Systematic vs. Narrative reviews
Scientific approach to a review article
Criteria determined at outset
Comprehensive search for relevant articles
Explicit methods of appraisal and synthesis
Meta-analysis may be used to combine data
Depend on authors’ inclination (bias)
Author gets to pick any criteria
Search any databases
Methods not usually specified
Vote count or narrative summary
Can’t replicate review

8. Advantages of systematic reviews
Reduce bias
Replicability
Resolve controversy between conflicting studies
Identify gaps in current research
Provide reliable basis for decision making
But there can also be conflicting systematic reviews due to different methodologies used.

9. Limitations of systematic reviews specific to health promotion
Results may still be inconclusive
There may be no trials/evidence
The trials may be of poor quality
The intervention may be too complex to be tested by a trial
Practice does not change just because you have the evidence of effect/effectiveness

10. Consider these interventions…
Interventions to promote smoke alarm ownership and function
School-based driver education for the prevention of traffic crashes
Helmets for preventing head and facial injuries in bicyclists
Do you think the results identified in SRs will be good, promising or absent (and potentially harmful)?

11. Results from systematic reviews
Helmets reduce bicycle-related head and facial injuries for bicyclists of all ages involved in all types of crashes including those involving motor vehicles.
The results provide no evidence that drive education reduces road crash involvement, and suggest that it may lead to a modest but potentially important increase in the proportion of teenagers involved in traffic crashes.
Results from this review suggest that area-wide traffic calming in towns and cities may be a promising intervention for reducing the number of road traffic injuries and deaths. However, further rigorous evaluations of this intervention are needed.

12. The Cochrane Collaboration
International non-profit organisation that prepares, maintains, and disseminates systematic up-to-date reviews of health care interventions
The Cochrane Collaboration was established in October 1992.
The Collaboration relies on funding from a number of different sources to complete its work.
I like to think of a review as starting out as a jigsaw puzzle – especially if there is heterogeneity – ie. differences in interventions, settings, study designs, outcomes.

13. Cochrane Collaboration
Named in honour of Archie Cochrane, a British researcher
In 1979:
“It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials”

14. The Cochrane Library
Cochrane Systematic reviews : Cochrane reviews and protocols
Database of Reviews of Effects: Other systematic reviews appraised by the Centre for Reviews and Dissemination.
Cochrane Central Register of Controlled Trials:
Bibliography of controlled trials (some not indexed in MEDLINE).
Health Technology Assessment Database: HTA reports
NHS Economic evaluation database:
Economic evaluations of health care interventions.
DATABASES:
Cochrane Systematic reviews : Cochrane reviews
Database of Reviews of Effectiveness
Other systematic reviews from around the world which have been appraised by the NHS Centre for Reviews and Dissemination.
Cochrane Central Register of Controlled Trials
Bibliography of controlled trials (some not indexed in MEDLINE).
Cochrane database of Methodology Reviews
Cochrane reviews of methodological studies.
The Cochrane Methodology register
Bibliography of studies relating to methodological aspects of research synthesis
About the Cochrane Collaboration: Information about review groups, Fields, Centres, etc. Contact details provided.
Health Technology Assessment Database:
Health Technology Assessment reports from around the world.
NHS Economic evaluation database:
Economic evaluations of health care interventions.

15. The Cochrane Library

17. Cochrane HPPH Field
Represent the needs and interests of those in health promotion and public health in Cochrane matters
Represent Cochrane in health promotion and public health forums
In transition from Field to Review Group
Will edit PH reviews for the Cochrane Library

18. Cochrane Collaboration Structure
Steering Group
Review Groups
Centres
Fields
Consumer Network
Methods Groups

19. Collaborative Review Groups (50)
Produce systematic reviews relevant to a particular disease or health issue for inclusion in the Cochrane Library
Examples
Airways Group
Drug and Alcohol Group
Heart Group
Injuries Group
Skin Group
Pregnancy and Childbirth Group
Stroke Group
Breast Cancer Group

20. Methods Groups (12)
Provide advice and support in the development of the methods of systematic reviews
Examples
Non-Randomised Studies
Screening and Diagnostic Tests
Empirical Methodological Studies
Qualitative Methods

21. Cochrane Centres (14)
Work to assist all Cochrane entities within a specific geographical area
Examples
Australasian Cochrane Centre (at Monash)
South African Cochrane Centre
Italian Cochrane Centre
Chinese Cochrane Centre

22. Cochrane Fields/Networks (9)
Represent an area of interest which spans a number of health problems - and hence a number of Review Groups
Examples
Health Promotion and Public Health Field
Primary Health Care Field
Cancer Network
Child Health Field

23. Cochrane HPPH Field
Cochrane Fields represent a population group, or type of care that overlaps multiple Review Group areas
HPPH Field
Registered in 1996
Administered from Melbourne
Funded by VicHealth
Over 400 members on contact database across >30 countries

24. Staff Elizabeth Waters (Director) Jodie Doyle (Coordinator)
Rebecca Armstrong (Senior Research Fellow)
Naomi Priest (Research Fellow)

25. Asking an answerable question

26. Questions of interest Effectiveness:
Does the intervention work/not work?
Who does it work/not work for?
Other important questions:
How does the intervention work?
Is the intervention appropriate?
Is the intervention feasible?
Is the intervention and comparison relevant?

27. Answerable questions EFFECTIVENESS A description of the populations P
An identified intervention I
An explicit comparison C
Relevant outcomes O

28. A PICO question Time-consuming question:
What is the best strategy to prevent smoking in young people?

29. An answerable question
Q. Are mass media (or school-based or community-based) interventions effective in preventing smoking in young people?

30. The PICO(T) chart Problem, population Intervention Comparison Outcome
Types of studies
Young people under 25 years of age
a) Television
b) Radio
c) Newspapers
d) Bill boards
e) Posters
f) Leaflets
g) Booklets
a) School-based interventions
b) No intervention
a) objective measures of smoking (saliva thiocyanate levels, alveolar CO)
b) self-reported smoking behaviour
c) Intermediate measures (intentions, attitude, knowledge, skills)
d) Media reach
a) RCT
b) Controlled before and after studies
c) Time series designs

32. Finding the evidence

33. Systematic review process
Well formulated question
Comprehensive data search
Unbiased selection and abstraction process
Critical appraisal of data
Synthesis of data
Interpretation of results

34. A good search Clear research question Comprehensive search
All domains, no language restriction, unpublished and published literature, up-to-date
Document the search (replicability)
Quality of evidence generated by a review depends entirely on the primary studies which make up the review.

35. Components of electronic searching
Describe each PICO component
Start with primary concept
Find synonyms
Identify MeSH / descriptors / subject headings
Add textwords
Add other components of PICO question to narrow citations (may use study filter)
Examine abstracts
Use search strategy in other databases (may need adapting)

36. So you want to do a ‘quick & dirty’?
DARE
CENTRAL
PubMed (clinical queries, related records)
CDC
NICE
Organisations who do work in your area
…google

37. The Cochrane Library: www.thecochranelibrary.com

38. Cochrane HPPH Field

39. Health-evidence.ca

40. The Guide to Community Preventive Services

41. National Institute for Health and Clinical Excellence

42. Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre)

43. Effective Public Health Practice Project (EPHPP)

44. Centre for Reviews and Dissemination

45. Other sources of primary research

46. Searching www.google.com.au
The order of terms will effect the results so start with the obvious or key concept
No need for ‘and’
Google will ignore common words
If they are important use + (e.g. policy + 3)
Phrase searching is useful eg “suicide prevention”
Google searches for variations on words eg diet, dietary

47. Searching
Where terms have multiple meanings you can direct google to remove sites you want to avoid (e.g. bass –music)
Keep your search strings brief
Mental health promotion initiatives to prevent suicide in young people
Compartmentalise your search strings
“mental health promotion” suicide
“Suicide prevention” “young people”
Prevention and suicide and youth

48. Searching http://www.ncbi.nlm.nih.gov/entrez/query.fcgi
Use the same principles for google – keep it short and sweet.
Key features
Journals Database MeSH Database Single Citation Matcher Clinical Queries

49. Searching

50. Searching

51. Searching
Select the Limits tab – just under the search string

52. Searching

53. These next few slides show you how to search MeSH terms in PubMed
These next few slides show you how to search MeSH terms in PubMed. Useful if you don’t have access to electronic databases. You combine this method with the one for text words outlined above. The process for combining text words and MeSH terms is outlines below.

57. Find MeSH and textwords to
Example
Mass media interventions to prevent smoking in young people
P= Young people
STEP ONE:
Find MeSH and textwords to
describe young people

58. Example Mass media interventions to prevent smoking in young people
P= Young people
MeSH: Adolescent
Child
Minors

59. Example Mass media interventions to prevent smoking in young people
P= Young people
Textwords:
Adolescent Girl
Child Boy
Juvenile Teenager
Young people Young adult
Student Youth

60. Textwords Truncation $: To pick up various forms of a word
Teen$.tw Smok$.tw Teenage Smoke
Teenager Smoking
Teenagers Smokes
Teens Smoker
Teen Smokers

61. Textwords Wild cards ? and #: To pick up different spellings
Colo?r.tw (? Can be substituted for one or no characters)
Colour
Color
Wom#n.tw (# Substitutes for one character)
Woman
Women

62. Textwords Adjacent ADJn:
retrieves two or more query terms within n words of each other, and in any order
Great when you are not sure of phraseology
Eg sport adj1 policy
Sport policy
Policy for sport
Eg mental adj2 health
Mental health
Mental and physical health

63. Example continued
Mass media interventions to prevent smoking in young people
I = Mass media interventions
STEP TWO:
Find MeSH and textwords to
describe mass media interventions

64. Example continued MeSH Mass media Audiovisual aids Television
Motion pictures
Radio
Telecommunications
Newspapers
Videotape recording
Advertising

65. Example continued
Mass media interventions to prevent smoking in young people
O = Prevention of smoking
STEP THREE:
Find MeSH and textwords to
describe prevention of smoking

66. Example of search OR OR OR P AND I C O O = PREVENTION OF SMOKING
P = YOUNG PEOPLE
MeSH Textwords
………………………. ………………………. ………………………. ……………………….
………………………. ……………………….
I = MASS MEDIA
C = (if required)
O = PREVENTION OF SMOKING P
AND I C O OR OR OR

67. Different bibliographic databases
Databases use different types of controlled vocabulary
Same citations indexed differently on different databases
Medline and EMBASE use a different indexing system for study type
PsycINFO and ERIC do not have specific terms to identify study types
Need to develop search strategy for each database

68. Study design filters RCTs Non-RCTs Qualitative research
See Cochrane Reviewer’s Handbook
Non-RCTs
Not yet developed, research in progress
Qualitative research
Specific subject headings used in CINAHL, ‘qualitative research’ used in Medline
CINAHL Filter: Edward Miner Library
Systematic reviews/meta-analyses
CINAHL: as above
Medline
Medline and Embase
PubMed

69. 2. Unpublished literature
Not all known published trials are identifiable in Medline (depending on topic)
Only 25% of all medical journals in Medline
Non-English language articles are under-represented in Medline (and developing countries)
Publication bias – tendency for investigators to submit manuscripts and of editors to accept them, based on strength and direction of results (Olsen 2001)

70. 2. Unpublished literature
Hand searching of key journals and conference proceedings
Scanning bibliographies/reference lists of primary studies and reviews
Contacting individuals/agencies/ academic institutions
Neglecting certain sources may result in reviews being biased
Handsearching: Not all journals are indexed in databases
May be indexed incorrectly
Knowing when to stop: depends on the return you are getting, resources, time constraints
Use Reference system for collecting references i.e. Endnote, Procite, IdeaList, Reference Manager

71. Librarians are your friends!

72. Principles of critical appraisal

73. Critical appraisal
The process of systematically examining research evidence to assess its validity, results and relevance before using it to inform a decision.
Alison Hill, Critical Appraisal Skills Programme, Institute of Health
Sciences, Oxford

74. Critical appraisal I: Quantitative studies

75. Why appraise validity?
Not all published and unpublished literature is of satisfactory methodological rigour
Just because it is in a journal does not mean it is sound!
Onus is on you to assess validity!
Quality may be used as an explanation for differences in study results
Guide the interpretation of findings and aid in determining the strength of inferences

76. Bias – quality assessment tool
Selection bias
Allocation bias
Confounding
Blinding (detection bias)
Data collection methods
Withdrawals and drop-outs
Statistical analysis
Intervention integrity

77. Allocation of concealment
Selection bias
Recruit participants
Allocation of concealment
Intervention
Allocation
Control
Confounding
Exposed to intervention
Not exposed to intervention
Integrity of intervention
Intention-to-treat
Withdrawals
Follow-up
Follow-up
Blinding of outcome assessment
Outcome
Outcome
Data collection methods
Analysis
Analysis
Statistical analysis

78. Critical appraisal tools
RCTs
The Quality Assessment Tool for Quantitative Studies (
Non-RCTs
Cochrane Effective Practice and Organisation of Care Group (
The Berkeley Systematic Reviews Group (

79. Critical appraisal II: Qualitative studies

80. Qualitative research
… explores the subjective world. It attempts to understand why people behave the way they do and what meaning experiences have for people.
Qualitative studies of experience
Process evaluation
Subjective experience of an intervention, or of an illness, condition, etc.
Undertaking Systematic Reviews of Research on Effectiveness. CRD’s Guidance for those Carrying Out or Commissioning Reviews. CRD Report Number 4 (2nd Edition). NHS Centre for Reviews and Dissemination, University of York. March 2001.

81. CASP appraisal checklist
Clear aims of research (goals, why it is important, relevance)
Appropriate methodology (what, how, why)
Sampling strategy
Data collection
Relationship between researcher and participants
Ethical issues
Data analysis
Findings
Value of research (context dependent)

82. Other qualitative checklist
Quality framework
Government Chief Social Researcher’s Office, UK
19 question checklist for process evaluations (EPPI-Centre)

83. Appraisal of a systematic review
10 questions
Clearly-focused question
The right type of study included
Identifying all relevant studies
Assessment of quality of studies
Reasonable to combine studies
What were the results
Preciseness of results
Application of results to local population
Consideration of all outcomes
Policy or practice change as a result of evidence
CASP

84. Interpretation of results

85. Factors influencing effectiveness
Type III error (integrity of intervention)
Theoretical framework of intervention
Context
For whom did the intervention work, why, in what circumstances, at what cost
See also the Public Health Schema by Rychetnik and Frommer, 2002 for formulating summary statements.

86. Difficulties addressing inequalities
Studies rarely present information on differential effects of interventions
Cannot locate studies addressing inequalities
May need original data from authors
Low power to detect subgroup differences
Almost impossible to determine whether studies report data by SES from literature searches. Useful to have a broad strategy. Information will come from a variety of sectors including social, education and legal as well as health.
Low power – many primary studies are small, and stratified analyses by SES will therefore only include a small number in each group. The power to detect differences will be low.

87. Integration of evidence into practice

88. Assessing the applicability and transferability of interventions
Applicability – whether the intervention process could be implemented in the local setting, no matter what the outcome is.
Is it possible to run this intervention in this local setting?
Eg. provision of condoms in area where they are not acceptable for religious reasons
Wang et al 2005

89. Assessing the applicability and transferability of interventions
Transferability – if the intervention were to be implemented in the local setting, would the effectiveness of the program be similar to the level detected in the study setting?
E.g. if the interventionists lack experience and have few skills in delivering the intervention then its effectiveness in the local setting may be lower than that demonstrated in the study setting
Wang et al 2005

90. Review of questions proposed by Wang et al

91. Contact details Rebecca Armstrong rarmstrong@vichealth.vic.gov.au
If I can’t help you I might be able to point you in the right direction.
If you are interested in training and support for conducting SRs or increasing uptake within your organisation, region, state please let me know