1. Systematic Reviews and evidence based syntheses of research
Pete Smith
Department of Languages
February 2013

2. Which of these would you trust to answer your question?
Best practice is …
The latest research shows that …..
A large scale study indicates …
A high quality randomised trial by Prof Blogs supports …
Experts think that …
The overall body of research evidence indicates that …

3. Norris & Ortega
Effectiveness of L2 Instruction: A Research Synthesis and Quantitative Meta-analysis
John Norris, Lourdes Ortega, Language Learning, Volume 50, Issue 3, pages 417–528, September 2000
Independent reviews

4. Norris & Ortega
This study employed (and reports in detail) systematic procedures for research synthesis and meta-analysis to summarize findings from experimental and quasi-experimental investigations into the effectiveness of L2 instruction published between 1980 and Comparisons of average effect sizes from 49 unique sample studies reporting sufficient data indicated that focused L2 instruction results in large target-oriented gains, that explicit types of instruction are more effective than implicit types, and that Focus on Form and Focus on Forms interventions result in equivalent and large effects. Further findings suggest that the effectiveness of L2 instruction is durable and that the type of outcome measures used in individual studies likely affects the magnitude of observed instructional effectiveness. Generalizability of findings is limited because the L2 type-of-instruction domain has yet to engage in rigorous empirical operationalization and replication of its central research constructs. Changes in research practices are recommended to enhance the future accumulation of knowledge about the effectiveness of L2 instruction.

5. Doing more harm than good?
20 years of guess work in ELT
Cochrane Collaboration in 1996
Evidence Based Medicine
5000+ reviews now
Evidence based decision making for social policy – NICE, EPPI, Campbell, Cabinet Office Policy Hub
Evidence based tokenism & objections

6. The Cochrane Library

7. The Cochrane Collaboration
The inner part of the logo illustrates a systematic review of data from seven randomized controlled trials (RCTs), comparing one health care treatment with a placebo. Each horizontal line represents the results of one trial (the shorter the line, the more certain the result); and the diamond represents their combined results. The vertical line indicates the position around which the horizontal lines would cluster if the two treatments compared in the trials had similar effects; if a horizontal line touches the vertical line, it means that that particular trial found no clear difference between the treatments. The position of the diamond to the left of the vertical line indicates that the treatment studied is beneficial. Horizontal lines or a diamond to the right of the line would show that the treatment did more harm than good.
This diagram shows the results of a systematic review of RCTs of a short, inexpensive course of a corticosteroid given to women about to give birth too early. The first of these RCTs was reported in The diagram summarises the evidence that would have been revealed had the available RCTs been reviewed systematically. A decade later it indicates strongly that corticosteroids reduce the risk of babies dying from the complications of immaturity. By 1991, seven more trials had been reported, and the picture had become still stronger. This treatment reduces the odds of the babies of these women dying from the complications of immaturity by 30 to 50 per cent.
Because no systematic review of these trials had been published until 1989, most obstetricians had not realised that the treatment was so effective. As a result, tens of thousands of premature babies have probably suffered and died unnecessarily (and needed more expensive treatment than was necessary). This is just one of many examples of the human costs resulting from failure to perform systematic, up-to-date reviews of RCTs of health care.

8. The Cochrane Library Is aspirin a good treatment for migraines??

9. Does praying for someone ill make them better ?
How would you find out?
Would a good randomised controlled trial answer the question?
Would a peer-refereed article by an expert in a high-impact, renowned journal provide a reliable answer?

10. Why a systematic review might answer the Question
Sums up the best available research on a specific question.
Synthesizes the results of several studies.
Transparent procedures to find, evaluate and synthesize the results of relevant research.
Studies included in a review are screened for quality
What are the vital characteristics of a systematic review ?
The purpose of a systematic review is to sum up the best available research on a specific question. This is done by synthesizing the results of several studies. A systematic review uses transparent procedures to find, evaluate and synthesize the results of relevant research. Procedures are explicitly defined in advance, in order to ensure that the exercise is transparent and can be replicated. This practice is also designed to minimize bias. Studies included in a review are screened for quality, so that the findings of a large number of studies can be combined. Peer review is a key part of the process; qualified independent researchers control the author's methods and results.

11. Cochrane reviews
The Cochrane Handbook outlines eight general steps for preparing a systematic review:
Defining the review question and developing criteria for including studies
Searching for studies
Selecting studies and collecting data
Assessing risk of bias in included studies
Analysing data and undertaking meta-analyses
Addressing reporting biases
Presenting results and "summary of findings" tables
Interpreting results and drawing conclusions

12. Campbell reviews Clear inclusion & exclusion criteria
An explicit search strategy
Systematic coding and analysis of included studies
Meta-analysis (where possible)
a systematic search for unpublished reports (to avoid publication bias).
international scope.
A protocol (project plan) for the review is developed in advance and undergoes peer review.
Study inclusion and coding decisions are accomplished by at least two reviewers who work independently and compare results.
peer review and editorial review.

13. EPPI
Evidence-informed policy and practice - basing policy and practice on sound evidence
Interest in systematic reviews and evidence-informed education and health promotion is part of a general move in the UK and elsewhere towards basing policy and professional practice on sound evidence. Systems, such as the Cochrane Collaboration and the Campbell Collaboration, have been set up to help professionals, policy makers and users base their decisions on up-to-date and reliable evidence by making the results of systematic reviews accessible. The two collaborations develop systems and methods in health care interventions (Cochrane Collaboration) and education and social care, employment and crime and justice (Campbell Collaboration).
Look at EPPI website

14. Controlling for bias Transparency & replication (protocol)
Literature searches (publication bias)
Types of studies (selection bias)
Sample selection (robustness)
Meta-analysis where appropriate
Limitations

15. Strategy training In 2004 we had a question
Does Strategy Training in language learning work?
Describe the Strategy training review

16. Components Protocol – prior to looking at data Review – in depth
Update – evolution of body of evidence & dialogue

17. Protocol Title Question Background Objectives
Initial scope of the review
Criteria for considering studies for this review
Types of studies
Mapping
In-depth review
Types of participants
Types of interventions
Types of outcome measures
Search strategy for identification of studies
Methods of the review

18. Review BACKGROUND Aims and rationale for current review
Definitional and conceptual issues
Policy and practice background
Research background
Authors, funders and other users of the review
Review questions
METHODS USED IN THE REVIEW
User-involvement
Identifying and describing studies
In-depth review
IDENTIFYING AND DESCRIBING STUDIES:
Studies included from searching and screening
Characteristics of the included studies
Identifying and describing studies: quality-assurance results
IN-DEPTH REVIEW: RESULTS
Selecting studies for the in-depth review
Nature of involvement of users in the review and its impact
(meta analysis or other synthesis)
FINDINGS AND IMPLICATIONS
Strengths and limitations of this systematic review
Implications
User perspectives

19. Your review….. What’s your question?
Outline the protocol for a systematic review