1. Assessing study quality for a systematic review Trudy Bekkering, PhD Center of Evidence-Based Medicine & Belgian Branch of the Dutch Cochrane Center Centre for Methodology of Educational Research Katholieke Universiteit Leuven, Belgium

2. Why is it so important? Meta-analysis aims to increase precision Meta-analysis of studies with bias in results gives very precise but wrong results Garbage in, garbage out

3. Bias versus imprecision Bias: A systematic error in the results or the inferences Methodological flaw Overestimation or underestimation

4. Bias versus imprecision BIAS Ideal study

5. Bias versus imprecision Imprecision: A random error in the results Sample variation Direction of error is random

6. Bias versus imprecision Ideal study IMPRECISION

7. Bias versus imprecision BIAS + IMPRECISION

8. Bias versus imprecision BIAS ? IMPRECISION?

9. Risk of bias versus bias Clear empirical evidence that particular flaws in study design can lead to bias. Usually impossible to know to what extent biases have affected the results. Key consideration = should the results be believed

10. Tools for assessing study quality Scales: discouraged Checklist: between 3 and 57 items Cochrane tool: “domain based” Depends on study design

11. Tool for RCTs: Cochrane tool Risk of bias on 6 domaines: 1.Sequence generation 2.Allocation concealment 3.Blinding 4.Incomplete outcome data 5.Selective reporting 6.Other sources of bias

12. Risk of which biases? Selection bias Differences between baseline characteristics of the groups compared Performance bias Differences in the care that is provided, or in exposure to other factors than the intervention Attrition bias Differences in withdrawals from a study Detection bias Differences in how outcomes are determined Reporting bias Differences between reported and unreported outcomes

13. How do you assess?

14. DomaineDescriptionJudgement Sequence generation QUOTE: “patients were randomly allocated” COMMENT: probably done, since earlier reports of this study describe use of random sequences YES (low risk of bias) Allocation concealment YES (low risk) NO (high risk) UNCLEAR (uncertain) Blinding YES NO UNCLEAR Incomplete outcome data YES NO UNCLEAR Selective outcome reporting YES NO UNCLEAR Other sources of bias YES NO UNCLEAR

15. AdequateNot adequateUnclear Random number table Computer generated list Coin tossing Shuffling cards / envelopes minimization Generated by: Odd/even date of birth Date/day of admission Hospital record number Allocation by: Clinical judgement Participant’s preference Result of lab test Availability of intervention If insufficient information about sequence generation! Sequence generation

16. Allocation concealment AdequateNot adequateUnclear Central randomisation (telephone, web-based, pharmacy) Sequentially numbered drug containers of identical appearance Sequentially numbered, opaque, sealed envelopes Open random allocation schedule Envelopes without appropriate safeguards Quasi-randomisation If insufficient information about sequence generation! (e.g. “sealed envelopes”)

17. Blinding of intervention Participants (patients, clients) Care providers (doctors, nurses, teachers …) Outcome assessors

18. Blinding of intervention AdequateNot adequateUnclear No blinding, outcome (assessment) not likely to be influenced by lack of blinding Blinding ensured and unlikely to have been broken Either participants or some personnel unblinded, but unlikely to introduce bias + outcome assessment blinded No or incomplete blinding and outcome (assessment) likely to be influenced by lack of blinding Blinding attempted but likely that could have been broken insufficient information Issue not addressed in the study

19. Incomplete outcome data “Attrition” (drop-out): no data Withdrawal Do not attend follow-up appointment Failure to complete questionnaire / diaries Cannot be located (lost to follow-up) Decision by investigator to cease follow-up Data or records are lost

20. Incomplete outcome data “Exclusion”:data available, but excluded from analysis Participants found to be ineligible after enrolment An “as treated” (or per-protocol) analysis: participants are only included if they received the intended intervention in accordance with the protocol

21. Assessing risk of bias Low risk of bias High risk of bias Complete outcome data Missing in both groups but reasons are reported and balanced across groups Reason unlikely to be connected with outcome (moved away) Difference in proportion of incomplete data across groups and related to outcomes (e.g. adverse effects in experimental group) Differences in the reasons for missing data (e.g. smoking cessation) “as treated” (per protocol) analysis

22. Selective outcome reporting = selection of a subset of the variables recorded for inclusion in publication, on the basis of the results For example: Omission of non-significant outcomes Choice of data for an outcome (e.g. osteoporosis) Choice of analysis (e.g. blood pressure) Reporting of subsets of data (e.g. sepsis) Under-reporting of data (e.g. only “not significant”)

23. Selective outcome reporting AdequateNot adequateUnclear All pre-specified outcomes are reported (protocol available) All expected outcomes are reported (protocol not available) Not all pre-specified outcomes are reported Primary outcome is reported using analysis method or subset which was not pre-specified Reported primary outcomes were not pre- specified Incomplete reporting Key outcome not reported Insuficient information

24. Presentation in your review “Risk of bias graph”

25. “Risk of bias summary”

26. Assessing risk of bias in NRS Selection bias(how was group allocation?) Performance bias(blinding, fidelity of interventions) Attrition bias(completeness of sample & follow-up) Reporting bias(selective outcome reporting) Confounding and adjustment

27. Confounding Comparison intervention - control Intervention versus control Difference in outcome Imbalance in prognostic factors ? ?

28. Confounding Association between 2 factors Presence of risk factor Occurrence of outcome Confounding factor

29. Confounding & adjustment At the stage of protocol: list potential confounding factors Identify the factors the authors have considered and omitted Assess balance between groups at baseline What did authors do to control for confounders (matching, restricting to subgroups, stratification, regression modelling)

30. Tool for NRS Downs and Black instrument (J Epidemiol Community Health 1998;52:377-84) 27 items: Reporting (10) External validity (applicability) (3) Internal validity - bias (7) Internal validity – confounding (6) Power (1)

31. Downs and Black instrument http://www.nccmt.ca/ registry/view/eng/9.html

32. Tool for NRS Newcastle-Ottawa Scale (NOS) (Wells 2008) 8 items covering 3 perspectives: Selection of study groups Comparibility of groups Ascertainement of exposure (case-control) or outcome (cohort)

33. http://www.ohri.ca/programs/ clinical_epidemiology/oxford.asp

34. Diagnostic studies QUADAS tool Whiting BMC Medical Research Methodology 2003;3:25 Cochrane version: 11 items (out of 14 original) Diagnostic Test Accuracy Working Group: handbook http://srdta.cochrane.org/ handbook-dta-reviews

36. Other risk of bias assessment tools SIGN (Scottish Intercollegiate Guidelines Network) http://www.sign.ac.uk/methodology/ checklists.html

38. In summary Risk of bias assessment is essential for systematic reviews For RCT: use the Cochrane tool For NRS: Higher risk of bias (selection bias & reporting bias) Use the appropriate tool to assess risk of bias Consider how potential confounders are addressed