Funnel Plot Medline Search Strategy for RCTs and Reviews -------------------------------------------------------------------------------- 1 diet, protein-restricted/ 2 diet, carbohydrate-restricted/ 3 1 or 2 4 diet fads/ 5 (carbohydrate* or protein*).ti,ab. 6 4 and 5 7 exp dietary proteins/ 8 dietary carbohydrates/ 9 (diet* or intake*).ti,ab. 10 (high* or increas* or rich or low* or restrict* or decreas* or reduc*).ti,ab. 11 (7 or 8) and 9 and 10 12((carbohydrate* or protein*) adj3 (high* or increas* or rich or low* or restrict* or decreas* or reduc*)).ti,ab. 13 12 and 9 14 3 or 6 or 11 or 13 15 randomized controlled trial.pt. 16 controlled clinical trial.pt. 17 randomized.ab. 18 placebo.ab. 19 clinical trials as topic.sh. 20 randomly.ab. 21 trial.ti. 22 or/15-21 23 humans.sh. 24 22 and 23 25 14 and 24
What information do you think would increase or decrease your confidence in these results? What information do you think would indicate that more research is or is not necessary? Work with your neighbor and discuss for 5 mins
To make conclusions consider.... likelihood of effect and confidence in that effect
To make conclusions consider.... Likelihood/magnitude of effect – Mood will drop by 5 points on a scale of 0 to 10 when on a high protein diet. quality of the evidence – Low: This estimate will likely change with more research.
Summary of Findings Table a summary of the key findings from the systematic review for users a transparent aid and record of the authors’ interpretation of the results to make conclusions
Weighing the criteria for overall quality of evidence In fact in this example, – Allocation concealment is unclear in one of the studies – Only three of five studies measured major bleeding - a primary outcome in anticoagulation studies – suggesting selective outcome reporting – The confidence intervals include potential for harm or no harm I might say that my confidence in the results is “low” and that more research is likely to change the results
The pooled SMD for pain reduction comparing glucosamine to placebo was 0.61, which represents a moderate clinically significant treatment benefit in favour of glucosamine
Quality of evidence across studies for an outcome HighFurther research is very unlikely to change our confidence in the estimate of effect or accuracy. ModerateFurther research is likely to have an important impact on our confidence in the estimate of effect or accuracy and may change the estimate. LowFurther research is very likely to have an important impact on our confidence in the estimate of effect or accuracy and is likely to change the estimate. Very lowAny estimate of effect or accuracy is very uncertain.
GRADE: recommendation – quality of evidence Clear separation: 1) 4 categories of quality of evidence: (High), (Moderate), (Low), (Very low) ? – methodological quality of evidence – likelihood of bias – by outcome and across outcomes 2) Recommendation: 2 grades – conditional (aka weak) or strong (for or against an intervention)? – Balance of benefits and downsides, values and preferences, resource use and quality of evidence *www.GradeWorking-Group.org
GRADE Quality of Evidence In the context of a systematic review The quality of evidence reflects the extent to which we are confident that an estimate of effect is correct. In the context of making recommendations The quality of evidence reflects the extent to which our confidence in an estimate of the effect is adequate to support a particular recommendation.
Determinants of quality RCTs observational studies 5 factors that can lower quality 1.limitations in detailed design and execution (risk of bias criteria) 2.Inconsistency (or heterogeneity) 3.Indirectness (PICO and applicability) 4.Imprecision (number of events and confidence intervals) 5.Publication bias 3 factors can increase quality 1.large magnitude of effect 2.all plausible residual confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed 3.dose-response gradient
1. Design and Execution/Risk of Bias Examples: Inappropriate selection of exposed and unexposed groups Failure to adequately measure/control for confounding Selective outcome reporting Failure to blind (e.g. outcome assessors) High loss to follow-up Lack of concealment in RCTs Intention to treat principle violated
3. Directness of Evidence generalizability, transferability, applicability differences in – populations/patients (HIC – L/MIC, women in general – pregnant women) – interventions (all techniques, new - old) – comparator appropriate (newer technique – old or no technique) – outcomes (important – surrogate: CIN I – cancer) indirect comparisons – interested in A versus B – have A versus C and B versus C – Cryo + antibiotics versus no intervention versus Cryo - antibiotics
EVIDENCE PROFILE Question: Cyrotherapy with antibiotics vs no antibiotics for histologically confirmed CIN 1 All rates presented at 12 months with assumption that events would occur within this time frame. 2 Indirect analysis between single arm observational studies Quality assessmentNo of patientsEffect QualityImportance No of studies DesignLimitations Inconsisten cy IndirectnessImprecisionOther Cryotherap y with antibiotics No antibiotics Relative (95% CI) Absolute Major infection (follow-up 12 months 1 ; requiring hospitalisation or blood transfusion) 16observation al studies no serious limitations no serious inconsistenc y serious 2 no serious imprecision none 0/1600 (0%) 10/4573 (0.22%) RD 0 (0 to 0) 0 fewer per 1000 IMPORTANT Resource use - not measured All severe adverse events (follow-up 12 months; (major infections and bleeding, pelvic inflammatory disease, stenosis, etc ) 17observation al studies no serious limitations no serious inconsistenc y serious 2 no serious imprecision none 0/1705 (0%) 22/5142 (0.43%) RD 0 (0 to 0) 0 fewer per 1000 IMPORTANT
4. Publication Bias Should always be suspected – Only small “positive” studies – For profit interest – Various methods to evaluate – none perfect, but clearly a problem
Egger M, Smith DS. BMJ 1995;310:752-54 45 I.V. Mg in acute myocardial infarction Publication bias Meta-analysis Yusuf S.Circulation 1993 ISIS-4 Lancet 1995
Egger M, Cochrane Colloquium Lyon 2001 46 Funnel plot Standard Error Odds ratio 0.10.313 3 2 1 0 100.6 Symmetrical: No publication bias
Egger M, Cochrane Colloquium Lyon 2001 47 Funnel plot Standard Error Odds ratio 0.10.313 3 2 1 0 100.6 Asymmetrical: Publication bias? 0.4
5. Imprecision Small sample size – small number of events Wide confidence intervals – uncertainty about magnitude of effect Extent to which confidence in estimate of effect adequate to support decision
For systematic reviews If the 95% CI excludes a relative risk (RR) of 1.0 and the total number of events or patients exceeds the OIS criterion, precision is adequate. If the 95% CI includes appreciable benefit or harm (we suggest a RR of under 0.75 or over 1.25 as a rough guide) rating down for imprecision may be appropriate even if OIS criteria are met.
Optimal information size We suggest the following: if the total number of patients included in a systematic review is less than the number of patients generated by a conventional sample size calculation for a single adequately powered trial, consider rating down for imprecision. Authors have referred to this threshold as the “optimal information size” (OIS)
2.0 0.50 Ischemic stroke point estimate and confidence interval Figure 1, Rating down for imprecision in guidelines: Thresholds are key Favors Intervention Favors Control Risk difference (%) Threshold if side effects and toxicity appreciable, NNT = 100. Confidence interval crosses threshold, rate down for imprecision Threshold if side effects, toxicity and cost minimal, NNT = 200. Entire confidence interval to left of threshold, do not rate down for imprecision
Figure 2: Corticosteroids to reduce hospital mortality in septic shock
Figure 4: Optimal information size given alpha of 0.05 and beta of 0.2 for varying control event rates and relative risks For any chosen line, evidence meets optimal information size criterion if sample size above the line
Total Number of EventsRelative Risk ReductionImplications for meeting OIS threshold 100 or less< 30%Will almost never meet threshold whatever control event rate 20030%Will meet threshold for control event rates for ~ 25% or greater 20025%Will meet threshold for control event rates for ~ 50% or greater 20020% Will meet threshold only for control event rates for ~ 80% or greater 300> 30%Will meet threshold 30025%Will meet threshold for control event rates ~ 25% or greater 30020%Will meet threshold for control event rates ~ 60% or greater 400 or more> 25%Will meet threshold for any control event rate 400 or more20%Will meet threshold for control event rates of ~ 40% or greater Table 1: Optimal information size implications from Figure 5
What can raise quality? 1. large magnitude can upgrade (RRR 50%/RR 2) – very large two levels (RRR 80%/RR 5) – criteria everyone used to do badly almost everyone does well
What can raise quality? 2. dose response relation – (higher INR – increased bleeding) – childhood lymphoblastic leukemia risk for CNS malignancies 15 years after cranial irradiation no radiation: 1% (95% CI 0% to 2.1%) 12 Gy: 1.6% (95% CI 0% to 3.4%) 18 Gy: 3.3% (95% CI 0.9% to 5.6%)
In terms of high altitude sickness, symptoms generally do not manifest below 1500 m. From about 1500 to 2500 m, symptoms are generally mild, if experienced at all. At 2500 m, symptoms of mild to moderate acute mountain sickness (AMS) become quite common among unacclimatized visitors after rapid ascent. At this altitude high altitude pulmonary edema (HAPE) may also occur, but it is more common above 3000 m. Above 3000 to 4000 m, AMS is common among people who have not properly acclimatized, and the risk of severe consequences, including life-threatening HAPE and cerebral edema, is substantial.
What can raise quality? 3. all plausible residual confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed
All plausible residual confounding would result in an overestimate of effect Hypoglycaemic drug phenformin causes lactic acidosis The related agent metformin is under suspicion for the same toxicity. Large observational studies have failed to demonstrate an association – Clinicians would be more alert to lactic acidosis in the presence of the agent Vaccine – adverse effects
Pulling it all together and drawing conclusions
carefully consider and assess all the factors that may influence the quality of evidence bear in mind that down- and upgrading for specific quality factors should be done in the context of all of the factors that influence the quality of evidence downgrading for one quality criterion may influence how the next quality criterion is dealt with
Within and among Downgrade or upgrade on a continuum Downgrade or upgrade – WITHIN each category – AMONG the categories
Example: Meta- analysis of 5 studies uncertainty about three factors: study limitations/RoB, inconsistency, and imprecision Uncertainty not serious enough to downgrade each factor Option to pick one or two levels to downgrade Indicate in footnotes why and why did not downgrade for those factors (e.g. There was some uncertainty but already downgraded for...)
No, there are no serious limitations – although there …. Yes, there are serious limitations – most people would agree that selective reporting is…. Yes, there are very serious limitations – there is a risk of bias but only for the one criteria of selective reporting Would you downgrade for risk of bias?
Quality now? Low Observational studies could have provided higher quality evidence
Flavanoids for Hemorrhoids venotonic agents – mechanism unclear, increase venous return popularity – 90 venotonics commercialized in France – none in Sweden and Norway – France 70% of world market possibilities – French misguided – rest of world missing out
Systematic Review 14 trials, 1432 patients key outcome – risk not improving/persistent symptoms – 11 studies, 1002 patients, 375 events – RR 0.4, 95% CI 0.29 to 0.57 minimal side effects is France right? what is the quality of evidence?
What can lower quality? Study limitations/risk of bias – lack of detail re concealment – questionnaires not validated rate down quality for study limitations/RoB? indirectness – no problem inconsistency, need to look at the results
What can lower quality? risk of bias – lack of detail re concealment – questionnaires not validated Inconsistency – heterogeneity p < 0.001; I 2 65.1% indirectness imprecision – RR 0.4, 95% CI 0.29 to 0.57 Publication bias – 40 to 234 patients, most around 100
Conclusions WHO guidelines should be based on the best available evidence to be evidence based GRADE is the approach used by WHO and gaining acceptance internationally combines what is known in health research methodology and provides a structured approach to improve communication Does not avoid judgments but provides framework Criteria for evidence assessment across questions and outcomes Criteria for moving from evidence to recommendations Transparent, systematic four categories of quality of evidence two grades for strength of recommendations Transparency in decision making and judgments is key