1. EBD for Dental Staff Seminar 2: Core Critical Appraisal Dominic Hurst r.d.hurst@qmul.ac.uk evidenced.qm

2. Learning outcomes By the end of this seminar participants should be able to: 1.Describe a structured approach to appraising the validity, results and relevance of a study (either primary or secondary research) 2.Explain the concepts of internal and external validity, and bias 3.List common terms for presenting the results of a study including relative risk reduction, absolute risk reduction, number needed to treat 4.Explain what a 95% confidence interval is and what the significance of a p value less than 0.05 is

3. Structure of the seminars Seminar 1 Recap of EBD Using an educational prescription Structured questions and search Seminar 2 Critical appraisal Seminar 3 Communicating evidence to patients Getting evidence into practice Workshop Focused work on one or more of these

4. Why critically appraise a study? Put rubbish in, get rubbish out

5. Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context.* *http://www.medicine.ox.ac.uk/bandoli er/painres/download/whatis/What_is_cr itical_appraisal.pdf Critical appraisal Relevance Value Trust- worthiness

6. Three key questions Is the study valid? Internal validity What are the results? Size and precision of the results Can I use it with the patients I manage in my context? External validity and usefulness in this setting

7. Is the study valid? Are the results what they claim to be?

8. Internal validity In intervention or risk factor questions: The degree of confidence we have that the difference in outcomes between 2 or more groups is due to the intervention / exposure and not something else Regular fluoride varnish reduces caries incidence in children by 30-40% TMJD resolves of its own accord in 70% of patients after 3 months

9. Does a red pill make a mouse grow more than a blue pill?

10. What messes up internal validity? Confounder variables or factors Any variable other than the one of interest, that can influence the outcome Study concludes that “night shifts cause dental caries”. Is there any other explanation? Confounders lead to bias A systematic deviation from the truth High internal validity if bias is low / not present Low internal validity if bias is high

11. Using critical appraisal checklists RCTs Systematic reviews

12. Common sources of bias in primary studies Selection: Choose who gets what Performance: Groups cared for differently Detection: Outcomes measured differently Attrition: One group withdraws from trial more due to intervention Reporting: Choose favourable outcomes to report rather than all

13. For RCTs students would be expected to comment on: Was randomisation properly done? Computer randomisation, flipping a coin Were patients, clinicians, and research staff blinded? (if it’s possible…) It may be that only the person assessing an outcome could be blinded Were the groups similar at the start? Imbalances at the start in, say, disease level could be confounder E.g. caries experience Were they treated in the same way? Intervals between recalls, other treatments provided…anything that could be a potential confounder

14. Biases in Systematic Reviews Sources Poor search strategies Limited set of databases searched Study designs Mixing up different study designs of varying risk of bias puts the review at increased risk of bias e.g. mixing cohort study results with RCT results Selection Authors choose which papers to include Inclusion criteria change dependent on the results of studies Publication Not something the review authors can do much about but it may be there Only positive findings get published and therefore included in review Language When only English language publications are included

15. For SRs students would be expected to comment on: Studies included appropriate? Thorough search of 2+ databases in any language? Was there a quality assessment of included studies – and what was the result? Were the studies similar enough to be able to combine them in a meta-analysis – if this was done?

16. What are the results? Think absolutes rather than relatives…

17. Size (or magnitude) of the effect In a trial with 2000 patients having non-surgical extractions 1000 get the socket rinsed with chlorhexidine 1000 get the socket rinsed with water We follow up for 14 days We count how many get dry socket

18. IC No dry socket 9509001850 Dry socket 50100150 1000 2000 Risk of getting a dry socket with the intervention? 50/1000=5% Risk of getting a dry socket with the comparison? 100/1000=10% Absolute risk reduction (ARR)? 10%-5%=5% (0.05) Relative risk reduction (RRR)? (10%-5%)/10%=50% Number needed to treat (NNT) with the chlorhexidine rather than water to prevent one additional dry socket? 1/ARR=1/0.05=20 IC No dry socket 9559901985 Dry socket 51015 1000 2000 Risk of getting a dry socket with the intervention? 5/1000=0.5% Risk of getting a dry socket with the comparison? 10/1000=1.0% Absolute risk reduction (ARR)? 1.0%-0.5%=0.5% (0.005) Relative risk reduction (RRR)? (1.0%-0.5%)/1.0%=50% Number needed to treat (NNT) with the chlorhexidine rather than water to prevent one additional dry socket? 1/ARR=1/0.005=200

19. Other ways results are expressed Mean difference E.g. mean difference in probing depth Odds ratios Odds: the chance that something happens / the chance that it doesn’t IC No dry socket 9509001850 Dry socket 50100150 1000 2000 Odds of a dry socket if using Intervention? 50/950=0.053 Odds of a dry socket if using comparison? 100/900=0.111 Odds ratio? 0.053/0.11=0.477

20. Results in Systematic Reviews: Forrest Plot

22. Precision and statistical significance 95% Confidence intervals We are 95% confident that the population result would lie within this range either side of the study result P values If P<0.05 = a less than 1/20 chance that the result is due to chance P<0.005 means there is a less than 1 in 200 chance the result is due to chance

23. Can you use the results locally? External validity, your skills, your patients and your resources

24. Things to consider of any study Are all the outcomes that would be of interest to you reported? Think of harms as well as benefits Are the patients that were in the study so different? Is what is being done feasible by a student / you in this setting?

25. Key messages Critical appraisal asks 3 questions: Is the study valid? What are the results? Can they be used locally? Internal validity affected by confounding factors that cause bias Relative outcomes can be misleading…so opt for absolutes if you can

26. Thanks for participating Dominic Hurst r.d.hurst@qmul.ac.uk