1. Randomized Controlled Trials, Systematic Reviews and Meta-analysis Achilleas Thoma, MD, MSc, FRCS(C) CORE Jun 11 2008 Division of Plastic Surgery, Departments of Surgery & Clinical Epidemiology and Biostatistics McMaster University

2. Learning Objectives 1.What a RCT and a Systematic Review are. 2.Why we use them.

4. Evidence-based Surgery Definition: The conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patientsThe conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients

6. Leve l Therapy/Prevention, Aetiology/Harm PrognosisDiagnosis Differential diagnosis/symptom prevalence study Economic and decision analyses 1a SR (with homogeneity*) of RCTs homogeneity* SR (with homogeneity*) of inception cohort studies; CDR† validated in different populations homogeneity* CDR†homogeneity* CDR† SR (with homogeneity*) of Level 1 diagnostic studies; CDR† with 1b studies from different clinical centres homogeneity*CDR†homogeneity*CDR† SR (with homogeneity*) of prospective cohort studies homogeneity* SR (with homogeneity*) of Level 1 economic studies homogeneity* 1b Individual RCT (with narrow Confidence Interval‡) Confidence Interval‡Confidence Interval‡ Individual inception cohort study with > 80% follow-up; CDR† validated in a single population CDR† Validating** cohort study with good††† reference standards; or CDR† tested within one clinical centre good††† CDR† good††† CDR† Prospective cohort study with good follow-up**** Analysis based on clinically sensible costs or alternatives; systematic review(s) of the evidence; and including multi-way sensitivity analyses 1c All or none§ All or none§ All or none case-series Absolute SpPins and SnNouts†† Absolute SpPins and SnNouts†† All or none case-series Absolute better-value or worse-value analyses 2a SR (with homogeneity*) of cohort studies homogeneity* SR (with homogeneity*) of either retrospective cohort studies or untreated control groups in RCTs homogeneity* SR (with homogeneity*) of Level >2 diagnostic studies homogeneity* SR (with homogeneity*) of 2b and better studies homogeneity* SR (with homogeneity*) of Level >2 economic studies homogeneity* 2b Individual cohort study (including low quality RCT; e.g., <80% follow-up) Retrospective cohort study or follow-up of untreated control patients in an RCT; Derivation of CDR† or validated on split- sample§§§ only CDR† Exploratory** cohort study with good†††reference standards; CDR† after derivation, or validated only on split-sample§§§ or databases good††† CDR† good††† CDR† Retrospective cohort study, or poor follow-up Analysis based on clinically sensible costs or alternatives; limited review(s) of the evidence, or single studies; and including multi-way sensitivity analyses 2c "Outcomes" Research; Ecological studies "Outcomes" Research Ecological studies Audit or outcomes research 3a SR (with homogeneity*) of case-control studies homogeneity* SR (with homogeneity*) of 3b and better studies homogeneity* SR (with homogeneity*) of 3b and better studies homogeneity* SR (with homogeneity*) of 3b and better studies homogeneity* 3b Individual Case-Control Study Non-consecutive study; or without consistently applied reference standards Non-consecutive cohort study, or very limited population Analysis based on limited alternatives or costs, poor quality estimates of data, but including sensitivity analyses incorporating clinically sensible variations. 4 Case-series (and poor quality cohort and case- control studies§§) poor quality cohort and case- control studies§§poor quality cohort and case- control studies§§ Case-series (and poor quality prognostic cohort studies***) poor quality prognostic cohort studies***poor quality prognostic cohort studies*** Case-control study, poor or non- independent reference standard Case-series or superseded reference standards Analysis with no sensitivity analysis 5 Expert opinion without explicit critical appraisal, or based on physiology, bench research or "first principles" Expert opinion without explicit critical appraisal, or based on economic theory or "first principles"

7. Hierarchy of Evidence Systematic Reviews of Randomized Controlled Trials (Meta-analysis) Single Randomized Controlled Trial (RCT) Systematic Review of Observational Studies Addressing Patient-Important Outcomes Single Observational Study Addressing Patient-Important Outcomes Physiologic Studies Unsystematic Clinical Observations

8. Hierarchy of Evidence A well-conducted systematic review or meta-analysis of well executed randomized controlled trials provides the highest level of evidence to support an answer to a surgical questionA well-conducted systematic review or meta-analysis of well executed randomized controlled trials provides the highest level of evidence to support an answer to a surgical question

9. Randomized Controlled Trial (RCT) Definition: An experiment in which individuals are randomly allocated to receive or not receive an experimental preventative, therapeutic, or diagnostic procedure and then followed to determine the effect of the interventionAn experiment in which individuals are randomly allocated to receive or not receive an experimental preventative, therapeutic, or diagnostic procedure and then followed to determine the effect of the intervention

10. RCTs in Surgery Most scientifically rigorous study design to evaluate the effect of a new surgical interventionMost scientifically rigorous study design to evaluate the effect of a new surgical intervention Offers the maximum protection against biasesOffers the maximum protection against biases Balances both known and unknown prognostic factors across treatment groupsBalances both known and unknown prognostic factors across treatment groups

11. RCTs in Surgery Not all of the questions we face in surgery can be answered by the RCT designNot all of the questions we face in surgery can be answered by the RCT design Must consider the plausibility and feasibility of the research questionMust consider the plausibility and feasibility of the research question

12. Plausibility Is the question answerable or not?Is the question answerable or not? For example: Population = congenitally absent ear children Intervention = Nagata technique Comparative = “genetic engineering method” Outcome = “new” ear What is the problem here?

13. Feasibility Can the study design we choose answer the research question?Can the study design we choose answer the research question? For Example: Population = cosmetic abdominoplasty patients Intervention = intermittent lower extremity pump Comparative = low molecular heparin Outcome = prevention of fatal pulmonary embolism What do you think of this RCT?

14. Questions of Harm For questions of harm, appropriate study designs include case-control studies and cohort studiesFor questions of harm, appropriate study designs include case-control studies and cohort studies For Example: Population = patients with replanted digits Intervention = continue smoking Comparative = non-smoking Outcome = short-term survival of replanted digits What is the problem here?

15. Conducting a RCT Considerations: Surgical equipoiseSurgical equipoise Surgical learning curveSurgical learning curve Differential careDifferential care RandomizationRandomization ConcealmentConcealment Expertise-based designExpertise-based design BlindingBlinding Intention to treat analysisIntention to treat analysis Loss to follow-upLoss to follow-up Treatment effects and implications for sample size calculations.Treatment effects and implications for sample size calculations.

16. Surgical Equipoise Equipoise = a state of genuine uncertainty RE: benefits, harms that may result from each of two or more surgical procedures.Equipoise = a state of genuine uncertainty RE: benefits, harms that may result from each of two or more surgical procedures. There are no scientific or ethical concerns about Surgery A being better than Surgery B for a particular patientThere are no scientific or ethical concerns about Surgery A being better than Surgery B for a particular patient

17. Surgical Learning Curve Surgeon’s cumulative experienceSurgeon’s cumulative experience Continuous refinement of patient selection, operative technique, and post-operative careContinuous refinement of patient selection, operative technique, and post-operative care What is the problem here?What is the problem here?

18. Differential Care In OR: better hemostasis, give antibiotics to A and not B, staff person does surgery as compared to residentIn OR: better hemostasis, give antibiotics to A and not B, staff person does surgery as compared to resident Outside the OR: more frequent follow- up, physiotherapy to A but not to BOutside the OR: more frequent follow- up, physiotherapy to A but not to B What is the problem here?What is the problem here?

19. Randomization Even or odd number birth dateEven or odd number birth date Flip a coinFlip a coin Alternate chart numbersAlternate chart numbers Random number tablesRandom number tables Automated telephone systemAutomated telephone system Internet randomization systemInternet randomization system What are appropriate randomization techniques?

20. Randomization Randomize as close to surgery as possibleRandomize as close to surgery as possible For example: Breast reduction RCTFor example: Breast reduction RCT

21. Concealment Randomization by itself is not adequate. We need to conceal the randomization process.Randomization by itself is not adequate. We need to conceal the randomization process. In some trials envelopes are used carrying the randomization armIn some trials envelopes are used carrying the randomization arm What is the problem here?What is the problem here?

22. Expertise-based Design Patients are randomized to surgeons who do their operation of preference, as opposed to randomizing patients to a treatment group.Patients are randomized to surgeons who do their operation of preference, as opposed to randomizing patients to a treatment group. Each participating centre must have some surgeons doing each type of operationEach participating centre must have some surgeons doing each type of operation

23. Blinding There are 6 potential levels of blinding: 1.The patients 2.The clinicians who administer the treatment 3.The clinicians care for the patients during the trial 4.The individuals who assess the patients throughout the trial and collect the data 5.The data analyst 6.The investigators who interpret and write the results of the trial

24. Blinding Surgeons ?Surgeons ? Patients ?Patients ?

25. Blinding Placebo Effect If a patient knows he/she received a treatment that he/she believes is better, than he/she may feel better even if the there is no underlying benefit.If a patient knows he/she received a treatment that he/she believes is better, than he/she may feel better even if the there is no underlying benefit. The same holds true for the surgeon or designated assessor for the study if they are not blindedThe same holds true for the surgeon or designated assessor for the study if they are not blinded

26. Intention to Treat Analysis The analysis of the outcomes is based on the treatment arm to which patients were randomized, and not on which surgical treatment they received.The analysis of the outcomes is based on the treatment arm to which patients were randomized, and not on which surgical treatment they received. Includes all patients, regardless of whether they actually satisfied the entry criteria, received the treatment to which they were randomly allocated, or deviated from the protocol.Includes all patients, regardless of whether they actually satisfied the entry criteria, received the treatment to which they were randomly allocated, or deviated from the protocol.

27. Loss to Follow-Up Failure to account for all patients at end of the study may invalidate the RCT and reduce study powerFailure to account for all patients at end of the study may invalidate the RCT and reduce study power Researchers suggest that 20% loss potentially threatens validityResearchers suggest that 20% loss potentially threatens validity

28. Treatment Effect and Sample Size Treatment Effect: –Effect size: The size of the difference that the study is designed to detect. The minimum clinically important difference (MCID) is the smallest difference between 2 groups that would be clinically worth detecting.

29. Treatment Effect and Sample Size Sample Size: –Power varies directly in proportion to the number of participants. The larger the sample size, the greater is the power and more information about the true difference is obtained.

30. Hulley SB, Cummungs SR, Browner WS, Grady D, Hearst N, Newman TB. Designing Clinical Research, 2nd Edition. Lippincott Williams & Wilkins, Philadelphia, 2001

31. Power Analysis To make a statistical inference, we need to set two hypotheses:To make a statistical inference, we need to set two hypotheses: –Null hypothesis ( there is no difference) –Alternate hypothesis (there is a difference).

32. Power Analysis Type I Error: alpha (  ) Type II Error: beta (  )

33. Understanding Power Analysis Typically  is = 0.05 and  = 0.2 and the power = 1 -  =.80Typically  is = 0.05 and  = 0.2 and the power = 1 -  =.80

34. Sample Size Calculation

35. Conducting a RCT in Surgery In order to conduct high quality, large RCTs multi-centre collaboration is required.In order to conduct high quality, large RCTs multi-centre collaboration is required. Collaboration with biostatisticians, health economists, epidemiologists, and clinical trialistsCollaboration with biostatisticians, health economists, epidemiologists, and clinical trialists

36. Beware! Not all published RCTs are reported well or are of good methodological quality.Not all published RCTs are reported well or are of good methodological quality. Guidelines exist for:Guidelines exist for: 1.The Reporting of RCTs (i.e. CONSORT) and 2.The appraisal of RCT methodological quality (i.e. Detsky Quality Assessment Scale)

37. Quality of Reporting Consolidated Standards of Reporting Trials (CONSORT)Consolidated Standards of Reporting Trials (CONSORT) Checklist of essential items that should be included in reports of RCTs and a diagram for documenting the flow of participants through a trialChecklist of essential items that should be included in reports of RCTs and a diagram for documenting the flow of participants through a trial

38. CONSORT Checklist 1. How participants were allocated to interventions 2. Scientific background and explanation of rationale. 3a. Eligibility criteria for participants. 3b. The settings & locations where the data were collected. 4. Precise details of the interventions 5. Specific objectives and hypotheses 6a. Defined primary and secondary outcome measures. 6b. Methods used to enhance the quality of measurements 7a. How sample size was determined. 7b. Explanation of any interim analyses 8a. Method used to generate the random allocation sequence. 8b. Details of any restriction [of randomization] 9. Method used to implement the random allocation sequence 10. Who generated the allocation sequence, who enrolled participants, and who assigned participants to their groups.

39. Assessing Methodological Quality In a quality assessment scale, the responses to the individual items are summed to create an overall summary score representing trial qualityIn a quality assessment scale, the responses to the individual items are summed to create an overall summary score representing trial quality –i.e. Detsky Quality Scale, Jadad Scale

40. Jadad Scale Please read the article and try to answer the following questions (see attached instructions): 1.Was the study described as randomized (this includes the use of words such as randomly, random, and randomization)? 2.Was the study described as double blind? 3.Was there a description of withdrawals and dropouts?

41. Jadad Scale cont. Scoring the items: Either give a score of 1 point for each “yes” or 0 points for each “no.” There are no in-between marks.Either give a score of 1 point for each “yes” or 0 points for each “no.” There are no in-between marks. Give 1 additional point if: For question 1, the method to generate the sequence of randomization was described and it was appropriate (table of random numbers, computer generated, etc.)Give 1 additional point if: For question 1, the method to generate the sequence of randomization was described and it was appropriate (table of random numbers, computer generated, etc.) And/or: If for question 2 the method of double blinding was described and it was appropriate (identical placebo, active placebo, dummy, etc.) Deduct 1 point if: For question 1, the method to generate the sequence of randomization was described and it was inappropriate (patients were allocated alternately, or according to date of birth, hospital number, etc.)Deduct 1 point if: For question 1, the method to generate the sequence of randomization was described and it was inappropriate (patients were allocated alternately, or according to date of birth, hospital number, etc.) And/or For question 2, the study was described as double blind but the method of blinding was inappropriate (e.g., comparison of tablet versus injection with no double dummy)

42. Jadad Scale cont. Thoma et al (Plast. Reconstr. Surg. 114: 1137, 2004.)

43. Systematic Reviews Definition: “The application of scientific strategies that limit bias to the systematic assembly, critical appraisal, and synthesis of all relevant studies on a specific topic”“The application of scientific strategies that limit bias to the systematic assembly, critical appraisal, and synthesis of all relevant studies on a specific topic”

44. Systematic Reviews Aim to: Summarize the existing literatureSummarize the existing literature Resolve conflicts or controversies in the literatureResolve conflicts or controversies in the literature Clarify the results of multiple studiesClarify the results of multiple studies Evaluate the need for further studiesEvaluate the need for further studies

45. Conducting Systematic Reviews 1.Before embarking on a RCT, you must be familiar with the “cutting edge” of knowledge for a health care problem 1.It is important to summarize this “cutting edge evidence” in the form of a systematic review and apply it to your clinical practice

46. Narrative Vs. Systematic Reviews Cook DJ, Mulrow CD, Haynes RB. Ann Intern Med 1997;126(5):376-380 Narrative Review Systematic Review Research question is often broad Well-focused clinical question Search strategy is not defined or systematic Explicit search strategy, outlining study inclusion/exclusion criteria Article selection is not systematic Article selection is specific to inclusion/exclusion criteria Appraisal of study quality may not be performed Articles are critically appraised and strengths and weaknesses documented Qualitative summary of findings Qualitative or quantitative analysis of findings

47. Meta-Analysis The results of the primary studies that meet the standards for inclusion in a review are mathematically pooled to give a result that is more precise because of the overall increase in numbers of study participants contributing data.The results of the primary studies that meet the standards for inclusion in a review are mathematically pooled to give a result that is more precise because of the overall increase in numbers of study participants contributing data.

48. Meta-analysis Aim to: Resolve controversy over whether a true effect exists, when results have been variable in single studiesResolve controversy over whether a true effect exists, when results have been variable in single studies Validate a statistically non-significant but clinically important result in a small studyValidate a statistically non-significant but clinically important result in a small study

55. Meta-Analysis Frequently, it is not feasible to conduct a meta-analysis in surgeryFrequently, it is not feasible to conduct a meta-analysis in surgery If the data accumulated from the primary studies are of poor quality or too heterogeneous to be reasonably pooled, a quantitative meta-analysis needs to be abandoned in favor of a systematic reviewIf the data accumulated from the primary studies are of poor quality or too heterogeneous to be reasonably pooled, a quantitative meta-analysis needs to be abandoned in favor of a systematic review Heterogeneity resulting from differences in clinical features of a population that is being studied or treated.

56. Systematic Reviews / Meta-analysis Systematic reviews and meta-analyses of the effectiveness of treatments can be performed based on RCTs and / or observational studiesSystematic reviews and meta-analyses of the effectiveness of treatments can be performed based on RCTs and / or observational studies However, RCTs are the traditional study design of choice for primary studies used in meta-analyses, as they are the most likely to be validHowever, RCTs are the traditional study design of choice for primary studies used in meta-analyses, as they are the most likely to be valid

57. Steps for Conducting a Systematic Review 1.Form a clinically relevant research question 2.Develop a detailed research protocol 3.Perform an exhaustive systematic search of the available literature 4.Critically appraise the quality of selected articles 5.Extract relevant data 6.Perform a statistical or qualitative analysis of the extracted results 7.Interpret the findings, and address the strengths and weaknesses of selected articles and of the review itself

58. Quality Assessment Methodological Quality Assessment Scales:Methodological Quality Assessment Scales: –Hoving index (Hoving et al. Spine 26:196, 2001) –Oxman and Guaytt index (Oxman and Guyatt. J. Clin. Epidemiol. 44:1271, 1991) Quality of Reporting Checklist:Quality of Reporting Checklist: –QUORUM

59. QUORUM Statement

60. QUORUM Flow Diagram

61. THANK YOU! athoma@mcmaster.ca