Critical appraisal of the medical literature Partini Pudjiastuti, Sudigdo Sastroasmoro Child Health Department Faculty of Medicine University of Indonesia

Sudigdo Sastroasmoro sudigdo_s@yahoo.com Population & Sample Sudigdo Sastroasmoro sudigdo_s@yahoo.com

Population is a large group of study subjects (human, animals, tissues, blood specimens, medical records, etc) with defined characteristics [“Population is a group of study subjects defined by the researcher as population”] Sample is a subset of population which will be directly investigated. Sample should be (or assumed to be) representative to the population; otherwise all statistical analyses will be invalid All investigations are always performed in the sample, and the results will be applied to the population

Avoid using ambiguous terms Sample population Sampled population Populasi sampel

Target population = domain = population in which the results of the study will be applied. Usually character-ized by demographic & clinical characteristics; e.g. normal infants, teens with epilepsy, post-menopausal women with osteoporosis. Accessible population = subset of target population which can be accessed by the investigator. Frame: time & place. Example: teens with epilepsy in RSCM, 2000-2005; women with osteoporosis, 2002 RSGS Intended sample = subjects who meet eligibility criteria and selected to be included in the study Actual study subjects = subjects who actually completed the participation in the study

(demographic, clinical) Usually based on practical purposes Target population = DOMAIN Accessible population (+ time, place) (demographic, clinical) Appropriate sampling technique Actual study subjects Subjects completed the study Intended Sample [Subjects selected for study] [Non-response, drop outs, withdrawals, loss to follow-up]

[Internal validity: does Target population External validity II: Does AP represent TP? Accessible population [External validity I: Does IS represent AP?} Actual study subjects Intended Sample [Internal validity: does ASS represent IS?]

Sampling methods B. Non-probability sampling A. Probability sampling Simple random sampling (r. table, computer) Stratified random sampling Systematic sampling Cluster sampling Others: two stage cluster sampling, etc B. Non-probability sampling Consecutive sampling Convenience sampling Judgmental sampling

Note All statistical analyses (inferences) are based on random sampling Whether or not a sample is representative to the population depends on whether or not it resembles the results if it were done by random sampling

Statistical significance vs. clinical importance IMPORTANT!!! Statistical significance vs. clinical importance Negligible clinical difference may be statistically very significant if the number of subjects >>>. e.g., difference in reduction of cholesterol level of 3 mg/dl, n1=n2 = 10,000; p = 0.00002 Large clinical difference may be statistically non-significant if the no of subjects <<<, e.g. 30% difference in cure rate, if n1 = n2 = 10, p = 0.74

Cholesterol level, mg/dl Clinical importance vs. statistical significance Cholesterol level, mg/dl Standard treatment x = 220 x = 217 x = 300 mg/dl R n=10000 Clinical n=10000 New treatment x = 300 mg/dl t = df = 9998 p = 0.00002 Statistical

Clinical importance vs. statistical significance Cured Died Standard Rx 0 10 (100%) New Rx 3 7 (70%) Clinical Absolute risk reduction = 30% Fischer exact test: p = 0.211 Statistical

Correlation between abdominal circumference and total cholesterol level in middle-aged men R = 0.22, p = 0.031 Conclusion: There was a significant correlation between abdominal circumference and total cholesterol level in the subjects studied. Measuring abdominal circumference may predict the cholesterol level in middle-aged healthy men.

How important is important? Two percent mortality reduction is probably not important in your clinic In a community prevention, a simple measure that reduce 2% severe morbidity is probably important. Low dose aspirin reduces 2% cardiac events in 5 years (without aspirin 400 cardiac events per 10,000, with aspirin 200 cardiac events) Requires judgment

How to infer? Can the results of the study (in sample) be applied in the accessible or target population? Hypothesis testing & confidence interval

Statistic and Parameter An observed value drawn from the sample is called a statistic (cf. statistics, the science) The corresponding value in population is called a parameter We measure, analyze, etc statistics and translate them as parameters

Examples of statistics: Proportion Percentage Mean Median Mode Difference in proportion/mean OR RR Sensitivity Specificity Kappa LR NNT

There are 2 ways in inferring statistic into parameter: Hypothesis testing  p value Estimation:  confidence interval (CI) P Value & CI tell the same concept in different ways

P value Determines the probability that the observed results are caused solely by chance (probability to obtain the observed results if Ho were true)

C 30 (60%) 20 (40%) 50 E 40 (80%) 10 (20%) 50 X2= ; df = 1; p = 0.0432 Group Success Failure Total

Group Success Failure Total X2= ; df = 1; p = 0.0432 If drugs E and C were equally effective, we still can have the above result (difference of success rate of 20%) but the probability is small (4.32%) If drugs E and C were equally effective, the probability that the result is merely caused by chance is 4.32% If we define in advance that p<0.05 is significant, than the result is called statistically significant

Similar interpretation applies to ALL hypothesis testing: t-test, Anova, non-parametric tests, Pearson correlation, multivariate tests, etc: If null-hypothesis null were true, the probability of obtaining the result was ……. (example 0,02 or 2%, etc)

Confidence Intervals Estimate the range of values (parameter) in the population using a statistic in the sample (as point estimate)

P S X X X A statistic (point estimate) If the observed result in the sample is X, what is the figure in the population? P X S CI A statistic (point estimate)

Most commonly used CI: CI 90% corresponds to p 0.10 Note: p value  only for analytical studies CI  for descriptive and analytical studies

How to calculate CI General Formula: CI = p  Z x SE p = point of estimate, a value drawn from sample (a statistic) Z = standard normal deviate for , if  = 0.05  Z = 1.96 (~ 95% CI)

Example 1 100 FKUI students  60 females (p=0.6) What is the proportion of females in Indonesian FK students? (assuming FKUI represents FK in Indonesia)

Example pq SE(p) = n . 6 x . 4 95 % CI = . 6 ± 1 . 96 100 = . 6 ± 1 . . 6 x . 4 95 % CI = . 6 ± 1 . 96 100 = . 6 ± 1 . 96 X0.5/10 = . 6 ± . 1 = . 5 ; . 7

Example 2: CI of the mean 95% CI = x  1.96 x SEM 100 newborn babies, mean BW = 3000 (SD = 400) grams, what is 95% CI? 95% CI = x  1.96 x SEM

Examples 3: CI of difference between proportions (p1-p2) 50 patients with drug A, 30 cured (p1=0.6) 50 patients with drug B, 40 cured (p2=0.8)

Example 4: CI for difference between 2 means Mean systolic BP: 50 smokers = 146.4 (SD 18.5) mmHg 50 non-smokers = 140.4 (SD 16.8) mmHg x1-x2 = 6.0 mmHg 95% CI(x1-x2) = (x1-x2)  1.96 x SE (x1-x2) SE(x1-x2) = S x V(1/n1 + 1/n2)

Example 4: CI for difference between 2 means

Other commonly supplied CI Relative risk (RR) Odds ratio (OR) Sensitivity, specificity (Se, Sp) Likelihood ratio (LR) Relative risk reduction (RRR) Number needed to treat (NNT)

Suggested CI presentation: 95%CI: 1.5 to 4.5 95%CI: -2.5 to 4.3 95%CI: -12 to -6 Not recommended: 3 +1.5 Not recommended: -9+ -3

In contrast to CI for proportion, mean, diff In contrast to CI for proportion, mean, diff. between proportions/means, where the values of CI are symmetrical around point estimate, CI’s for RR, OR, LR, NNT are asymmetrical because the calculations involve logarithm

Examples RR = 5.6 (95% CI 1.2 ; 23.7) OR = 12.8 (95% CI 3.6 ; 44,2) NNT = 12 (95% CI 9 ; 26)

If p value <0.05, then 95% CI: exclude 0 (for difference), because if A=B then A-B = 0  p>0.05 exclude 1 (for ratio), because if A=B then A/B = 1,  p>0.05 For small number of subjects, computer calculated CI may not meet this rule due to correction for continuity automatically done by the computer

Concluding remarks In every study sample should (assumed to) be representative to the population. Otherwise all statistical calculations are not valid p values (hypothesis testing) gives you the probability that the result in the sample is merely caused by chance, it does not give the magnitude and direction of the difference Confidence interval (estimation) indicates estimate of value in the population given one result in the sample, it gives the magnitude and direction of the difference

Concluding remarks p value alone tends to equate statistical significance and clinical importance CI avoids this confusion because it provides estimate of clinical values and exclude statistical significance whenever applicable, supply CI especially for the main results of study in critical appraisal of study results, focus should be on CI rather than on p value.

1 The ultimate goal of clinical research is the use of evidence in source population

2 The best non-probabiity sampling is consecuitve sampling

3 P value refers to the probability of getting the observed result when the Ho were false

4 The mean difference of 2 measurements is 20 mmHg, with 95% CI 15 to 25 mmHg. The p value should be “statistically significant”

5 Confidence intervals give more information than p value

6 It is possible to have a study with good internal validity but poor external validity

7 If the odds ratio is 5, then the 95% CI may have values from 3 to 11

8 It is possible to have a significant difference even when the clinical difference is not important, but clinically important difference always statistically significant

9 Appropriate sampling method is mandatory to ensure generalization

10 Clinical epidemiology may include animal studies

11 The more wide the confidence interval, the more precise the result of any study

12 Assessment of clinical importance requires judgment

13 The confidence interval of any measure must include the point estimate

14 Selection of source population usually based on practical reasons

15 Diagnostic test, therapy, etiology, harm, are examples of basic research

Critical appraisal (making reading more worthwhile) What is Critical Appraisal? Critical appraisal = quality assessment ….process of weighing up evidence to see how useful it is in decision making .…a process of assessing the validity, importance, and usefulness of evidence Critical appraisal is about considering, evaluating and interpreting information in a systematic and objective way

Critically appraise what you read Separating the wheat from the chaff Time is limited – you should aim to quickly stop reading the dross Others contain useful information mixed with rubbish Simple checklists enable the useful information to be identified

Critical appraisal – Critical thinking Appraising (evaluating/reviewing) the available evidence to construct clinical reasoning strategies and to make decisions Finding strengths and limitations of written ‘evidence’ You need to decide what evidence to pay attention to (what is “worthy” of your attention) versus what to ignore

Why critically appraise? Supports sound decision making based on best available evidence Helps us determine (three R’s): How rigorous a piece of research is (Valid?) What the results are telling us (Important?) How relevant it is to our patient (Applicable?)

Why do we need evidence? Resources should be allocated to things that are EFFECTIVE The only way of judging effectiveness is EVIDENCE “In God we trust – all others need evidence”

Sources of Evidence Primary sources Based on experiments and published research Secondary sources Systematic reviews Clinical guidelines Journals of secondary publication e.g. Evidence Based Medicine

“5S” Pyramid of Evidence Resources

Levels of evidence Systematic reviews of RCTs/high quality RCTs Systematic reviews of cohort studies, lower quality RCTs, outcomes research Systematic reviews of case controls, case control studies Case Series Expert opinion See http://www.cebm.net/levels_of_evidence.asp for complete description

Types of Evidence - Question Types Type of Question Best Evidence Health care interventions: treatment, prevention Quantitative: Systematic Review of RCTs or RCT Harm or Etiology Quantitative: Observational Study - Cohort or Case Control Prognosis Quantitative: Observational Study - Cohort, Case Control Diagnosis or Assessment Comparison to Gold Standard Economics Cost-effectiveness Study Meaning Qualitative: case study,

Key quality parameters Validity Reliability Importance

Validity: internal and external Internal - Is the study designed in such a way that we can trust the findings? External - Is the study designed in such a way that I can generalize the findings? Studies with good internal validity may not have good external validity if the study subjects do not represent population With poor internal validity, question about external validity is not relevant

Reliability If the study was conducted again, would the results be the same? Usually interpreted as the accuracy of measurement.

Importance What was the effect size or magnitude of effect? (Would the evidence change your practice?) Clinical vs. statistical significance.

Tools for Critical Appraisal EBM “simplified” approach: Are the results valid? What are the results? Will the results help me in patient care? V I A

Evidence based medicine 5 steps Formulate question Evaluate performance Efficiently track down best available evidence Implement changes in clinical practice Critically review the validity and usefulness of the evidence

Check list for medical literature (completeness) Title Authors Abstract: structured? Informative? Abbreviation? Introduction: length? Relevant references? Target population? Methods: Design Eligibility (inclusion and exclusion) criteria Sample size, sampling method Randomization: technique, concealment Intervention: masking? Measurement: blinding? - Primary & secondary outcome Definitions Analysis

Check list for medical literature (contd.) Results Baseline characteristics Main outcome Secondary outcome Discussion General Strength and weakness Conclusions References Vancouver style Constant Acknowledgments Ethics approval Conflict of interest

What to assess? (in study of cause-effect relationship) General description Type of design Target population, source population, sample Sampling method Dependent and independent variables Main results?

What to assess? (in study of cause-effect relationship) B. Internal validity, non-causal relationship Influence of bias Influence of chance Influence of confounders

Bias What is a bias? A process that tends to produce results that depart systematically from the true values existing in the study population Types of bias Sample (subject selection) biases, which may result in the subjects in the sample being unrepresentative of the population which you are interested in Measurement (detection) biases, which include issues related to how the outcome of interest was measured Intervention (performance) biases, which involve how the treatment itself was carried out.

What to assess? (in study of cause-effect relationship) C. Internal validity, causal relationship Temporality (cause precedes effect) Strength of association (large difference, RR, OR, etc) or small p value or narrow confidence interval Biological gradient (dose dependence) Consistency among studies (diff. populations/designs) Specificity (certain factor results in certain effect) Coherence (does not conflict with current knowledge) Biological plausibility: can be explained with current knowledge (at least in part)

What to assess? (in study of cause-effect relationship) D. External validity Applicable to study subjects Applicable to source population Applicable to target population

11 items, each with 3 sections Can you find this information in the paper? Is there any problem? Does this problem threaten the validity? You can quickly work through any journal article using this format.

11 items What is the research question? What is the study type? What are the outcome factors and how are they measured? What are the study factors and how are they measured? What important confounders are considered? What are the sampling frame and sampling method? In an exp., how were the subjects assigned to groups? In a longitudinal study, how many reached final follow-up? In a case control study, are the controls appropriate? (etc) Are statistical tests considered? Are the results clinically/socially significant (important)? Is the study ethical? What conclusions did the authors reach? These are the main items on the worksheet, and can be applied to most journal articles

1. What is the research question? Any problem? Is it concerned with the impact of an intervention, causality or determining the magnitude of a health problem? (Does this problem threaten the validity?) Is it a well stated research question/hypothesis?

2. What is the study type? (Any problem?) Is the study type appropriate to the research question? (Does this problem threaten the validity?) If not, how useful are the results produced by this type of study?

3. What are the outcome factors and how are they measured? (Any problem?) a) are all relevant outcomes assessed b) is there measurement error? (Does this problem threaten the validity?) a) how important are omitted outcomes b) is measurement error an important source of bias?

4. What are the study factors and how are the measured? (Any problem?) Is there measurement error? (Does this problem threaten the validity?) Is measurement error an important source of bias?

5. What important potential confounders are considered? (Any problem?) Are potential confounders examined and controlled for? (Does this problem threaten the validity?) Is confounding an important source of bias?

6. What are the sampling frame and sampling method? (Any problem?) Is there selection bias? (Does this problem threaten the validity?) Does this threaten the external validity of the study? We have now considered the major sources of bias – selection, measurement and confounding.

7. Questions of internal validity (Any problem?) Experimental: how were the subjects assigned to groups? Longitudinal study, how many reached follow-up? Case control study, are the controls appropriate? Note: other issues of relevance to internal validity are considered under the other headings in this critical appraisal system. You can add your own questions, and also design your own questions for other study types such as cross sectional studies and systematic reviews (Does this problem threaten the validity?) Does this threaten the internal validity of the study? It is difficult to include all possible sources of bias that might threaten the internal validity of a study, and you are welcome to add your own favourites. The ones included in this side are those that might be considered as citiical for these types of study

8. Are statistical tests considered? (Any problem?) Were the tests appropriate for the data? Are confidence intervals given? Is the power given if a null result? In a trial, are results presented as absolute risk reduction as well as relative risk reduction? (Does this problem threaten the validity?) If not, how useful are the results?

9. Are the results clinically/socially significant? (Any problem?) Was the sample size adequate to detect a clinically/socially significant result? Are the results presented in a way to help in health policy decisions? (Does this problem threaten the validity?) Is the study useful? The difference between statistical significance and a result that is of use is an important distinction to make

10. Are ethical issues considered? (Any problem?) Does the paper indicate ethics approval? Can you identify potential ethical issues? (Does this problem threaten the validity? Are the results or their application compromised? There are a number of ethical issues to consider in both the conduct of research and the application of the results

11. What conclusions did the authors reach about the study question? (Any problem?) Do the results apply to the population in which you are interested? (Does this problem threaten the validity? Will you use the results of the study? The final issue is whether the study has passed your critical appraisal in terms of your decision to use the results in your own practice, teaching or further research

Appraisal Tools Systematic Reviews Randomised Controlled Trials Tools from the Critical Appraisal Skills Programme (CASP) Systematic Reviews Randomised Controlled Trials Qualitative Research Studies Cohort Studies Case-Control Studies Diagnostic Test Studies Economic Evaluation Studies Available at: http://www.phru.nhs.uk/casp/critical_appraisal_tools.htm

Study Designs Recap Effectiveness of Therapy Risk Factors / Prognosis Diagnosis Attitudes & Beliefs Randomised Controlled Trial Cohort Study Survey using gold standard Qualitative (Interviews, Observations, etc)

Critical appraisal - Validity Methods - Importance Results - Applicability Methods Results Discussion

Thanks