1. Critical appraisal of the medical literature
Partini Pudjiastuti, Sudigdo Sastroasmoro
Child Health Department
Faculty of Medicine University of Indonesia

2. Sudigdo Sastroasmoro sudigdo_s@yahoo.com
Population & Sample
Sudigdo Sastroasmoro

3. Population is a large group of study subjects (human, animals, tissues, blood specimens, medical records, etc) with defined characteristics [“Population is a group of study subjects defined by the researcher as population”]
Sample is a subset of population which will be directly investigated. Sample should be (or assumed to be) representative to the population; otherwise all statistical analyses will be invalid
All investigations are always performed in the sample, and the results will be applied to the population

4. Avoid using ambiguous terms
Sample population
Sampled population
Populasi sampel

5. Target population = domain = population in which the results of the study will be applied. Usually character-ized by demographic & clinical characteristics; e.g. normal infants, teens with epilepsy, post-menopausal women with osteoporosis.
Accessible population = subset of target population which can be accessed by the investigator. Frame: time & place. Example: teens with epilepsy in RSCM, ; women with osteoporosis, 2002 RSGS
Intended sample = subjects who meet eligibility criteria and selected to be included in the study
Actual study subjects = subjects who actually completed the participation in the study

6. (demographic, clinical)
Usually based on practical
purposes
Target population =
DOMAIN
Accessible
population
(+ time,
place)
(demographic, clinical)
Appropriate
sampling
technique
Actual
study
subjects
Subjects
completed
the study
Intended
Sample
[Subjects selected
for study]
[Non-response, drop outs,
withdrawals, loss to follow-up]

7. [Internal validity: does
Target
population
External validity II:
Does AP represent TP?
Accessible
population
[External validity I:
Does IS represent AP?}
Actual
study
subjects
Intended
Sample
[Internal validity: does
ASS represent IS?]

8. Sampling methods B. Non-probability sampling A. Probability sampling
Simple random sampling (r. table, computer)
Stratified random sampling
Systematic sampling
Cluster sampling
Others: two stage cluster sampling, etc
B. Non-probability sampling
Consecutive sampling
Convenience sampling
Judgmental sampling

9. Note
All statistical analyses (inferences) are based on random sampling
Whether or not a sample is representative to the population depends on whether or not it resembles the results if it were done by random sampling

10. Statistical significance vs. clinical importance
IMPORTANT!!!
Statistical significance vs. clinical importance
Negligible clinical difference may be statistically very significant if the number of subjects >>>. e.g., difference in reduction of cholesterol level of 3 mg/dl, n1=n2 = 10,000; p =
Large clinical difference may be statistically non-significant if the no of subjects <<<, e.g. 30% difference in cure rate, if n1 = n2 = 10, p = 0.74

11. Cholesterol level, mg/dl
Clinical importance vs. statistical significance
Cholesterol level, mg/dl
Standard
treatment
x = 220
x = 217
x = 300 mg/dl R
n=10000
Clinical
n=10000
New
treatment
x = 300
mg/dl
t = df = p =
Statistical

12. Clinical importance vs. statistical significance
Cured Died
Standard Rx (100%)
New Rx (70%)
Clinical
Absolute risk reduction = 30%
Fischer exact test: p = 0.211
Statistical

13. Correlation between abdominal circumference and total cholesterol level in middle-aged men
R = 0.22, p = 0.031
Conclusion: There was a significant correlation between abdominal circumference and total cholesterol level in the subjects studied. Measuring abdominal circumference may predict the cholesterol level in middle-aged healthy men.

14. How important is important?
Two percent mortality reduction is probably not important in your clinic
In a community prevention, a simple measure that reduce 2% severe morbidity is probably important.
Low dose aspirin reduces 2% cardiac events in 5 years (without aspirin 400 cardiac events per 10,000, with aspirin 200 cardiac events)
Requires judgment

15. How to infer?
Can the results of the study (in sample) be applied in the accessible or target population?
Hypothesis testing & confidence interval

16. Statistic and Parameter
An observed value drawn from the sample is called a statistic (cf. statistics, the science)
The corresponding value in population is called a parameter
We measure, analyze, etc statistics and translate them as parameters

17. Examples of statistics:
Proportion
Percentage
Mean
Median
Mode
Difference in proportion/mean OR RR
Sensitivity
Specificity
Kappa LR
NNT

18. There are 2 ways in inferring statistic into parameter:
Hypothesis testing  p value
Estimation:  confidence interval (CI)
P Value & CI tell the same concept in different ways

19. P value
Determines the probability that the observed results are caused solely by chance (probability to obtain the observed results if Ho were true)

20. C (60%) 20 (40%)
E (80%) 10 (20%)
X2= ; df = 1; p =
Group Success Failure Total

21. Group Success Failure Total
X2= ; df = 1; p =
If drugs E and C were equally effective, we still can have
the above result (difference of success rate of 20%)
but the probability is small (4.32%)
If drugs E and C were equally effective, the probability
that the result is merely caused by chance is 4.32%
If we define in advance that p<0.05 is significant,
than the result is called statistically significant

22. Similar interpretation applies to ALL hypothesis testing: t-test, Anova, non-parametric tests, Pearson correlation, multivariate tests, etc:
If null-hypothesis null were true, the probability of obtaining the result was ……. (example 0,02 or 2%, etc)

23. Confidence Intervals
Estimate the range of values (parameter) in the population using a statistic in the sample (as point estimate)

24. P S X X X A statistic (point estimate) If the observed result in the
sample is X, what
is the figure in
the population? P X S CI
A statistic (point estimate)

25. Most commonly used CI: CI 90% corresponds to p 0.10
Note:
p value  only for analytical studies
CI  for descriptive and analytical studies

26. How to calculate CI General Formula: CI = p  Z x SE
p = point of estimate, a value drawn from sample (a statistic)
Z = standard normal deviate for , if  = 0.05  Z = 1.96 (~ 95% CI)

27. Example 1 100 FKUI students  60 females (p=0.6)
What is the proportion of females in Indonesian FK students? (assuming FKUI represents FK in Indonesia)

28. Example pq SE(p) = n . 6 x . 4 95 % CI = . 6 ± 1 . 96 100 = . 6 ± 1 .. 6 x . 4 95 % CI = . 6 1 . 96
100 = . 6 1 . 96
X0.5/10 = . 6 . 1 = . 5 ; . 7

29. Example 2: CI of the mean 95% CI = x  1.96 x SEM
100 newborn babies, mean BW = 3000 (SD = 400) grams, what is 95% CI?
95% CI = x  1.96 x SEM

30. Examples 3: CI of difference between proportions (p1-p2)
50 patients with drug A, 30 cured (p1=0.6)
50 patients with drug B, 40 cured (p2=0.8)

31. Example 4: CI for difference between 2 means
Mean systolic BP:
50 smokers = (SD 18.5) mmHg
50 non-smokers = (SD 16.8) mmHg
x1-x2 = 6.0 mmHg
95% CI(x1-x2) = (x1-x2)  1.96 x SE (x1-x2)
SE(x1-x2) = S x V(1/n1 + 1/n2)

32. Example 4: CI for difference between 2 means

33. Other commonly supplied CI
Relative risk (RR)
Odds ratio (OR)
Sensitivity, specificity (Se, Sp)
Likelihood ratio (LR)
Relative risk reduction (RRR)
Number needed to treat (NNT)

34. Suggested CI presentation:
95%CI: 1.5 to 4.5
95%CI: -2.5 to 4.3
95%CI: -12 to -6
Not recommended:
Not recommended:

35. In contrast to CI for proportion, mean, diff
In contrast to CI for proportion, mean, diff. between proportions/means, where the values of CI are symmetrical around point estimate, CI’s for RR, OR, LR, NNT are asymmetrical because the calculations involve logarithm

36. Examples RR = 5.6 (95% CI 1.2 ; 23.7) OR = 12.8 (95% CI 3.6 ; 44,2)
NNT = 12 (95% CI 9 ; 26)

37. If p value <0.05, then 95% CI:
exclude 0 (for difference), because if A=B then A-B = 0  p>0.05
exclude 1 (for ratio), because if A=B then A/B = 1,  p>0.05
For small number of subjects, computer calculated CI may not meet this rule due to correction for continuity automatically done by the computer

38. Concluding remarks
In every study sample should (assumed to) be representative to the population. Otherwise all statistical calculations are not valid
p values (hypothesis testing) gives you the probability that the result in the sample is merely caused by chance, it does not give the magnitude and direction of the difference
Confidence interval (estimation) indicates estimate of value in the population given one result in the sample, it gives the magnitude and direction of the difference

39. Concluding remarks
p value alone tends to equate statistical significance and clinical importance
CI avoids this confusion because it provides estimate of clinical values and exclude statistical significance
whenever applicable, supply CI especially for the main results of study
in critical appraisal of study results, focus should be on CI rather than on p value.

40. 1
The ultimate goal of clinical research is the use of evidence in source population

41. 2
The best non-probabiity sampling is consecuitve sampling

42. 3
P value refers to the probability of getting the observed result when the Ho were false

43. 4
The mean difference of 2 measurements is 20 mmHg, with 95% CI 15 to 25 mmHg. The p value should be “statistically significant”

44. 5
Confidence intervals give more information than p value

45. 6
It is possible to have a study with good internal validity but poor external validity

46. 7
If the odds ratio is 5, then the 95% CI may have values from 3 to 11

47. 8
It is possible to have a significant difference even when the clinical difference is not important, but clinically important difference always statistically significant

48. 9
Appropriate sampling method is mandatory to ensure generalization

49. 10
Clinical epidemiology may include animal studies

50. 11
The more wide the confidence interval, the more precise the result of any study

51. 12
Assessment of clinical importance requires judgment

52. 13
The confidence interval of any measure must include the point estimate

53. 14
Selection of source population usually based on practical reasons

54. 15
Diagnostic test, therapy, etiology, harm, are examples of basic research

55. Critical appraisal (making reading more worthwhile)
What is Critical Appraisal?
Critical appraisal = quality assessment
….process of weighing up evidence to see how useful it is in decision making
.…a process of assessing the validity, importance, and usefulness of evidence
Critical appraisal is about considering, evaluating and interpreting information in a systematic and objective way

56. Critically appraise what you read
Separating the wheat from the chaff
Time is limited – you should aim to quickly stop reading the dross
Others contain useful information mixed with rubbish
Simple checklists enable the useful information to be identified

57. Critical appraisal – Critical thinking
Appraising (evaluating/reviewing) the available evidence to construct clinical reasoning strategies and to make decisions
Finding strengths and limitations of written ‘evidence’
You need to decide what evidence to pay attention to (what is “worthy” of your attention) versus what to ignore

58. Why critically appraise?
Supports sound decision making based on best available evidence
Helps us determine (three R’s):
How rigorous a piece of research is (Valid?)
What the results are telling us (Important?)
How relevant it is to our patient (Applicable?)

59. Why do we need evidence?
Resources should be allocated to things that are EFFECTIVE
The only way of judging effectiveness is EVIDENCE
“In God we trust – all others need evidence”

60. Sources of Evidence Primary sources
Based on experiments and published research
Secondary sources
Systematic reviews
Clinical guidelines
Journals of secondary publication e.g. Evidence Based Medicine

61. “5S” Pyramid of Evidence Resources

62. Levels of evidence Systematic reviews of RCTs/high quality RCTs
Systematic reviews of cohort studies, lower quality RCTs, outcomes research
Systematic reviews of case controls, case control studies
Case Series
Expert opinion
See for complete description

63. Types of Evidence - Question Types
Type of Question
Best Evidence
Health care interventions: treatment, prevention
Quantitative:
Systematic Review of RCTs or RCT
Harm or Etiology
Quantitative: Observational Study - Cohort or Case Control
Prognosis
Quantitative: Observational Study - Cohort, Case Control
Diagnosis or Assessment
Comparison to Gold Standard
Economics
Cost-effectiveness Study
Meaning
Qualitative: case study,

64. Key quality parameters
Validity
Reliability
Importance

65. Validity: internal and external
Internal - Is the study designed in such a way that we can trust the findings?
External - Is the study designed in such a way that I can generalize the findings?
Studies with good internal validity may not have good external validity if the study subjects do not represent population
With poor internal validity, question about external validity is not relevant

66. Reliability
If the study was conducted again, would the results be the same?
Usually interpreted as the accuracy of measurement.

67. Importance
What was the effect size or magnitude of effect? (Would the evidence change your practice?)
Clinical vs. statistical significance.

68. Tools for Critical Appraisal
EBM “simplified” approach:
Are the results valid?
What are the results?
Will the results help me in patient care? V I A

69. Evidence based medicine 5 steps
Formulate question
Evaluate
performance
Efficiently track
down best
available
evidence
Implement
changes in clinical practice
Critically review the
validity and usefulness
of the evidence

70. Check list for medical literature (completeness)
Title
Authors
Abstract: structured? Informative? Abbreviation?
Introduction: length? Relevant references? Target population?
Methods:
Design
Eligibility (inclusion and exclusion) criteria
Sample size, sampling method
Randomization: technique, concealment
Intervention: masking?
Measurement: blinding? - Primary & secondary outcome
Definitions
Analysis

71. Check list for medical literature (contd.)
Results
Baseline characteristics
Main outcome
Secondary outcome
Discussion
General
Strength and weakness
Conclusions
References
Vancouver style
Constant
Acknowledgments
Ethics approval
Conflict of interest

72. What to assess? (in study of cause-effect relationship)
General description
Type of design
Target population, source population, sample
Sampling method
Dependent and independent variables
Main results?

73. What to assess? (in study of cause-effect relationship)
B. Internal validity, non-causal relationship
Influence of bias
Influence of chance
Influence of confounders

74. Bias
What is a bias? A process that tends to produce results that depart systematically from the true values existing in the study population
Types of bias
Sample (subject selection) biases, which may result in the subjects in the sample being unrepresentative of the population which you are interested in
Measurement (detection) biases, which include issues related to how the outcome of interest was measured
Intervention (performance) biases, which involve how the treatment itself was carried out.

75. What to assess? (in study of cause-effect relationship)
C. Internal validity, causal relationship
Temporality (cause precedes effect)
Strength of association (large difference, RR, OR, etc) or small p value or narrow confidence interval
Biological gradient (dose dependence)
Consistency among studies (diff. populations/designs)
Specificity (certain factor results in certain effect)
Coherence (does not conflict with current knowledge)
Biological plausibility: can be explained with current knowledge (at least in part)

76. What to assess? (in study of cause-effect relationship)
D. External validity
Applicable to study subjects
Applicable to source population
Applicable to target population

77. 11 items, each with 3 sections
Can you find this information in the paper?
Is there any problem?
Does this problem threaten the validity?
You can quickly work through any journal article using this format.

78. 11 items What is the research question? What is the study type?
What are the outcome factors and how are they measured?
What are the study factors and how are they measured?
What important confounders are considered?
What are the sampling frame and sampling method?
In an exp., how were the subjects assigned to groups? In a longitudinal study, how many reached final follow-up? In a case control study, are the controls appropriate? (etc)
Are statistical tests considered?
Are the results clinically/socially significant (important)?
Is the study ethical?
What conclusions did the authors reach?
These are the main items on the worksheet, and can be applied to most journal articles

79. 1. What is the research question?
Any problem?
Is it concerned with the impact of an intervention, causality or determining the magnitude of a health problem?
(Does this problem threaten the validity?)
Is it a well stated research question/hypothesis?

80. 2. What is the study type? (Any problem?)
Is the study type appropriate to the research question?
(Does this problem threaten the validity?)
If not, how useful are the results produced by this type of study?

81. 3. What are the outcome factors and how are they measured?
(Any problem?)
a) are all relevant outcomes assessed
b) is there measurement error?
(Does this problem threaten the validity?)
a) how important are omitted outcomes
b) is measurement error an important source of bias?

82. 4. What are the study factors and how are the measured?
(Any problem?)
Is there measurement error?
(Does this problem threaten the validity?)
Is measurement error an important source of bias?

83. 5. What important potential confounders are considered?
(Any problem?)
Are potential confounders examined and controlled for?
(Does this problem threaten the validity?)
Is confounding an important source of bias?

84. 6. What are the sampling frame and sampling method?
(Any problem?)
Is there selection bias?
(Does this problem threaten the validity?)
Does this threaten the external validity of the study?
We have now considered the major sources of bias – selection, measurement and confounding.

85. 7. Questions of internal validity
(Any problem?)
Experimental: how were the subjects assigned to groups?
Longitudinal study, how many reached follow-up?
Case control study, are the controls appropriate?
Note: other issues of relevance to internal validity are considered under the other headings in this critical appraisal system. You can add your own questions, and also design your own questions for other study types such as cross sectional studies and systematic reviews
(Does this problem threaten the validity?)
Does this threaten the internal validity of the study?
It is difficult to include all possible sources of bias that might threaten the internal validity of a study, and you are welcome to add your own favourites. The ones included in this side are those that might be considered as citiical for these types of study

86. 8. Are statistical tests considered?
(Any problem?)
Were the tests appropriate for the data?
Are confidence intervals given?
Is the power given if a null result?
In a trial, are results presented as absolute risk reduction as well as relative risk reduction?
(Does this problem threaten the validity?)
If not, how useful are the results?

87. 9. Are the results clinically/socially significant?
(Any problem?)
Was the sample size adequate to detect a clinically/socially significant result?
Are the results presented in a way to help in health policy decisions?
(Does this problem threaten the validity?)
Is the study useful?
The difference between statistical significance and a result that is of use is an important distinction to make

88. 10. Are ethical issues considered?
(Any problem?)
Does the paper indicate ethics approval?
Can you identify potential ethical issues?
(Does this problem threaten the validity?
Are the results or their application compromised?
There are a number of ethical issues to consider in both the conduct of research and the application of the results

89. 11. What conclusions did the authors reach about the study question?
(Any problem?)
Do the results apply to the population in which you are interested?
(Does this problem threaten the validity?
Will you use the results of the study?
The final issue is whether the study has passed your critical appraisal in terms of your decision to use the results in your own practice, teaching or further research

90. Appraisal Tools Systematic Reviews Randomised Controlled Trials
Tools from the Critical Appraisal Skills Programme (CASP)
Systematic Reviews
Randomised Controlled Trials
Qualitative Research Studies
Cohort Studies
Case-Control Studies
Diagnostic Test Studies
Economic Evaluation Studies
Available at:

91. Study Designs Recap Effectiveness of Therapy Risk Factors / Prognosis
Diagnosis
Attitudes & Beliefs
Randomised Controlled Trial
Cohort Study
Survey using gold standard
Qualitative (Interviews, Observations, etc)

92. Critical appraisal - Validity Methods - Importance Results
- Applicability
Methods
Results
Discussion

93. Thanks