1. March 20141 Back to Basics, 2014 POPULATION HEALTH (1): Epidemiology Methods, Critical Appraisal, Biostatistical Methods N. Birkett, MD Epidemiology & Community Medicine Other resources available on Individual & Population Health web siteIndividual & Population Health web site

2. March 20142 THE PLAN (1) Session 1 (March 18, 1300-1700) –Diagnostic tests Sensitivity, specificity, validity, PPV –Critical Appraisal –Intro to Biostatistics –Brief overview of epidemiological research methods

3. March 20143 THE PLAN (2) Aim to spend about 2.5 hours on lectures –Review MCQs in remaining time A 10 minute break about half-way through You can interrupt for questions, etc. if things aren’t clear. –Goal is to help you, not to cover a fixed curriculum.

4. March 20144 INVESTIGATIONS (1) 78.2 –Determine the reliability and predictive value of common investigations –Applicable to both screening and diagnostic tests.

5. March 20145 Reliability = reproducibility. Does it produce the same result every time? Related to chance error Averages out in the long run –In patient care you hope to do a test only once –Therefore, you need a reliable test

6. March 20146 Validity Whether it measures what it purports to measure in long run –is a disease present (or absent) Normally use criterion validity, comparing test results to a gold standard Link to SIM web on validityvalidity

7. March 20147 Reliability and Validity: the metaphor of target shooting. Here, reliability is represented by consistency, and validity by aim Reliability Low High Low Validity High

8. March 20148 Test Properties (1) DiseasedNot diseased Test +ve90595 Test -ve1095105 100 200 True positivesFalse positives False negativesTrue negatives

9. March 20149 Test Properties (2) DiseasedNot diseased Test +ve90595 Test -ve1095105 100 200 Sensitivity = 0.90Specificity = 0.95

10. March 201410 2x2 Table for Testing a Test Gold standardDisease PresentAbsent Test Positivea (TP)b (FP) Test Negativec (FN)d (TN) SensitivitySpecificity = a/(a+c) = d/(b+d)

11. March 201411 Test Properties (6) Sensitivity =Pr(test positive in a person with disease) Specificity =Pr(test negative in a person without disease) Range: 0 to 1 –> 0.9:Excellent –0.8-0.9:Not bad –0.7-0.8:So-so –< 0.7:Poor

12. March 201412 Test Properties (7) Sensitivity and Specificity –Values depend on cutoff point between normal/abnormal –Generally, high sensitivity is associated with low specificity and vice-versa. –Not affected by prevalence, if ‘case-mix’ is constant Do you want a test to have high sensitivity or high specificity? –Depends on cost of ‘false positive’ and ‘false negative’ cases –PKU – one false negative is a disaster –Ottawa Ankle Rules: insisted on sensitivity of 1.00

13. March 201413 Test Properties (8) Sens/Spec not directly useful to clinician, who knows only the test result Patients don’t ask: –“If I’ve got the disease, how likely is that the test will be positive?” They ask: –“My test is positive. Does that mean I have the disease?” → Predictive values.

14. March 201414 Predictive Values Based on rows, not columns –PPV = a/(a+b); interprets positive test –NPV = d/(c+d); interprets negative test Depend upon prevalence of disease, so must be determined for each clinical setting Immediately useful to clinician: they provide the probability that the patient has the disease

15. March 201415 Test Properties (9) DiseasedNot diseased Test +ve90595 Test -ve1095105 100 200 PPV = 0.95 NPV = 0.90

16. March 201416 2x2 Table for Testing a Test Gold standard Disease Present Absent Test +a (TP) b (FP) PPV = a/(a+b) Test -c (FN) d (TN) NPV= d/(c+d) a+c b+dN

17. March 201417 Prevalence of Disease Prevalence: the probability that someone has a disease, condition at a point in time. For diagnostic tests: –Is your best guess about the probability that the patient has the disease, before you do the test Also known as Pretest Probability of Disease (a+c)/N in 2x2 table Is closely related to Pre-test odds of disease: (a+c)/(b+d)

18. March 201418 Test Properties (10) DiseasedNot diseased Test +veaba+b Test -vecdc+d a+cb+da+b+c+d =N Prevalence odds Prevalence proportion

19. March 201419 Prevalence and Predictive Values Predictive values of a test are dependent on the pre-test prevalence of the disease –Tertiary hospitals see more pathology then FP’s Their positive tests are more often true positives. Most tests are developed and studied in tertiary care settings. How do you determine how useful a test is in a different patient setting?

20. March 201420 Prevalence and Predictive Values Process is often called ‘calibrating’a test –Relies on the stability of sensitivity & specificity across populations. –Allows us to estimate what the PPV and NPOV would be in a new population.

21. March 201421 Methods for Calibrating a Test Four methods can be used: –Apply definitive test to a consecutive series of patients from the new population rarely feasible, especially during the LMCCs –Hypothetical table Assume the new population has 10,000 people Fill in the cells based on the prevalence, sensitivity and specificity [My recommended way]

22. March 201422 Methods for Calibrating a Test Four methods can be used (cnt’d): –Bayes’s Theorem (Likelihood Ratio) –Nomogram only useful if you have access to the nomogram You need to be able to do one of the last 3. The easiest is using a hypothetical table.

23. March 201423 Calibration by hypothetical table Fill cells in following order: “Truth” DiseaseDiseaseTotal PV PresentAbsent Test Pos 4 th 7 th 8 th 10 th Test Neg 5 th 6 th 9 th 11 th Total 2 nd 3 rd 10,000 (1 st )

24. March 201424 Test Properties (11) DiseasedNot diseased Test +ve450 25475 Test -ve 50475525 500 1,000 Tertiary care: research study. Prev=0.5 PPV = 0.89 Sens = 0.90Spec = 0.95

25. March 201425 Test Properties (12) DiseasedNot diseased Test +ve Test -ve 10,000 Primary care: Prev=0.01 PPV = 0.1538 9,900 90 10 100 495 9,405 585 9,415 Sens = 0.90Spec = 0.95

26. March 201426 Calibration by Bayes’ Theorem You don’t need to learn Bayes’ theorem Instead, work with the Likelihood Ratio (+ve)Likelihood Ratio –Equivalent process exists for Likelihood Ratio (–ve), but we shall not calculate it here Consider the following table (from a research study) –How do the ‘odds’ of having the disease change once you get a positive test result?

27. March 201427 Test Properties (13) DiseasedNot diseased Test +ve 90595 Test - ve 1095105 100 200 Pre-test odds = 1.00 Post-test odds (+ve) = 18.0 Odds (after +ve test) are 18-times higher than the odds before you had the test. This is the LIKELIHOOD RATIO.

28. March 201428 Calibration by Bayes’s Theorem Likelihood ratios are related to sens & spec –LR(+) = Sometime given as the definition of the LR(+) LR(+) is fixed across populations just like sensitivity & specificity. –Bigger is better.

29. March 201429 Calibration by Bayes’s Theorem How does this help? Remember: –Post-test odds(+) = pretest odds * LR(+) –And, the LR(+) is ‘fixed’ across populations To ‘calibrate’ your test for a new population: –Get the LR(+) value from the reference source –Estimate the pre-test odds for your population –Compute the post-test odds –Convert to post-test probability to get PPV

30. March 201430 Converting between odds & probabilities if prevalence = 0.20, then pre-test odds = = 0.25 (1 to 4) if post-test odds = 0.25, then PPV = = 0.20

31. March 201431 Example of Bayes' Theorem (sens 90%, spec 95%, ‘new’ prevalence 1%) Compare to the ‘hypothetical table’ method (PPV=15.38%)

32. March 201432 Calibration with Nomogram Graphical approach which avoids arithmetic Scaled to work directly with probabilities –no need to convert to odds Draw line from pretest probability (=prevalence) through likelihood ratio – extend to estimate posttest probabilities Only useful if someone gives you the nomogram!

33. April 201133 Example of Nomogram (pretest probability 1%, LR+ 18, LR– 0.105) Pretest Prob. LR Posttest Prob. 1% 18.105 15% 0.01% March 201433

34. March 201434 Are sens & spec really constant? Generally, assumed to be constant. BUT….. Sensitivity and specificity usually vary with case mix (severity of disease) –May vary with age and sex Therefore, you can use sensitivity and specificity only if they were determined on patients similar to your own Risk of spectrum bias –populations may come from different points along the spectrum of disease

35. Cautionary Tale #1: Data Sources March 201435 The Government is extremely fond of amassing great quantities of statistics. These are raised to the nth degree, the cube roots are extracted, and the results are arranged into elaborate and impressive displays. What must be kept ever in mind, however, is that in every case, the figures are first put down by a village watchman, and he puts down anything he damn well pleases! Sir Josiah Stamp, Her Majesty’s Collector of Internal Revenue.

36. March 201436 78.2: CRITICAL APPRAISAL (1) “Evaluate scientific literature in order to critically assess the benefits and risks of current and proposed methods of investigation, treatment and prevention of illness” UTMCCQE does not present hierarchy of evidence –as used by Task Force on Preventive Health Services

37. March 201437 Hierarchy of evidence (lowest to highest quality, approximately) Systematic reviews Experimental (Randomized) Quasi-experimental Prospective Cohort Historical Cohort Case-Control Cross-sectional Ecological (for individual-level exposures) Case report/series Expert opinion } similar/identical

38. Cautionary Tale #2: Analysis March 201438 Consider a precise number: the normal body temperature of 98.6°F. Recent investigations involving millions of measurements have shown that this number is wrong: normal body temperature is actually 98.2°F. The fault lies not with the original measurements - they were averaged and sensibly rounded to the nearest degree: 37°C. When this was converted to Fahrenheit, however, the rounding was forgotten and 98.6 was taken as accurate to the nearest tenth of a degree.

39. March 201439 BIOSTATISTICS Core concepts (1) Sample: –A group of people, animals, etc. which is used to represent a larger ‘target’ population. Best is a random sample Most common is a convenience sample. –Subject to strong risk of bias. Sample size: –the number of units in the sample Much of statistics concerns how samples relate to the population or to each other.

40. March 201440 BIOSTATISTICS Core concepts (2) Mean: –average value. Measures the ‘centre’ of the data. Will be roughly in the middle. Median: –The middle value: 50% above and 50% below. Used when data is skewed. Variance: –A measure of how spread out the data are. –Defined by subtracting the mean from each observation, squaring, adding them all up and dividing by the number of observations.

41. March 201441

42. March 201442 BIOSTATISTICS Core concepts (2) Standard deviation: –square root of the variance.

43. March 201443 BIOSTATISTICS Core concepts (3) Standard error (of the mean): –Standard deviation looks at the variation of the data in individuals –We usually study samples. Select 10 people measure BMI take the group average –Repeat many times. Each time, we get a mean of the sample –What is the distribution of these means? Will be ‘normal’, ‘Gaussian’ or ‘Bell curve’ –Mean of the means same as population mean –Variance of the means is smaller than population variance

44. March 201444 BIOSTATISTICS Core concepts (4) Standard error (of the mean): Confidence Interval: –A range of numbers which tells us where the correct answer lies. For a 95% confidence interval, we are 95% sure that the true value lies inside the interval. –Usually computed as: mean ± 2 SE

45. March 201445 Example of Confidence Interval If sample mean is 80, standard deviation is 20, and sample size is 25 then: –We can be 95% confident that the true mean lies within the range: 80 ± (2*4) = (72, 88).

46. March 201446 Example of Confidence Interval If the sample size were 100, then –95% confidence interval is: 80 ± (2*2) = (76, 84). –More precise.

47. March 201447 Core concepts (4) Random Variation (chance): –every time we measure anything, errors will occur. –Any sample will include people with values different from the mean, just by chance. –These are random factors which affect the precision (SD) of our data but not the validity. –Statistics and bigger sample sizes can help here.

48. March 201448 Core concepts (5) Bias: –A systematic factor which causes two groups to differ. A study uses a two section measuring scale for height which was incorrectly assembled (with a 1” gap between the upper and lower section). Under-estimates height by 1” (a bias). –Bigger numbers and statistics don’t help much; you need good design instead.

49. March 201449 BIOSTATISTICS Inferential Statistics Draws inferences about populations, based on samples from those populations. –Inferences are valid only if samples are representative (to avoid bias). Polls, surveys, etc. use inferential statistics to infer what the population think based on talking to a few people. –1,000 people can represent all of Canada RCTs use them to infer treatment effects, etc. 95% confidence intervals are a very common way to present these results.

50. An experiment (1) Here is a ‘fair’ coin I will toss it to generate some data (heads or tails) –[Write the sequence on the board] March 201450

51. An experiment (2) At some point, you get suspicious –the number of ‘heads’ in a row exceeds what is reasonable. This is the core of hypothesis testing March 201451

52. An experiment (3) Start with a theory –Null Hypothesis My coin is ‘fair’ Generate some data Check to see if the data is consistent with the theory. –if the data is ‘unlikely’, then reject the theory or null hypothesis. Statistics just puts a mathematical overlay on top of this intuitive approach March 201452

53. March 201453 Hypothesis Testing (1) Used to compare two or more groups. –We first assume that the two groups have the same outcome results. null hypothesis (H 0 ) –Generate some data –From the data, compute some number (a statistic) Under this null hypothesis (H 0 ), this should be ‘0’. –Compare the value I get to ‘0’. If it is ‘too large’, we can conclude that our assumption (null hypothesis) is unlikely to be true –reject the null hypothesis

54. March 201454 Hypothesis Testing (2) Quantity the extent of our discomfort with the statistic through the p-value. –If the null hypothesis were true, how likely is it that our statistic would be as big as we saw (or bigger). Reject H 0 if the p-value is ‘too small’ What is ‘too small’? –arbitrary. –tradition sets it at < 0.05

55. March 201455 Example of significance test Is there an association between sex and smoking: –35 of 100 men smoke but only 20 of 100 women smoke Usually present data in a 2X2 table: SmokeDon’t smoke Men3565100 Women2080100 55145200

56. March 201456 Example of significance test Calculate the chi-square (the statistic) – = 5.64. –If there is no effect of sex on smoking (the null hypothesis), a chi-square value as large as 5.64 would occur only 1.8% of the time. P=0.018 –Instead of computing the p-value, could compare your statistic to the ‘critical value’ The value of the Chi-square which gives p=0.05 is 3.84 Since 5.64 > 3.84, we conclude that p<0.05

57. March 201457 Hypothesis Testing (3) Common methods used are: –T-test –Z-test –Chi-square test –ANOVA Approach can be extended through the use of regression models –Linear regression Toronto notes are wrong in saying this relates 2 variables. It can relate many independent variables to one dependent variable. –Logistic regression –Cox models

58. March 201458 Hypothesis Testing (4) Once you select a method for hypothesis testing, interpretation involves: –Type 1 error (alpha) –Type 2 error (beta) –P-value Essentially the alpha value –Power Related to type 2 error (Beta)

59. March 201459 Hypothesis testing (5) No effectEffect No effectNo errorType 2 error (β) EffectType 1 error (α) No error Actual Situation Results of Stats Analysis

60. March 201460 Hypothesis Testing (7) Statistical Power: –‘Easy’ to show that a drug increases survival by 10 times –‘Hard’ to show that a drug increase survival by 1.2 times –More likely to ‘miss’ the small effect than the large effect –Statistical Power is: The chance you will find a difference between groups when there really is a difference (of a given amount). Basically, this is 1-β –Power depends on how big a difference you consider to be important

61. March 201461 How to improve your power? Increase sample size Improve precision of the measurement tools used (reduces standard deviation) Use better statistical methods Use better designs Reduce bias

62. Cautionary Tale #3: Anecdotes March 201462 Laboratory and anecdotal clinical evidence suggest that some common non-antineoplastic drugs may affect the course of cancer. The authors present two cases that appear to be consistent with such a possibility: that of a 63-year-old woman in whom a high- grade angiosarcoma of the forehead improved after discontinuation of lithium therapy and then progressed rapidly when treatment with carbamezepine was started, and that of a 74-year-old woman with metastatic adenocarcinoma of the colon which regressed when self- treatment with a non-prescription decongestant preparation containing antihistamine was discontinued. The authors suggest...... ‘that consideration be given to discontinuing all nonessential medications for patients with cancer.’

63. March 201463 Epidemiology overview Key study designs to examine –Case-control –Cohort –Randomized Controlled Trial (RCT) Confounding Relative Risks/odds ratios –All ratio measures have the same interpretation 1.0 = no effect < 1.0  protective effect > 1.0  increased risk –Values over 2.0 are of strong interest

64. March 201464 The Epidemiological Triad Host Agent Environment

65. March 201465 Terminology Prevalence: –The probability that a person has the outcome of interest today. Relates to existing cases of disease. Useful for measuring burden of illness. Incidence: –The probability (chance) that someone without the outcome will develop it over a fixed period of time. Relates to new cases of disease. Useful for studying causes of illness.

66. March 201466 Prevalence On July 1, 2014, 140 graduates from the U. of O. medical school start working as interns. Of this group, 100 had insomnia the night before. Therefore, the prevalence of insomnia is: 100/140 = 0.72 = 72%

67. March 201467 Incidence Proportion (risk) On July 1, 2014, 140 graduates from the U. of O. medical school start working as interns. Over the next year, 30 develop a stomach ulcer. Therefore, the incidence proportion (risk) of an ulcer in the first year post-graduation is: 30/140 = 0.21 = 214/1,000 over 1 yr

68. March 201468 Incidence Rate (1) Incidence rate is the ‘speed’ with which people get ill. Everyone dies (eventually). It is better to die later  death rate is lower. Compute with person-time denominator: PT = # people * duration of follow-up

69. March 201469 Incidence rate (2) 140 U. of O. medical students were followed during their residency –50 did 2 years of residency –90 did 4 years of residency –Person-time = 50 * 2 + 90 * 4 = 460 PY’s During follow-up, 30 developed ‘stress’. Incidence rate of stress is:

70. March 201470 Prevalence & incidence As long as conditions are ‘stable’ and disease is fairly rare, we have this relationship: That is, Prevalence ≈ Incidence rate * average disease duration

71. March 201471 Cohort study (1) Select non-diseased subjects based on their exposure status Main method used: Select a group of people with the exposure of interest Select a group of people without the exposure Can also simply select a group of people without the disease and study a range of exposures. Follow the group to determine what happens to them. Compare the incidence of the disease in exposed and unexposed people If exposure increases risk, incidence will be higher in exposed subjects than unexposed subjects Compute a relative risk. Framingham Study is standard example.

72. March 201472 Exposed group Unexposed group No disease Disease No disease Disease time Study beginsOutcomes

73. March 201473 Cohort study (2) YES NO YES a b a+b NO c d c+d a+c b+d N Disease Exp RISK RATIO Risk in exposed: = Risk in Non-exposed= If exposure increases risk, you would expect to be larger than. How much larger can be assessed by the ratio of one to the other:

74. March 201474 Cohort study (3) YES NO Yes 42 80 122 No 43302 345 85 382 467 Death Exposure Risk in exposed: = 42/122 = 0.344 Risk in Non-exposed= 43/345 = 0.125

75. March 201475 Cohort study (4) Historical cohort study Recruit subjects sometime in the past Follow-up to the present Usually use administrative records Can continue to follow into the future Example: cancer in Gulf War Vets Identify soldiers deployed to Gulf in 1991 Identify soldiers not deployed to Gulf in 1991 Compare development of cancer from 1991 to 2010

76. March 201476 Case-control study (1) Select subjects based on their final outcome. –Select a group of people with the outcome/disease (cases) –Select a group of people without the outcome (controls) –Ask them about past exposures –Compare the frequency of exposure in the two groups If exposure increases risk, the odds of exposure in the case should be higher than the odds in the controls –Compute an Odds Ratio –Under many conditions, OR ≈ RR

77. March 201477 Disease (cases) No disease (controls) Exposed Unexposed Exposed Unexposed The study begins by selecting subjects based on Review records Review records

78. March 201478 Case-control study (2) YES NO YES a b a+b NO c d c+d a+c b+d N Disease? Exp? ODDs RATIO Odds of exposure in cases = Odds of exposure in controls = If exposure increases risk, you would to find more exposed cases than exposed controls. That is, the odds of exposure for cases would be higher This can be assessed by the ratio of one to the other:

79. March 201479 Yes No Yes 42 18 No 43 67 85 85 Exposure Odds of exp in cases: = 42/43 = 0.977 Odds of exp in controls: = 18/67 = 0.269 Case-control study (3) Death

80. March 201480 Randomized Controlled Trials Basically a cohort study where the researcher decides which exposure (treatment) the subject get. –Recruit a group of people meeting pre-specified eligibility criteria. –Randomly assign some subjects (usually 50% of them) to get the control treatment and the rest to get the experimental treatment. –Follow-up the subjects to determine the risk of the outcome in both groups. –Compute a relative risk or otherwise compare the groups.

81. March 201481 Randomized Controlled Trials (2) Some key design features –Allocation concealment –Blinding (masking) Patient Treatment team Outcome assessor Statistician –Monitoring committee Two key problems –Contamination Control group gets the new treatment –Co-intervention Some people get treatments other than those under study

82. March 201482 Randomized Controlled Trials: Analysis Outcome is often an adverse event –RR is expected to be <1 Absolute risk reduction

83. Number needed to treat, (to prevent one adverse event) March 201483 Randomized Controlled Trials: Analysis Relative risk reduction

84. March 201484 RCT – Example of Analysis Asthma No TotalIncid attackattack Treatment 15 35 50.30 Control 25 25 50.50 Relative Risk = 0.30/0.50 = 0.60 Absolute Risk Reduction = 0.50-0.30 = 0.20 Relative Risk Reduction = 0.20/0.50 = 40% Number Needed to Treat = 1/0.20 = 5

85. March 201485 Confounding Does alcohol drinking cause oral cancer? –Do a case-control study –OR=3.4 (95% CI: 2.1-4.8). BUT, the effect of alcohol is ‘mixed up’ with the effect of smoking. –Smoking causes mouth cancer –Heavy drinkers tend to be heavy smokers. –Smoking is not part of causal pathway for alcohol.

86. March 201486 The Confounding Triangle Alcohol Oral cancer Smoking Causal Association

87. March 201487 Confounding The effect of this third factor ‘confounds’ the relationship we are interested in. –Produces a biased results. –Can make result more or less strong than it really is A confounder is an extraneous factor which is associated with both exposure and outcome, and is not an intermediate step in causal pathway Proper statistical analysis must adjust for the confounder. We do a statistical adjustment (logistic regression is most common): –OR=1.3 (95% CI: 0.92-1.83)

88. March 201488 The Confounding Triangle Exposure Outcome Confounder Causal Association

89. March 201489 Standardization An method of adjusting for confounding (usually used for differences in age between two populations) Refers observed events to a standard population, producing hypothetical values Direct: –yields age-standardized rate (ASMR) Indirect: –yields standardized mortality ratio (SMR) You don’t need to know how to do this Nearly always used when presenting population rates and trends.

90. March 201490 Measures of Population Health Mortality data Mortality rates –crude Overall all-cause mortality rate –specific mortality rate for a specific group (men), disease (lung cancer), etc. –standardized Mortality rate adjustment to take account of the aging population

91. March 201491 Mortality data Life expectancy: –average age at death if current mortality rates continue. Derived from a life table. Potential Years of Life Lost (PYLL): –subtract age at death from some “acceptable” age of death. –Sum up over a group estimates ‘potential’ lost due to early death Places more emphasis on causes that kill at younger ages.

92. March 201492 Impact of different causes of death in Canada 2001: Mortality rates and PYLL Source: Statistics Canada

93. March 201493 Measures of population health Mortality is a ‘crude’ measure of population health Need to consider –morbidity –quality of life –disability –and so on. Many other measures have been developed Quality Adjusted Life Years (QALYs) –Years lived are weighted according to quality of life, disability, etc. Two ‘classes’ of these types of measures: –Health expectancies point up from zero –Health gaps point down from ideal

94. March 201494 Attributable Risks (1) Would like to know the amount of a disease which might be prevented if we eliminate a risk Tricky area since there are several measures with similar names. –Attributable risk –Attributable fraction –Population Attributable Risk –and so on Gives an upper limit on amount of preventable disease. Meaningful only if association is causal.

95. March 201495 Attributable Risks (2) Two main targets for these measures The amount of disease due to exposure in the exposed subjects. The same as the risk difference. The proportion of risk attributed to the exposure in the general population –depends on Risk due to exposure How common the exposure is.

96. March 201496 Attributable risks (3) ExpUnexp Risk Difference or Attributable Risk I exp I unexp RD = AR = I exp - I unexp

97. March 201497 Attributable risks (4) ExpUnexp Population Attributable Risk I exp I unexp I pop Population

98. March 201498