1. Hunting side effects and explaining them: should we reverse evidence hierarchies upside down? Barbara Osimani Catholic University of Milan Evidence and Causality in the Sciences Canterbury, 5-7 September 2012

2. Hunting side effects and explaining them: should we reverse evidence hierarchies upside down? In their comparative analysis of RCTs and observational studies, Papanikolau et al. (2006) assert: “it may be unfair to invoke bias and confounding to discredit observational studies as a source of evidence on harms” (p. 640, my emphasis).

3. Hunting side effects and explaining them: should we reverse evidence hierarchies upside down? Recent contributions by philosophers and health scientists have acknowledged the role of so called "lower level" evidence as a valid source of information contributory to assessing the risk profile of medications both on theoretical (Aronson and Hauben, 2006; Howick et al. 2009) and on empirical grounds (Benson and Hartz, 2000; Golder et al. 2011). Nevertheless current practices have difficulty in assigning a precise epistemic status to this kind of evidence and in amalgamating it with standard methods of hypothesis testing.

4. Topics Recent proposals to amend evidence hierarchies especially Vandenbroucke’s suggestion to reverse hierarchies when addressing the issue of risk discovery and assessment; The requirement of total evidence as applied to this specific context: how it is and how it should be interpreted (vs. lexicographic rule implicit in ranking); I will illustrate some of these points by reference to the recent debate on the causal association between paracetamol and increased asthma prevalence/exhacerbation.

5. Hierarchy reversal for risk assessment Vandenbroucke J.P. (2008) Observational Research, Randomised Trials, and Two Views of Medical Science. Plos Medicine, 5 (3): 339-43 Hierarchy of study designs for intended effects of therapy Hierarchy of study designs for discovery and explanation i. Randomised controlled trials i. Anecdotal: case report and series, findings in data, literature ii. Prospective follow-up studies ii. Case-control studies iii. Retrospective follow-up studies iv. Case-control studies iv. Prospective follow-up studies v. Anecdotal: case report and seriesv. Randomised controlled trials

6. Vandenbroucke’s defence of hierarchy reversal (I) 1.Methodological point: Observational studies concerning adverse reactions will not suffer from confounding in the same way asobservational studies for intended effects do. selection bias is less likely to affect observational studies with respect to adverse reactions. This because unintended effects, qua unintended, are not known in advance, and thus also not known by the drug prescriber, who cannot take them into consideration and thus bias treatment allocation. Ignorance of possible effect = “natural masking”

7. Vandenbroucke’s defence of hierarchy reversal (II) 2. Epistemological point: Context of discovery vs. context of evaluation: Discovery is focused on explanation and hypothesis generation; Evaluation instead on hypothesis testing/confirmation. And research methods differ in the opportunities they offer with respect to either of these goals.

8. Vandenbroucke’s defence of hierarchy reversal (III) Vandenbroucke (2008) formalizes the contrast between the context of evaluation and the context of discovery in terms of different priors assigned to hypotheses of benefits and of adverse reactions. High priors for intended effects Low priors for unintended ones

9. Vandenbroucke’s defence of hierarchy reversal (III) 1.It is the higher priors which make the results more robust, not the method (Vandenbroucke, 2008: 16-17). 2.The reason why we accept uncertain results for risks rather than for benefits is that evaluation and discovery studies are associated with different loss functions: 1.evaluation is related to the approval of health technologies and is required to assure stakeholders about their efficacy and safety, 2.whereas discovery is more related to the context of research for its own sake, which might explain why certain study designs are preferred to others in different circumstances.

10. Vandenbroucke’s defence of hierarchy reversal (III) 1.Priors are quickly swamped by data 2.Stakes are not lower for detecting risks than for testing the drug benefit: adverse drug reactions might be so severe as to reverse the safety profile of the drug and determine its withdrawal.

11. Prior knowledge about drug’s general capacity to produce unintended adverse reactions The acceptability of anecdotal evidence or of uncontrolled studies for assessing risk has to do with a high prior about the general capacity of the drug to bring about side-effects. Whereas there is total ignorance as to some specific side effects which might be possibly caused by the drug, still there is almost certainty about the fact that the drug will indeed cause side-effects beyond the ones already detected in the pre-marketing phase. This high prior derives from historical knowledge and past experience with pharmaceutical products and is also strongly reflected in the regulation which introduced the notion of “development (or potential) risk”, the pharmacosurveillance system, and the precautionary principle.

12. Vandenbroucke’s defence of hierarchy reversal on abductive grounds “For discoveries, the original case reports, lab observations, data analysis, or juxtaposition in literature may be so convincing that they stand by themselves, either because of the magnitude of the effect or because the new explanation suddenly and convincingly makes the new finding fall into place with previous unexplained data or previous ideas”. (Vandenbroucke, 2008: 6).

13. Case Study: hypothesis of causal connection between paracetamol and asthma Asthma increase in the United States and in Western countries in the last 3 decades: up to a 75% increase among adults and to a 160% among children in the same period. (Burr et al., 1989; Eneli et al., 2005, Ninan and Russel, 1992; Mannino et al., 1998, 2002, Seaton et al. 1994).

14. Explanatory hypotheses for asthma epidemic 1)increased exposure to outdoor and indoor pollutants; 2)decreased exposure to bacteria and childhood illnesses during infancy (the “hygiene hypothesis”); 3)increased obesity incidence and prevalence; 4)changes in diet and oxidant intake; 5)cytokine imbalance as a reaction to environmental allergens in early childhood leading to lifelong T-helper type 2 (allergic) dominance over T-helper type 1 (nonallergic) reactions, thus increasing the risk for atopic disease Eneli et al., 2005; Seaton et al. 1994, Shaheen et al. 2000.

15. How suspicion fell upon paracetamol Varner and colleagues (1998) detected a precise correspondence between increase of asthma incidence and increased paracetamol use as a substitute for aspirin (following the recognition of an association between aspirin and Reye’s syndrome). Varner and colleagues (1998 The trend levelled off in the 1990s, i.e. at a time when paracetamol had already become one of the most widespread analgesics. Varner and colleagues tentative explanation was however that asthma increase was due to aspirin avoidance, for the reason that aspirin may protect from asthma through inhibition of prostaglandins. However, this hypothesis was soon discounted on grounds that, if this had been the case, then one should have observed a decrease of asthma incidence when aspirin was first introduced (Shaheen et al. 2000). Thus the suspicion finally fell upon paracetamol itself and subsequent investigations explicitly aimed to examine the hypothesis of causal connection between paracetamol and asthma.

16. Evidence for causal association between paracetamol and asthma “Many observations suggest that the epidemiologic association between acetaminophen and asthma is causative: 1)consistency of the association across geography, culture and age;consistency of the association across geography, culture and age 2)strength of the association (comparative studies); 3)the dose-response relationship between paracetamol exposure and asthma; 4)the coincidence of the timing of increasing asthma prevalence and increasing paracetamol use; 5)the relationship between per-capita sales of paracetamol and asthma morbidity across countries; 6)our inability to identify any other abrupt environmental change that could explain this increase in asthma morbidity; 7)plausible mechanism: glutathione depletion in airway mucosa caused by paracetamol”. McBride JT (2011) The Association of Acetaminophen and Asthma Prevalence and Severity, Pediatrics, 128 (6).

17. Consistency of the association across geography, culture and age (I)

18. Consistency of the association across geography, culture and age (II)

19. Consistency of the association across geography, culture and age (III)

20. Evidence for causal association between paracetamol and asthma “Many observations suggest that the epidemiologic association between acetaminophen and asthma is causative: 1)consistency of the association across geography, culture and age; 2)strength of the association (comparative studies);comparative studies 3)the dose-response relationship between paracetamol exposure and asthma; 4)the coincidence of the timing of increasing asthma prevalence and increasing paracetamol use; 5)our inability to identify any other abrupt environmental change that could explain this increase in asthma morbidity; 6)the relationship between per-capita sales of paracetamol and asthma morbidity cross countries; 7)plausible mechanism: glutathione depletion in airway mucosa caused by paracetamol”. McBride JT (2011) The Association of Acetaminophen and Asthma Prevalence and Severity, Pediatrics, 128 (6).

21. Comparative studies

22. Evidence for causal association between paracetamol and asthma “Many observations suggest that the epidemiologic association between acetaminophen and asthma is causative: 1)consistency of the association across geography, culture and age; 2)strength of the association (comparative studies); 3)the dose-response relationship between paracetamol exposure and asthma; 4)the coincidence of the timing of increasing asthma prevalence and increasing paracetamol use; 5)the relationship between per-capita sales of paracetamol and asthma morbidity and across countries; 6)our inability to identify any other abrupt environmental change that could explain this increase in asthma morbidity; 7)plausible mechanism: glutathione depletion in airway mucosa caused by paracetamol”. McBride JT (2011) The Association of Acetaminophen and Asthma Prevalence and Severity, Pediatrics, 128 (6).

23. Coincidence of the timing of increasing asthma prevalence and increasing paracetamol use (4) “Although other changes in the environment have been suggested that might explain an increase in childhood asthma, none so easily explains the rapid increase in asthma in the 1980s and the subsequent levelling off of asthma prevalence over the last 15 years. Furthermore, the prevalence of childhood wheezing in 36 countries around the world is predicted by each country’s per- capita sales of paracetamol”. McBride (2011) (5)

24. Relationship between per-capita sales of paracetamol and asthma morbidity and across countries (5)

25. Possible mechanisms Paracetamol Low glutathione level Lack of suppression of cyclooxigenase pathway **IgE mediated immune response Inability to counteract oxydative stress Defective antigen processing **Toxic effects of n.acetyl-p- benzoquinone mine Lung Injury with Bronchoconstriction Source: Eneli et al. 2005

26. Proof onus McBride (2011) explicitly warns against the use of Paracetamol in children with asthma or at risk for asthma and claims that if further evidence is required, then this is for documenting product safety rather than the contrary. This explicitly addresses the reluctance of sceptical commentators to accept such evidence as a sufficient basis for practice change and for establishing a causal relationship between paracetamol and asthma, on grounds that it does not result from randomized clinical trials (Eneli et al. 2005, Allmers et al. 2009, Johnson and Ownby, 2011; Karimi et al., 2006, Wickens et al. 2011, Chang 2011). Chang 2011

27. Hierarchy reversal = chronological 1.Non-controlled observational studies (e.g. ecological studies) 2. comparative studies (case-controlled retrospective) 3. prospective studies + evidence coming from cellular and molecular studies.

28. However, the justification of this hypothesis is neither exclusively focused on the exclusion of confounders (Vandenbroucke’s point 1), nor based on utility functions(Vandenbroucke’s point 2). Instead, several independent pieces of evidence jointly support a given hypothesis and are considered to do at least as good a job as the one-shot proof presumably provided by an RCT (or a meta-analysis of RCTs).

29. 1. Both according to Vandenbroucke’s point 1, as well as to the recent contribution by Aronson and Haube, (or Howick’s contribution to the topic) case reports and observational data are considered sufficient evidence for causal claims to the extent that possible confounders (i.e. alternative causes for the experimental result) can be confidently excluded. This kind of reasoning also guides the general framework of evidence hierarchies: The higher the likelihood that the study design rules out more confounders than others, the higher it is settled in the ranking. And it is a straightforward consequence of the method of hypothesis testing, which is an hypothetico/deductive mode of investigation, for which the evidence is supposed to refute the null hypothesis. 2. Vandenbroucke’s call on priors instead (point 2) invokes a Bayesian epistemology, where hypotheses are assigned probabilities and these are updated in the light of new data. 3. In McBride’s, the causal hypothesis is assessed abductively, by putting things together and inferring the implications of their joint occurrence. The hypothesis of causal connection provides a unified explanation of the different pieces of evidence, which would otherwise need a series of distinct explanatory facts.

30. Epistemological paradigms EpistemologyMethodAssumptions Justification of “lower level” evidence Unificationist (qualitative abduction) Connection of data in light of explanatory hypothesis Connectedness (ontological) Explanatory power of hypothesis in light of data. (see also recent proposal of integration of Bradford-Hill criteria: Stegenga, 2011; Howick et al. 2009). Inductive-Bayesian (quantitative abduction) Bayes theorem Principle of total evidence - coherence Probability of hypothesis given evidence Hypothetico- deductive (statistical mode) Hypothesis testing: likelihood of evidence if H 0 = true (p-value) Homogeneous populations with regard to all possible relevant causal factors (randomization) Only if alternative hypotheses (confounders) can be safely excluded, or treatment effect swamps them by a statistically significant amount (Howick, 2011).

31. The unificationist paradigm regards hypotheses as explanatory factors for the observed data. In order to be really explanatory they must accommodate as many data as possible. Data which fail to be taken into account are left unexplained, thus making the hypothesis less virtuous from a theoretical point of view. In the Bayesian paradigm hypotheses can be associated with any probability in the unit interval. The main requirement is coherence (in the mathematical sense of standard probability calculus) and that all available evidence is used: this because, bayesian epistemology tracks inductive uncertainty and all non-deductive logics are non-monotonic. Epistemological paradigms

32. Non-monotonicity  Principle of total evidence Nonmonotonicity means that an addition to the premises may invalidate some previous conclusion. The principle of total evidence responds to this issue and is an essential principle of uncertain inference (Carnap, 1947). Linda is getting out of the bank  It is 4 pm. Linda is getting out of the bank and she is going to an antinuclear demonstration  It is earlier than 4 pm.

33. Hypothetico-deductive method (statistical mode) Instead statistical hypothesis-testing is a kind of approach which admittedly follows a Popperian hypothetico-deductive method of scientific enquiry. And being this paradigm inherently deductive, it does not feel urged to address the issue of non- monotonicity. The very idea of hierarchies follows from the assumption that if you have a study which has the capacity to eliminates more confounders than another, than the former should trump the latter. Trumping means that higher level evidence discards any evidence of inferior ranking, and also makes it irrelevant.

34. Lexicographic rule for hierarchy implementation Higher level studies trump lower level ones: 1.when two studies of different levels deliver contradictory findings, then the higher in the evidence study is considered more reliable and is allowed to discard the lower level one 2.lower level evidence adds nothing to higher level one and thus it can be neglected without loss of information.

35. The strongest way to interpret hierarchies is to claim that, because it is assumed that randomization provides a guarantee for causality, than it should follow that there is no guarantee of causality without randomization. “No randomization in, no causes out” Lexicographic rule for hierarchy implementation Cause

36. I think all instruments of investigation play an important role in scientific investigation. Analytical approaches work with truth conditions (RCTs set strict desiderata for considering sthg to be a cause for sthg else) but have strong methodological (only certain kind of evidence is at all meaningful) and epistemological limitations (failure to account for important phenomena such as causal interaction - see Muller). Inductive approaches work with imperfect indicators (P(E/H); dose- response relationship; specificity of the association); or theoretical virtues such as the explanatory power), thus not only they can, but they must take into account all evidence (non-monotonicity). Nevertheless they can give you no sure-fire guarantee that the delivered result demonstrates the truth of the hypothesis.

37. 1. Probabilistic Hypothesis of Causal Connection From the time a risk is not known, to the moment in which it is incontrovertibly proven to be causally associated with the drug, there is a period of evidence accumulation which constitutes a state of partial and imperfect (but continuously increasing) knowledge. In this period it cannot be claimed that there is a causal link between the drug and the detected risk; but neither can we behave as if we knew nothing about it. Still, the latter attitude is precisely the only possible policy allowed by an epistemology grounded on hypothesis rejection.

38. 2. Implementation of the precautionary principle Following the precautionary principle, you are not supposed to wait for the causal connection between harm and suspected drug to be certain, before you take adequate countermeasures, but instead, you should act as soon as the probability of causal connection is high enough to recommend countermeasures because of a negative risk/benefit balance. This probability might be also very low, in case the risk magnitude is considerably big with respect to the expected benefit. The frequentist mode of summarizing statistical data, following which hypotheses may only be accepted or rejected, cannot be of any use to this purpose.

39. The problem of external validity and causal interaction is dramatic in the case of side- effects. 3. Problem of external validity (ontological

40. Conclusion My aim is to raise awareness about the different epistemological paradigms underlying the distinct evidence policies we may intuitively endorse. The take-home lesson is that different epistemologies grant different methodological actions and impose different, which in turn bring about relevant practical implications. Thus it is worthy to bear in mind the criteria underlying our evidence constraints whether we want to rank it or not, or else to reverse rankings.