1. 1 Evaluation of Non-Randomized Studies S. Stanley Young National Institute of Statistical Sciences Robert L. Obenchain Eli Lilly & Company FDA 21Sept2006

2. 2 UPSIDE: the uniform principle for sharing integral data and materials expeditiously Committee on Responsibilities of Authorship in the Biological Sciences National Research Council National Academy of Science http://www.nap.edu/catalog/10613.html 2003

3. 3 Community Standards More specifically, the act of publishing is a quid pro quo in which authors receive credit and acknowledgment in exchange for disclosure of their scientific findings. An author’s obligation is not only to release data and materials to enable others to verify or replicate published findings (as journals already implicitly or explicitly require) but also to provide them in a form on which other scientists can build with further research. …whatever is necessary to support the major claims of the paper and would enable one skilled in the art to verify or replicate the claims.

4. 4 Materials and Methods Most journals today explicitly or implicitly require that authors provide enough detail about their materials and methods to allow a qualified reader to verify, replicate, or refute the findings reported in a paper.

5. 5 Further Comment It is unacceptable to require collaboration or coauthorship as a condition of providing a published material, because that requirement can inhibit a scientist from publishing findings that are contrary to the provider’s published conclusions.

6. 6 Mangano et al. Claim Claim: “The association between aprotinin and serious end-organ damage indicates that continued use is not prudent.” This is a serious claim. The data structure and analysis is quite complex. Mangano et al. have refused multiple requests for the data set. Mangano et al. give insufficient analysis details for their analysis results to be replicated.

7. 7 External Consideration It is error only, and not truth, that shrinks from inquiry. Thomas Paine, 1737-1809

8. 8 Analysis of Complex Data Sets 1.There are multiple response questions. 2.Responses can be combined in numerous ways. 3.There are infinitely many analysis options with hundreds of covariates. 4. P-values can be moved many orders of magnitude. 5. In exploratory analysis, p-values and RR have no meaning.

9. 9 Very high risk High risk Moderate risk Low risk Control Tranexamic acid Aminoproic acid Aprotinin Unadjusted Analysis Patient Allocation

10. 10 External Consideration 80% of highly cited non-randomized studies were contradicted versus only 23% of randomized controlled trials were contradicted. Contradicted and Initially Stronger Effects in Highly Cited Clinical Research John Ioannidis, JAMA (2005)

11. 11 Summary 1.Until the data are made available to others, and a proper analysis computed, these claims should be ignored. 2.The statistical analysis is seriously flawed. Where were the NEJM editors and referees?