1. 19 de Dezembro de 2007

2. A Meta-Analysis is a review in which bias has been reduced by the systematic identification, appraisal, synthesis and statistical aggregation of all relevant studies on a specific topic according to a predetermined and explicit method. Overview Systematic Review Meta-Analysis

3. In 1987, a survey showed that only 24, out of 86 English-language meta-analyses, reported all the six areas considered important to be part of a meta-analysis: Study Design Control of Bias Statistical Analysis Application of Results Sensitivity Analysis CombinalityCombinality

4. In 1992 this survey was updated with 78 meta-analyses and the researchers noted that methodology has definitely improved since their first survey; However it needed better searches of the: Literature; Quality evaluations of trials; Synthesis of the results. In 1992 this survey was updated with 78 meta-analyses and the researchers noted that methodology has definitely improved since their first survey; However it needed better searches of the: Literature; Quality evaluations of trials; Synthesis of the results.

5. So, in 1999, several researchers suggested the creation of the Quality of Reporting of Meta-Analyses (QUOROM) Statement to improve and standardise reporting.

6. The QUOROM (Quality Of Reporting Of Meta- analyses) conference resulted in the creation of the QUOROM Statement (checklist and flow diagram); This checklist describes the preferred way to present the Abstract, Introduction, Methods, Results, and Discussion sections of a report of a Meta-Analysis; It is organized into 21 headings and subheadings. The QUOROM (Quality Of Reporting Of Meta- analyses) conference resulted in the creation of the QUOROM Statement (checklist and flow diagram); This checklist describes the preferred way to present the Abstract, Introduction, Methods, Results, and Discussion sections of a report of a Meta-Analysis; It is organized into 21 headings and subheadings.

7. But what is Reproducibility? Why is it so important? Reproducibility is one of the main principles of the scientific method, which refers to the ability of a test or experiment to be accurately reproduced by someone else working independently. But what is Reproducibility? Why is it so important? Reproducibility is one of the main principles of the scientific method, which refers to the ability of a test or experiment to be accurately reproduced by someone else working independently.

8. A Non-Reproducible article will most probably end in: Results' heterogeneity; At a clinical level, the risk of a patient being wrongly diagnosed; Lead to a decrease on the credibility of a checklist and, consequently, of the meta-analyses that used it as a model. A Non-Reproducible article will most probably end in: Results' heterogeneity; At a clinical level, the risk of a patient being wrongly diagnosed; Lead to a decrease on the credibility of a checklist and, consequently, of the meta-analyses that used it as a model.

9. The question we want to answer is if the QUOROM Checklist is a reproducible method in the evaluation of Meta-Analysis. Primary Aim: Evaluate the reproducibility degree of the QUOROM Checklist The question we want to answer is if the QUOROM Checklist is a reproducible method in the evaluation of Meta-Analysis. Primary Aim: Evaluate the reproducibility degree of the QUOROM Checklist

10. Secondary Aims: Specify which points of the QUOROM Checklist are less reproducible, i.e., those which present more disparity in the inter-observer results; Verify if there are differences in the reproducibility between the evaluation of meta- analysis from Low Impact Factor journals and from High Impact Factor ones. Secondary Aims: Specify which points of the QUOROM Checklist are less reproducible, i.e., those which present more disparity in the inter-observer results; Verify if there are differences in the reproducibility between the evaluation of meta- analysis from Low Impact Factor journals and from High Impact Factor ones.

11. First, we will separate 30 journals using a Stratified Sampling Method. Low IF Journals High IF Journals From all Journals of ISI Web of Knowledge, we will select: 15 Journals 0 < IF ≤ 1 IF > 10 IF – Impact Factor Inclusion Criteria: all of them must have been categorized as Medicine Subject’s Journals

12. Allergy; Anatomy & Morphology; Andrology; Anaesthesiology; Cardiac & Cardiovascular Diseases; Clinical Neurology; Critical Care Medicine; Dentistry; Oral Surgery & Medicine; Dermatology; Emergency Medicine; Endocrinology & Metabolism; Gastroenterology & Hepatology; Genetics & Heredity; Geriatrics & Gerontology; Health Care Sciences & Services; Haematology; Immunology; Infectious Diseases; Allergy; Anatomy & Morphology; Andrology; Anaesthesiology; Cardiac & Cardiovascular Diseases; Clinical Neurology; Critical Care Medicine; Dentistry; Oral Surgery & Medicine; Dermatology; Emergency Medicine; Endocrinology & Metabolism; Gastroenterology & Hepatology; Genetics & Heredity; Geriatrics & Gerontology; Health Care Sciences & Services; Haematology; Immunology; Infectious Diseases; Integrative & Complementary Medicine; Medical - Ethics, Informatics, Laboratory Technology; Virology; Oncology; Medicine – General & Internal, Legal, Research & Experimental; Neurosciences; Nursing; Nutrition & Diabetics; Obstetrics & Gynaecology; Ophthalmology; Orthopaedics; Otorhinolaryngology; Parasitology; Pathology; Paediatrics; Integrative & Complementary Medicine; Medical - Ethics, Informatics, Laboratory Technology; Virology; Oncology; Medicine – General & Internal, Legal, Research & Experimental; Neurosciences; Nursing; Nutrition & Diabetics; Obstetrics & Gynaecology; Ophthalmology; Orthopaedics; Otorhinolaryngology; Parasitology; Pathology; Paediatrics; Peripheral Vascular Disease; Pharmacology & Pharmacy; Physiology; Psychology; Psychiatry; Public, Environmental & Occupational Health; Radiology, Nuclear Medicine & Medical Imaging; Rehabilitation; Respiratory System; Substance Abuse; Surgery; Toxicology; Transplantation; Tropical Medicine; Urology & Nephrology. Peripheral Vascular Disease; Pharmacology & Pharmacy; Physiology; Psychology; Psychiatry; Public, Environmental & Occupational Health; Radiology, Nuclear Medicine & Medical Imaging; Rehabilitation; Respiratory System; Substance Abuse; Surgery; Toxicology; Transplantation; Tropical Medicine; Urology & Nephrology.

13. After this, we will proceed to the selection of the Meta- Analyses. For that, we will use a Multi-Stage Sampling Method. The totality of the Journals’ articles will be removed from each stratum, following 3 inclusion criteria: The Article must be a Meta-Analysis; The Article must have at least one link for full text; The Article must have a publication date comprised between 1999 and 2007. After this, we will proceed to the selection of the Meta- Analyses. For that, we will use a Multi-Stage Sampling Method. The totality of the Journals’ articles will be removed from each stratum, following 3 inclusion criteria: The Article must be a Meta-Analysis; The Article must have at least one link for full text; The Article must have a publication date comprised between 1999 and 2007.

14. Pool n.1 Pool n.2 Low IF Meta-Analyses High IF Meta-Analyses Pool n.3 52 Meta- -Analyses 52 Meta- -Analyses 26 52 If we don’t have 26 meta-analyses in each pool, we will repeat the selection of the Journals until this number is achieved

15. The impact factor of the journal from where each Meta- Analysis came will be recorded in a database, which will be kept in secret until the evaluation of the checklist is concluded. It will be used only at the end to find out if Reproducibility and Impact Factor are related. This way, after the formation of these 2 pools: Pool n.1 Pool n.2 Low IF Meta-Analyses High IF Meta-Analyses

16. Each student/investigator analyses a group of 4 articles and submits them to the QUOROM Checklist. After the students’ analysis, the articles will be mixed again. Then, each student will analyse another 4 articles, randomly selected from the 48 articles previously analysed by the rest of the group. This way, each student/investigator will analyse different articles. When analyzing a Meta-Analysis, each student must insert the data in the SPSS program. For each item, number 1 will be attributed to those which are covered by the Checklist in the Meta-Analyses, and number 0 to those which aren’t. Thus, our study can be classified as a Systematic Review. When analyzing a Meta-Analysis, each student must insert the data in the SPSS program. For each item, number 1 will be attributed to those which are covered by the Checklist in the Meta-Analyses, and number 0 to those which aren’t. Thus, our study can be classified as a Systematic Review.

17. Concordance in each Item of the Checklist: (reproducibility of each Item) As the QUOROM checklist has eighteen items to be considered and each one of these will be seen twice for each Meta-Analysis, we’ll consider thirty-six categorical variables, which can have two numerical codes: 1 or 0. So, eighteen concordance tables will be made in order to calculate the proportion of concordance, i.e. the number of times the two observers agree with each other about an item, in order to find their similarity or disparity in each point. Concordance in each Item of the Checklist: (reproducibility of each Item) As the QUOROM checklist has eighteen items to be considered and each one of these will be seen twice for each Meta-Analysis, we’ll consider thirty-six categorical variables, which can have two numerical codes: 1 or 0. So, eighteen concordance tables will be made in order to calculate the proportion of concordance, i.e. the number of times the two observers agree with each other about an item, in order to find their similarity or disparity in each point.

18. Reproducibility of the Checklist : Later, the sum of the numerical codes will consist on two new variables: “summation of the present items by observer 1” and “summation of the present items by observer 2”. From these two summations an average will be done, being another variable. The analyses of the summation will be done using the ICC method (Interclass Correlation Coefficient). Reproducibility of the Checklist : Later, the sum of the numerical codes will consist on two new variables: “summation of the present items by observer 1” and “summation of the present items by observer 2”. From these two summations an average will be done, being another variable. The analyses of the summation will be done using the ICC method (Interclass Correlation Coefficient).

19. Relation between IF and Reproducubility: At last, considering impact factor as our last variable to be analyzed, we will place, side by side, our results and the database in which we recorded the impact factor of the journal from where each meta- analysis came to compare them. This way, we can find out if reproducibility and impact factor are related. Relation between IF and Reproducubility: At last, considering impact factor as our last variable to be analyzed, we will place, side by side, our results and the database in which we recorded the impact factor of the journal from where each meta- analysis came to compare them. This way, we can find out if reproducibility and impact factor are related.

20. We expect to find, after our statistical analyses, that the QUOROM Checklist is reproducible in its global view. However, we also expect to find that some items of the QUOROM Checklist are less reproducible. There are some items which enclose many requirements and the articles may have some of them while others don’t. We expect to find, after our statistical analyses, that the QUOROM Checklist is reproducible in its global view. However, we also expect to find that some items of the QUOROM Checklist are less reproducible. There are some items which enclose many requirements and the articles may have some of them while others don’t.

21. Due to the primary selection of the articles by the impact factor of the journal in which they were published, we expect to obtain different results. Between the two groups of meta-analyses, one with IF ≤ 1 and the other with IF ≥ 10, we think the latter will satisfy more topics of the Quorum Checklist because, on our regard, the revision of these journals is more severe. Thus, we think that, on this group, there will be more concordance between our two reviewers, which will demonstrate that QUOROM Checklist is more reproducible on this kind of journals. Due to the primary selection of the articles by the impact factor of the journal in which they were published, we expect to obtain different results. Between the two groups of meta-analyses, one with IF ≤ 1 and the other with IF ≥ 10, we think the latter will satisfy more topics of the Quorum Checklist because, on our regard, the revision of these journals is more severe. Thus, we think that, on this group, there will be more concordance between our two reviewers, which will demonstrate that QUOROM Checklist is more reproducible on this kind of journals.

22. We think that this work is important because it may lead to the modification of some items of the QUOROM Checklist and consequently the improvement of its reproducibility. The modifications of some items of the QUOROM Checklist leads to the upgrading of its reproducibility which can guide authors to write better structured Meta-Analyses. We think that this work is important because it may lead to the modification of some items of the QUOROM Checklist and consequently the improvement of its reproducibility. The modifications of some items of the QUOROM Checklist leads to the upgrading of its reproducibility which can guide authors to write better structured Meta-Analyses.