Criteria to assess quality of observational studies evaluating the incidence, prevalence, and risk factors of chronic diseases Minnesota EPC Clinical Epidemiology Methods Centre, Ottawa Health Research Institute, Ottawa Tufts University Medical Center, Boston RTI International – University of North Carolina, Chapel Hill Blue Cross and Blue Shield Association, Chicago Centers for Disease Control and Prevention, Atlanta Southern California EPC; RAND Corporation, Santa Monica German Institute for Quality and Efficiency in Health Care, Köln, Germany Johns Hopkins University, Baltimore

Analytical tasks

Therapeutic vs. non therapeutic studies PICOTSNon therapeutic studiesNon randomized studies PopulationGeneral populationPatients, health care recipients Exposure (interventions, independent variables) Risk factors that are independent of health care (allocation irrelevant) Treatments allocation by investigators or health care professionals ComparatorsLevels of riskAnother trtm, usual care, standard care, previous trtm OutcomesIncidence/prevalence of chronic conditions Morbidity, mortality, quality of life, adverse events after treatments TimeLength of exposureLength of trtm, followup off the trtm SettingsGeneral populationHealth care settings Comparability (selection bias) Not relevantFlaw Intention-to treatNot relevantFlaw

Systematic literature review Systematic reviews and meta-analyses of observational nontherapeutic studies published in English in core clinical journals (145) Published tools (scales or checklists) for quality assessment of observational studies (96) Annotated bibliography of the methodological literature to identify biases and strategies to reduce bias in observational studies

Development of Checklists Separate evaluation of external and internal validity Exhaustive range of criteria Discriminating reporting and methodological quality Predefined major and minor flaws Study and hypotheses level analyses Automated reports Templates for the protocols of quality evaluations with predefined research specific quality standards Manual, help files, instructions

Collaboration with participating experts Criteria evaluation: face and content validity Pilot reliability testing (10 randomly selected articles): overall, by topic, by article, by domains of external and internal validity, and by quality component Discriminant validity: testing hypotheses of detected differences in quality and reporting vs. methodological quality

Finalizing checklists Checklist for studies of incidence or prevalence of chronic diseases (MORE) Checklist for studies of risk factors of chronic diseases (MEVORECH) Synthesis of evidence from the studies with major flaws: Level A exclusion from synthesis Level B full review and quality abstraction Level C separate limited synthesis if major flaws detected

Quality criteria Incidence or prevalence studies Aim of study Study design External Validity 1. Sampling of the subjects by the investigators 2. Assessment of sampling bias 3. Estimation of sampling bias 4. Exclusion rate from the analysis 5. Sampling bias is addressed in the analysis 6. Subject flow Internal Validity 1. Source to measure outcomes 2. Definition of outcomes 3. Measurements of outcomes 4. Outcomes in race, ethnic, age, or gender subpopulations 5. Reporting of outcomes Risk factors studies Aim of study Objectives Study design External Validity 1. Sampling of the subjects by the investigators 2. Assessment of sampling bias 3. Estimation of sampling bias 4. Exclusion rate from the analysis 5. Sampling bias is addressed in the analysis 6. Subject flow Internal Validity 1.Source to measure outcomes 2. Definition of outcomes 3. Measurements of outcomes 4. Definition of the exposure 5.Measurements of the exposure 6. Confounding factors 7. Loss of followup 8. Masking of exposure status 9. Statistical analysis 10. Appropriateness of statistical models 11. Reporting of tested hypothesis 12. Precision of the estimates 13. Sample size justification

Major flaws External validity Sampling in clinics for incidence in the general population; Sampling of controls from different population as cases; Response rate <40%; Exclusion of >10% of eligible subjects from the analyses Reasons to exclude from the analyses differ for exposed and not exposed Internal validity Severity of the outcome can be relevant but not assessed in the study; Non valid methods were obtained to measure the outcomes, exposure, or confounding factors; Different methods to measure exposure (risk factors, independent variable) in cases and controls; No strategies to reduce bias

Minor Flaws External validity Random sampling restricted to geographic area Convenient sampling The authors did not assess sampling bias Sampling bias was not addressed in the analysis Sampling of controls from health care related sources Internal validity Outcomes proxy reported Obtained from medical records (data mining) or administrative database Inter-methods validation Subjective judgment of reliability Intensity/dose of exposure can be relevant but not assessed in the study

Example of the report Article: _________________ Evaluator: _______________ External Validity Not Reported Addressing sampling biasNot reported Sampling: General population basedNot reported Major flaw Exclusion rate from the analysis>10% Internal Validity Major flaw Definition of incidence/prevalence: Severity Can be relevant but not assessed in the study Minor Definition of incidence/prevalence: Frequency of symptomsCan be relevant but not assessed in the study

Conclusions We propose two checklists for transparent and standardized quality assessment in systematic reviews and evidence-based reports Reasonable validity Poor random overall and good component reliability in a pilot testing of the articles from different topics

Future Research with Quality Assessment Using Proposed Checklists Protocols of systematic reviews of nontherapeutic observational studies with justified definitions of research specific quality standards and preplanned reliability testing Synthesis of evidence incorporating quality assessment (sensitivity of the results to quality) The evaluation of the level of evidence from several observational nontherapeutic studies Quality assessment of primary studies should improve quality of systematic reviews and evidence-based reports