1. Research Designs, Threats to Validity and the Hierarchy of Evidence and Appraisal of Limitations (HEAL) Grading System

2. The Cochrane Collaboration has developed a tool that review authors are expected to use for assessing risk of bias in randomized trials.
Six bias domains:
Selection bias
Performance bias
Detection bias
Attrition bias
Reporting bias
Other bias

3. List of study design features (studies formed by classifying individuals or clusters by intervention and comparator) From Higgins et al.,. 2011
Was there a relevant comparison:
Between two or more groups of participants receiving different interventions?
Within the same group of participants over time?
Were groups of individuals formed by:
Randomization?
Quasi-randomization?
Other action of researchers?
Time differences?
Location differences?
Healthcare decision makers?
Participants’ preferences?
On the basis of outcome?
Some other process? (specify)
Were these features of the study carried out after the study was designed:
Identification of participants?
Assessment before intervention?
Actions/choices leading to an individual becoming a member of a group?
Assessment of outcomes?
On which variables was comparability between groups assessed:
Potential confounders?
Assessment of outcome variables before intervention?

4. Threats to Validity of Research Designs/Findings Gigui et al. , 2012
Threats to Validity of Research Designs/Findings Gigui et al., A new evidence grading system, Evaluation & the Health Professions
Major threats
Controlled study without a true control group
Low dosage/poor treatment fidelity
Treatment crossover/no intent-to-treat
Insufficient follow-up period
Differential follow-up period
Low statistical power
Selection bias
Differential attrition between groups
Contaminated baseline measure
Unstable cohort sample
Design-Specific Limitations
Biased allocation in an RCT (e.g., poor blinding or allocation concealment)*
Flawed randomization in a RCT (e.g., poor sequence generation)*
Insufficient number of matching variables
No tests for statistical equivalence
Non-equivalence or evidence of significant difference/differences
Minor limitations
Lengthy data collection period
Constantly changing intervention
Comparison groups are not mutually exclusive
Used postbaseline covariates in analysis
Potential crossover effect
Poor generalizability of results (low participation or high attrition)
* Meta-epistemological research has not conclusively shown that poor sequence generation, blinding, and allocation concealment yield biased outcomes (Gugiu & Gugiu, 2010). Therefore, such limitations should only be employed to downgrade a study when there is ample cause for doing so (e.g., differences in key baseline variables).

5. Hierarchy of Evidence and Appraisal of Limitations (HEAL) Grading System From Table 1 of Gugiu et al., A new evidence grading system, Evaluation & the Health Professions, 36 (1), 3-43
Study Design
Grade Description
Randomized Controlled Trial (RCT)
A well-conducted RCT free of bias or confounding factors in which the randomization process is able to ensure the equivalence of the treatment and control groups beyond acceptable statistical chance
An RCT with a flawed randomization process but which properly employed an equivalent controlled trial (ECT) method for ensuring the equivalence of the treatment and control groups beyond a reasonable doubt
An RCT with a flawed randomization process, which cannot ensure the equivalence of the treatment and control groups beyond a reasonable doubt. Alternatively, a cohort RCT that failed to show the population from which the samples were drawn remained stable over time
An RCT with one or more of the following fatal flaws:
The treatment group received a very low dosage or poor treatment fidelity;
significant crossover existed between treatment and control subjects;
the follow-up period was insufficient to detect change;
the follow-up period (either timing or length of time) was significantly different between the treatment and control arms;
the sample size was significantly lower than the recommendations from an a priori power analysis;
a sig. selection bias or differential attrition rates occurred between the treatment and control groups that produced baseline group differences;
significant contamination of baseline outcome measures occurred.

6. HEAL grading system (cont.)
Pseudo-RCT, matched comparison, stratified random sampling, paired comparison, regression discontinuity
B. A well-conducted Equivalent Control Trial (ECT) that demonstrated statistical equivalence of the treatment and control groups on key baseline variables or the pre-post regression model.
C. An ECT with demonstrated baseline differences on key variables or that did not employ an adequate number of covariates, pairing, or strata variables to sufficiently remove reasonable doubt regarding the statistical equivalence of the treatment and control groups on key baseline variables or a regression discontinuity design that failed to demonstrate the equivalence of the pre–post regression model. Alternatively, a cohort ECT that did not demonstrate the population from which the samples were drawn remained stable over time.
D. An ECT with one or more of the fatal flaws listed under RCT Grade D.
Cohort study, case–control
C. A controlled study that did not adequately establish the equivalence of the treatment and control groups beyond a reasonable doubt (i.e., a nonequivalent controlled trial [NECT])
D. An NECT with one or more of the fatal flaws listed under RCT Grade D.  
Before–after, case series
D. Any study that did not employ a controlled comparison between two or more groups (i.e., an uncontrolled trial [UT]), including RCT, ECT, and NECT studies that did not employ a true comparison group (i.e., compared two treatments against each other but not against a treatment as usual group)

7. HEAL grading system (cont.)
Footnote. If the overall grade is greater than D then lower the grade by half a level for each of the following flaws that occur:
The length of data collection period was extensive enough to significantly increase within group variance on primary outcomes;
It was difficult to identify the active components of the intervention because the intervention changed throughout the study;
The comparison groups were not mutually exclusive (e.g., overlap in group membership);
Post-baseline covariates that are not expected to remain constant over time were used to establish equivalence at baseline;
A potential or very minor crossover may have occurred; or
The generalizability of the results were compromised due to differences between the sample and population.
Statistical significance was exaggerated because analysis did not take proper account of nesting.
I added this one – this issue is never discussed by Gugiu (but see Higgins et al., 2011).

8. From Gugiu et al., 2012 – A new grading system

9. Implications for your meta-analysis
The first few pages of Chapter 6 of P&R
Set up a spreadsheet with columns for all of the critical information (most of which you will be able to cut and paste into CMA)
Brief title
Authors, year, year of intervention
PICO factors –
population/sample, intervention/exposure, comparison, context
Research design
RCT, ECT, NECT, UT, prospective/retrospective
QUALITY RATING
Use an identified system
Outcomes – one row for each outcome (you will copy most of the other items to each row)
several statistics, e.g., means, SDs and Ns for Tx and Ctrl groups
Moderators
others besides sample and context factors