1. The Newcastle-Ottawa Scale (NOS) for Assessing the Quality of Nonrandomized Studies in Meta-Analysis
G. Wells, B. Shea, D. O’Connell, J. Robertson, J. Peterson, V. Welch, M. Losos, P. Tugwell

2. Development
Applications
Current Developments

3. Development: Item Selection
Newcastle quality assessment form
Ottawa comprehensive list
Panel review
Critical review by experts

4. Development: Grouping Items
Cohort studies
Selection of cohorts
Comparability of cohorts
Assessment of outcome
Case-Control studies
Selection of case and controls
Comparability of cases and controls
Ascertainment of exposure

5. Development: Identifying Items
Identify ‘high’ quality choices with a ‘star’
A maximum of one ‘star’ for each h item within the ‘Selection’ and ‘Exposure/Outcome’ categories; maximum of two ‘stars’ for ‘Comparability’

8. Newcastle-Ottawa Quality Assessment Scale: Case-Control Studies
Selection (4)
Comparability (1)
Exposure (3)
A study can be awarded a maximum of one star for each numbered item within the Selection and Exposure categories. A maximum of two stars can be given for Comparability

9. Selection e.g. ICD codes in database >1 person/record/time/process
1. Is the case definition adequate?
a) yes, with independent validation 
b) yes, eg record linkage or based on self reports
c) no description
2. Representativeness of the cases
a) consecutive or obviously representative series of cases 
b) potential for selection biases or not stated
3. Selection of Controls
a) community controls 
b) hospital controls
4. Definition of Controls
a) no history of disease (endpoint) 
b) no description of source
e.g. ICD codes in database
or self-report with no
reference to primary record
or no description
>1 person/record/time/process
to extract information, or
reference to primary record
source such as x-rays or
medical/hospital records

10. Comparability
1. Comparability of cases and controls on the basis of the design or analysis
a) study controls for ___________ (select the most important factor) 
b) study controls for any additional factor (This criteria could be modified to indicate specific control for a second important factor.) 

11. Exposure 1. Ascertainment of exposure
a) secure record (eg surgical records) 
b) structured interview where blind to case/control status 
c) interview not blinded to case/control status
d) written self report or medical record only
e) no description
2. Same method of ascertainment for cases and controls
a) yes 
b) no
3. Non-Response Rate
a) same rate for both groups 
b) non respondents described
c) rate different and no designation

12. Newcastle-Ottawa Quality Assessment Scale: Cohort Studies
Selection (4)
Comparability (1)
Outcome (3)
A study can be awarded a maximum of one star for each numbered item within the Selection and outcome categories. A maximum of two stars can be given for Comparability

13. Selection
1. Representativeness of the exposed cohort
a) truly representative of the average ___________ (describe) in the community 
b) somewhat representative of the average ___________ in the community 
c) selected group of users eg nurses, volunteers
d) no description of the derivation of the cohort
2. Selection of the non exposed cohort
a) drawn from the same community as the exposed cohort 
b) drawn from a different source
c) no description of the derivation of the non exposed cohort
3. Ascertainment of exposure to implants
a) secure record (eg surgical records) 
b) structured interview 
c) written self report
d) no description
4. Demonstration that outcome of interest was not present at start of study
a) yes 
b) no
In the case of mortality studies, outcome of interest is still the presence of a disease/ incident, rather than death; that is a statement of no history of disease or incident earns a star

14. Comparability
1. Comparability of cohorts on the basis of the design or analysis
a) study controls for ___________ (select the most important factor) 
b) study controls for any additional factor (This criteria could be modified to indicate specific control for a second important factor.) 

15. Outcome 1. Assessment of outcome a) independent blind assessment 
b) record linkage 
c) self report
d) no description
2. Was follow up long enough for outcomes to occur
a) yes (select an adequate follow up period for outcome of interest) 
b) no
3. Adequacy of follow up of cohorts
a) complete follow up - all subjects accounted for 
b) subjects lost to follow up unlikely to introduce bias - small number lost - > ___ % (select an adequate %) follow up, or description of those lost) 
c) follow up rate < ___% (select an adequate %) and no description of those lost
d) no statement

16. Applications: Assess quality of nonrandomized studies
Incorporate assessments in interpretation of meta-analytic results
Design, content and ease of use

17. Long Term Hormone Replacement Therapy and Coronary Heart Disease Events

18. Steps of a Cochrane Systematic Review
Clearly formulated question
Comprehensive data search
Unbiased selection and abstraction process
Critical appraisal of data
Synthesis of data
Perform sensitivity and subgroup analyses if appropriate and possible
Prepare a structured report

19. Objective
Is there a relationship between hormone replacement therapy and the incidence of coronary heart disease in postmenopausal women

20. Inclusion Criteria Types of studies Population Intervention Outcomes
case-control, cohort or cross-sectional studies
Population
postmenopausal women
Intervention
women exposed to hormone replacement therapy (estrogen or estrogen + progesterone)
ever, current, past
Outcomes
coronary heart disease (events)
fatal, non-fatal, both

21. Steps of a Cochrane Systematic Review
Clearly formulated question
Comprehensive data search
Unbiased selection and abstraction process
Critical appraisal of data
Synthesis of data
Perform sensitivity and subgroup analyses if appropriate and possible
Prepare a structured report

22. Search Strategy Electronic Search of: Other Data Sources:
MEDLINE (1966 to May 2000)
Current Contents (to May 2000)
Other Data Sources:
review of references cited in retrieved articles

23. Steps of a Cochrane Systematic Review
Clearly formulated question
Comprehensive data search
Unbiased selection and abstraction process
Critical appraisal of data
Synthesis of data
Perform sensitivity and subgroup analyses if appropriate and possible
Prepare a structured report

24. Data Extraction 2 independent reviewers selected trials
2 independent reviewers extracted data using pre-determined forms
study design
population characteristics
exposure to implants
outcomes measures
results
differences resolved by consensus

25. Results
16 case-control or cross-sectional
14 cohort

26. Quantification of Effects
Exposure (ever, current, past)
Outcome (fatal, non-fatal, both)
Effect estimates (EE)
Relative Risk (RR)
Odds Ratio (OR)
Adjusted effect estimates
Effects vs population, follow-up periods, etc. (homogeneity)

27. Steps of a Cochrane Systematic Review
Clearly formulated question
Comprehensive data search
Unbiased selection and abstraction process
Critical appraisal of data
Synthesis of data
Perform sensitivity and subgroup analyses if appropriate and possible
Prepare a structured report

28. Cohort Star Template Selection Comparability Outcome Avila / 90
Bush / 87
Cauley / 97
Criqui / 98
Ettinger / 96
Folsom / 95
Grodstein / 96
Henderson / 91
Lafferty / 94
Lauritzen / 83
Petitti / 87
Sourander / 98
Wilson / 85
Wolf / 96

29. Case-Control Star Template
Selection Comparability Exposure
Adam / 81
Beard / 89
Croft / 89
Grodstein / 97
Heckbert / 97
LaVecchia / 87
Mann / 94
Pfeffer / 78
Rosenberg / 76
Rosenberg / 80
Rosenberg / 93
Ross / 81
Sidney / 97
Szklo / 84
Talbott / 77
Thompson / 89

30. Adjusted Effect Estimates for Coronary Heart Disease (All Events) (HRT: Estrogen Current Use) Case-Control Studies
Selection Comparability Exposure

31. Adjusted Effect Estimates for Coronary Heart Disease (All Events) (HRT: Estrogen Past Use) Case-Control Studies
Selection Comparability Exposure

32. Adjusted Effect Estimates for Coronary Heart Disease (All Events) (HRT: Estrogen Ever Use) Case-Control Studies
Selection Comparability Exposure

33. Adjusted Effect Estimates for Coronary Heart Disease (All Events) (HRT: Estrogen + Progestin Ever Use) Case-Control Studies
Selection Comparability Exposure

34. Adjusted Effect Estimates for Coronary Heart Disease (All Events) (HRT: Estrogen Current Use) Cohort Studies
Selection Comparability Outcome

35. Adjusted Effect Estimates for Coronary Heart Disease (All Events) (HRT: Estrogen Ever Use) Cohort Studies
Selection Comparability Outcome

36. Current Development: Validity
Face/content validity
Criterion validity
compare to more comprehensive scales
compare to expert judgement
Construct validity
external criteria
‘convergent validity’
‘divergent validity’
internal structure
‘factorial validity’

37. Current Development: Reliability
Inter-rater reliability
Intra-rater reliability

38. Future Development: Scoring
Identify threshold score distinguishing between ‘good’ and ‘poor’ quality studies

39. The Newcastle-Ottawa Scale (NOS) for Assessing the Quality of Nonrandomized Studies in Meta-Analysis
NOS Quality Assessment Scales:
Case-control studies
Cohort studies
Manual for NOS Scales