Presentation on theme: "Quality assurance & control"— Presentation transcript:
1 Quality assurance & control Beyond the BasicsJaleh Gholami MD,MPH, PhD candidate
2 Accuracy of researchThe inferences made from results of epidemiologic research depends on the accuracy of its methods and procedures.
3 Quality Assurance & Quality Control The activities to ensure quality of the data before data collection are regarded as quality assurance.Quality Control:The efforts to monitor and maintain the quality of data during the conduct of the study are regarded as quality control.
4 Quality Assurance Study protocol : consist of a description of the general components of the investigation (10 steps).
5 Key features of the design of an epidemiologic study Formulation of study’s main hypothesis/ hypotheses (QA)A priori specification of potential confounding variables (QA).Definition of the characteristics of the study population for external validity (generalizability) purposes (QA).Definition of the design strategy (e.g., cohort, case-control, case-cohort) and the groups to be compared, and specification of selection procedures of internal validity. (QA)Definition of the design strategy and samples for studies of reliability and validity (QA/QC)
6 Key features of the design of an epidemiologic study 6. specification of the study power necessary to detect the hypothesized association(s) at a given level of significance (QA).7. standardization of procedures (QA)activities during data collection, including analysis of quality control and remedial actions (QC).Data AnalysisReporting of data
7 Key features of the design of an epidemiologic study Formulation of study’s main hypothesis/ hypotheses (QA)Specify:the independent variable(risk factor)the dependent variable (disease)the potential effect modifiers
8 Key features of the design of an epidemiologic study 2. A priori specification of potential confounding variables (QA).Review of pertinent literature helps:identifying the main confounding variableschoosing the most appropriate study designselecting the data that need to be collected.
9 Key features of the design of an epidemiologic study 3. Definition of the characteristics of the study population for external validity (generalizability) purposes (QA).Whether findings are applicable to other population (distribution of effect modifiers and population characteristics).فراوانی اعتیاد، تغذیه و نظایر آن در دانش آموزان 12 تا 18 ساله در مدارس قابل تعمیم به جامعه نمی باشد.
10 Key features of the design of an epidemiologic study 4. Definition of the design strategy (e.g., cohort, case-control, case-cohort) and the groups to be compared, and specification of selection procedures of internal validity. (QA) - prevention of selection bias, search for confounders, ..
11 Key features of the design of an epidemiologic study 5. Definition of the design strategy and samples for studies of reliability and validity (QA/QC) - gold standard and samples for studies of repeat measurements.
12 Key features of the design of an epidemiologic study 6. specification of the study power necessary to detect the hypothesized association(s) at a given level of significance (QA). - sample size determinant to have sufficient power at a given alpha error level.
13 Key features of the design of an epidemiologic study 7. standardization of procedures (QA) - preparation of written manuals that contain a detailed description of the procedures for selection of the study population and data collection, as well as training and certification of staff.
14 Key features of the design of an epidemiologic study 8. activities during data collection, including analysis of quality control and remedial actions (QC). - ongoing monitoring of data collection procedures, as well as conducting studies on samples to assess validity and reliability of measurements, which may result in retraining and recertification of study staff.
15 Key features of the design of an epidemiologic study 9. Data Analysis - establish analytic strategies in advance (e.g., the choice of "cutoff" points)
16 Key features of the design of an epidemiologic study 10. Reporting of data - findings should be reported as soon as possible after data collection activities are finished so as to preserve the timeliness of the study .To void publication bias , data should be reported regardless of the direction of findings.
17 Quality Assurance Study protocol : provides a global picture of the strategiesleads to the development of more detailed manuals of operationdescribes the general design and proceduresand assists the staff in understanding the context in which their specific activities are carried out
18 Quality Assurance Manuals of operation: detailed descriptions of exactly how the procedures specific to each data collection activity are to be carried out :MeasurementsInterviewsCategorizationsDerived variablesmaximizes the likelihood the tasks will be performed as uniformly as possible (esp., in multicenter studies)
19 Development or choice or modification the previously used one? Quality AssuranceData collection instrument:Questionnaires:Development or choice or modification the previously used one?Validity and reliability are sometimes knownComparing findings of the study with those of previous studies
20 Quality Assurance Training of staff: Familiarizing them with the procedures under his or her responsibilitydata collectionprocessing proceduressetting up appointments for interviews or visitspreparing materials for the interviewers and other data collectorscalibrating instrumentsassigning interviewers to study participantscertification of the staff member to perform the specific procedureperiodic retraining and recertification (QC)
21 Quality Assurance Pretesting and pilot studies: feasibility and efficiencypretest: to try to assessing specific procedures on a grab or convenience sampling in order to detect major flaws.pilot study: is a formal rehearsal of study procedures that attempt to reproduce the entire flow of operations in a sample as similar as possible to study participants .
22 Pretesting and pilot studies Assess participant recruitmentAssess data collection proceduresA questionnaire can assessed:flow of questionspresence of sensitive questionsappropriateness of categorization of variablesclarity of wording to the respondent and the interviewerclarity of the "question- by question" instructions to the interviewerEvaluating alternative strategies for participant recruitment and data collection
23 Quality Assurance Quality assurance Steps: Specify study hypothesis. Specify general design (study protocol)Choose or prepare instruments, develop procedures for data collection and processing (Develop manuals)Train staff, Certify staff.Using certified staff, pretest and pilot-study data collection and processing instruments and procedures; pilot-study alternative strategies for data collection
24 Quality controlQuality control strategies include observation of procedures performed by staff members, which allows the identification of obvious protocol deviations.
25 Observation Monitoring and Monitoring of Trends "over-the-shoulder" observationstatistical assessment of trends over time, after adjustment for age, sex, and other relevant characteristics, for each technician or observerdigit preference (e.g. age, blood pressure)external or internal standards (e.g. lab data)
28 Validity studiesA compromise between accuracy on the one hand and costs and participants' burden on the other is often necessary.
29 Validity studiesValidity studies in subsamples of participants who undergo both the study-wide procedure and a more accurate procedure serving as a "gold standard" allow assessing the impact of these errors on the study estimates.
30 laboratory Measurements External quality control program conducted in a masked fashion.using the same biological (e.g., serum) pool to compare measurements obtained from applying study procedures with those resulting from the application of the gold standard procedures.The biological pool can be external or internal to the study.
32 Example: serum cholesterol Screening measurements: blood obtained by the finger stick method in a nonfasting stateGold standard measurements: The standard measurements were done in fasting state under carefully controlled conditions and assayed in a nationally recognized standard laboratory
34 Questionnaires Comparing with the gold standard: information given by participants and physicians about treatmentfood frequency questionnaire and a 1-week diet diaryIt is important to assess samples of both "positive" and "negative" answers (estimation of both sensitivity and specificity) and collect Information separately for the groups being compared (cases and controls or exposed and unexposed) to evaluate misclassification (nondifferential or differential)
35 Availability of Validity Data from Previous Studies National Health and Nutrition Examination Survey III (NHANES III): the actual blood pressure levels measured (gold standard) and reports of antihypertensive medication use (self-reported hypertension)Lipid Research Clinics (LRC) Family Study: self-reported anthropometric variables in participants, whose weights and heights were both self-reported and measured.
36 Importance of Validity Studies the knowledge of the validity of exposure, outcome, main confounding variables, and effect modifiers is important in the causal inferential process.many variables, even those considered "objective," have a relatively poor validity. (sensitivity and specificity)
37 Limitations of Validity Studies the "gold standard" itself may not be entirely valid.validity studies samples frequently constitute a selected group of volunteers (may not be representative).usually small sample size of validity studies resulting statistical imprecision.sensitivity and specificity of the procedure may vary according to certain characteristics of the study population.extrapolating the results of a validation study from one population to another, particularly if the data collection instrument is a questionnaire.
38 conclusionThen the "corrected" estimate may be even less "correct" than the original one.
39 Reliability Studiesreliability studies assess the extent to which results agree when obtained bydifferent observers,study instruments or procedures, or by the same observer,study instrument, or procedure at different points in time.Ideally, the only source of variability in a study should be that between study participants.
40 Other sources of variability Variability due to imprecision of the observer or the method, which can be classified in two types:Within-observer (or intra-observer) or within-method variability,Between-observer (or interobserver) or between-method variabilityVariability within study participants
41 Within individual variability the within-individual variability masks the between-individual variability
42 Control the variability quality assurance procedures prevent or minimize within- and between-observer or method variabilityphysiologic within-individual variability is not amenable to prevention.Solution:standardizing the timing of data collection for measures with known temporal fluctuations (levels of cortisol or blood pressure)standardizing measurement conditions for variables affected by stress or activity (blood pressure)collect data at several points in time and use the average of all values.
43 Reliability studiesobtaining random repeat measurementsMasked or not?
45 misclassificationIf the sample size is of sufficient size, it is important to assess whether reliability estimates obtained for a sample of study participants differ according to relevant characteristics, which may result in differential levels of misclassification.
48 Sensitivity and Specificity Sensitivity and specificity are fixed properties of the test (or diagnostic criteria or procedure) itself, regardless of the characteristics of the study population.Do you agree on this?
49 Sensitivity and Specificity 1. dependence of sensitivity and specificity estimates on the cutoff level (arbitrariness in assessing and reporting the validity)2. the sensitivity and specificity of a given definition of a condition (based on a cutoff in a continuous distribution) depend on the distribution of the severity of the condition.
51 Sensitivity and Specificity 3. The validity of a test can vary from population to population when the test is not a direct marker of the condition (OB and prevalence of Peptic ulcer, PPD and prevalence of atypical mycobacterium)
52 Sensitivity and Specificity 4. sensitivity and specificity can vary according to the characteristics of individuals (limitation to the generalizability). if these variables represent the exposures of interest or are correlated with them, differential misclassification may arise.
54 Youden 's J StatisticJ = Sensitivity + Specificity Maximum value of the index =1 When there is perfect agreement Minimum value of the index =0 When sensitivity and specificity = 0.5 Theoretically the value of J can be negative
56 Percent AgreementAlthough the percent agreement is a reliability indice, it can be also used to assess validity (the agreement between the test and gold standard)the simplest method of summarizing agreement for categorical variablesit can be calculated for any number of categories, not just two
59 limitation of the percent agreement its values tend to be high whenever the proportion of negative-negative results is high (resulting from a low prevalence of positivity in the study population), particularly when the specificity is high.
60 Percent Positive Agreement Chamberlain’s percent positive agreement:the number of occurrences for which both observers report a positive result, out of the average number of positives by either observer.the number of occurrences for which both observers report positive results out of the total number of observations for which at least one observer does SO
67 KAPPA+ 1 means that the two observers are perfectly reliable. They classify everyone exactly the same way.0 means there is no relationship at all between the two observer’s classifications, above the agreement that would be expected by chance.- 1 means the two observers classify exactly the opposite of each other. If one observer says yes, the other always says no.
72 Dependence of Kappa on Prevalence for the same sensitivity and specificity of the observers, the kappa value is greater in the population in which the prevalence of positivity is higher.kappa tends toward 0 as the prevalence approaches either 0 or 1.
73 conclusioncomparisons of kappa values among populations may be unwarranted.kappa should be used and interpreted with caution. Use it in conjunction with other measures of agreement, such as the percent agreement .take into consideration its variability as a function of the prevalence of the condition and of the degree of similarity between observers with regard to the prevalence of positivity.
74 Indices of Validity/Reliability for Continuous Data: Correlation Graph (Scatter Diagram) the 45° lines, intercept = 0, regression coefficient = 1.0 unitIt is not as sensitive as alternative graphic techniques
75 Correlation Coefficient Pearson's r is probably one of the most frequently used measures of agreement for continuous variables .it is also one of the least appropriate!1. It is insensitive to systematic differences (bias) between two observers or readings and it is misleading assessment of agreement.
76 Correlation Coefficient 2. the value of r is very sensitive to the range of values. (broader distribution)3. Pearson's correlation coefficient is unduly sensitive to extreme values (outliers)
77 Vb = variance between individuals Vt = total variance lntraclass Correlation Coefficient (ICC) or reliability coefficient (R)Vb = variance between individualsVt = total varianceVe = unwanted variance (error)
78 lntraclass Correlation Coefficient (ICC) The ICC is equivalent to the kappa statistic for continuous variables.The range of ICC is from -1.0 (more realistically from 0) to + 1.0ICC is affected by the range of values in the study population (like Pearson's correlation coefficient)when Vb is small, ICC also will be small. This is particularly important in studies within populations in which the exposure values are either very high or very low.
79 Calculation of ICC from ANOVA table K is the number or repeat readingsn is the number of individual studies or specimens being studied
80 Coefficient of Variability This calculation would have to be repeated for all pairs of measurements, and the overall coefficient of variability would be the average of all pair-wise coefficients of variability.
82 it is much easier to assess: Bland-Altman Plotthe difference between the paired measurements (A - B in the ordinate) and their mean value [(A + B)/2, in the abscissa).it is much easier to assess:The magnitude of disagreement (including systematic differences)OutliersAny trend