Reliability and Validity Sa’ed H. Zyoud

The error of research Random error (Chance) (increase sample number) (Precision: free from RE) Precision is the degree to which repeated measurements under unchanged conditions show the same results Systematic error (Bias) (matching+ blinding) (Accuracy: free from SE) Accuracy is the degree of closeness of measurements of a quantity to that quantity's actual (true) value Sampling error (type 1 error, type 2 error) Measurement error (wrong stat, wrong in scoring)

Reliability Means "repeatability" or "consistency". A measure is considered reliable if it would give us the same result over and over again (assuming that what we are measuring isn't changing!). There are four general classes of reliability estimates, each of which estimates reliability in a different way.

Reliability (continued) Inter-Rater or Inter-Observer Reliability Intra-Rater or Intra-Observer Reliability (Test-Retest Reliability) Internal Consistency Reliability Inter-method reliability.

Inter-Rater or Inter-Observer Reliability Used to assess the degree to which different raters/observers give consistent estimates of the same phenomenon (i.e. is the variation in measurements when taken by different persons but with the same method or instruments.

Establish reliability on pilot data or a subsample of data and retest often throughout. There are a number of statistics which can be used to determine inter-rater reliability. Cohen's kappa, intra-class correlation.

Test-Retest Reliability Used to assess the consistency of a measure from one time to another. This approach assumes that there is no substantial change in the construct being measured between the two occasions. The amount of time allowed between measures is critical. intra-rater reliability is the degree of agreement among multiple repetitions of a diagnostic test performed by a single rater

Internal Consistency Reliability Used to assess the consistency of results across items within a test. We are  looking at how consistent the results are for different items for the same construct within the measure. Example   "I like to eat bran bread“ “I've enjoyed eating bran bread in the past” “I hate bran bread” -

Kinds of Internal Reliability There are a number of statistics which can be used to determine Internal Reliability. Inter-item Correlation Cronbach's Alpha (a)

cronbach's alpha ----------------------Internal consistency α ≥ .9-----------------------------------------Excellent. 9 > α ≥ .8------------------------------------Good. 8 > α ≥ .7------------------------------------Acceptable. 7 > α ≥ .6---------------------Questionable and Acceptable. 6 > α ≥ .5-----------------------------------Poor. 5 > α----------------------------------------Unacceptable

Inter-method reliability- is the variation in measurements of the same target when taken by a different methods or instruments, but with the same person

Validity Validity is the degree of closeness of measurements of a quantity to that quantity's actual (true) value (i.e. Does it measure what you think it measures. This is more familiarly called Construct Validity.

Types of Construct Validity Translation validity Face validity Content validity Criterion-related validity (Known groups validity) Predictive validity Concurrent validity Convergent validity Discriminant validity

Translation validity

Face Validity “On its face" does it seems like a good translation of the construct. i.e. Does it appear to measure what it is supposed to measure? Weak Version: If you read it does it appear to ask questions directed at the concept. Strong Version: If experts in that domain assess it, they conclude it measures that domain.

Content Validity How well elements of the test relate to the content domain? For example, a depression measure should cover the checklist of depression symptoms

Criteria-Related Validity (Known groups validity) involves the correlation between the test and a criterion variable (or variables)

Predictive Validity A high correlation would provide evidence for predictive validity – i.e. it would show that our measure can correctly predict something that we theoretically thing it should be able to predict.

Concurrent Validity Assess the operationalization's ability to distinguish between groups that it should theoretically be able to distinguish between. As in any discriminating test, the results are more powerful if you are able to show that you can discriminate between two groups that are very similar.

Convergent Validity Examine the degree to which the operationalization is similar to (converges on) other operationalizations that it theoretically should be similar to. To show the convergent validity of a test of arithmetic skills, one might correlate the scores on a test with scores on other tests that purport to measure basic math ability, where high correlations would be evidence of convergent validity.

Discriminant Validity Examine the degree to which the operationalization is not similar to (diverges from) other operationalizations that it theoretically should be not be similar to. To show the discriminant validity of a test of arithmetic skills, we might correlate the scores on a test with scores on tests that of verbal ability, where low correlations would be evidence of discriminant validity.

Conclusion validity

Differentiate between prevalence and incidence Prevalence: The total number of cases of a disease in a given population at a specific time Incidence:  the number of new cases of a specific disease occurring during a certain period in a population at risk.

odds ratio odds ratio: the ratio, used particularly in case-control studies or cohort study (exposed versus non-exposed), estimates the chances of a particular event occurring in one population in relation to its rate of occurrence in another population. Odds ratio= a/b / c/d Odds ratio= a*d/c*b a b a/b c d c/d

a b a/b c d c/d Relative risk is a ratio of the probability of the event occurring in the exposed group versus a non-exposed group Here, a = 20, b = 80, c = 1, and d = 99. Then the relative risk of cancer associated with smoking would be Smokers would be twenty times as likely as non-smokers to develop lung cancer.

Smokers are ???? times more likely to have lung cancer than non-smokers