Reliability and Validity of Research Instruments An overview Reliability and Validity of Research Instruments

Measurement error Error variance--the extent of variability in test scores that is attributable to error rather than a true measure of behavior. Observed Score=true score + error variance (actual score obtained) (stable score) (chance/random error) (systematic error)

Validity The accuracy of the measure in reflecting the concept it is supposed to measure.

Reliability Stability and consistency of the measuring instrument. A measure can be reliable without being valid, but it cannot be valid without being reliable.

Validity The extent to which, and how well, a measure measures a concept. face content construct concurrent predictive criterion-related

Face validity Just on its face the instrument appears to be a good measure of the concept. “intuitive, arrived at through inspection” e.g. Concept=pain level Measure=verbal rating scale “rate your pain from 1 to 10”. Face validity is sometimes considered a subtype of content validity. Question--is there any time when face validity is not desirable?

Content validity Content of the measure is justified by other evidence, e.g. the literature. Entire range or universe of the construct is measured. Usually evaluated and scored by experts in the content area. A CVI (content validity index) of .80 or more is desirable.

Construct validity Sensitivity of the instrument to pick up minor variations in the concept being measured. Can an instrument to measure anxiety pick up different levels of anxiety or just its presence or absence? Measure two groups known to differ on the construct. Ways of arriving at construct validity Hypothesis testing method Convergent and divergent Multitrait-multimatrix method Contrasted groups approach factor analysis approach

Concurrent validity Correspondence of one measure of a phenomenon with another of the same construct.(administered at the same time) Two tools are used to measure the same concept and then a correlational analysis is performed. The tool which is already demonstrated to be valid is the “gold standard” with which the other measure must correlate.

Predictive validity The ability of one measure to predict another future measure of the same concept. If IQ predicts SAT, and SAT predicts QPA, then shouldn’t IQ predict QPA (we could skip SATs for admission decisions) If scores on a parenthood readiness scale indicate levels of integrity, trust, intimacy and identity couldn’t this test be used to predict successful achievement of the devleopmental tasks of adulthood? The researcher is usually looking for a more efficient way to measure a concept.

Criterion related validity The ability of a measure to measure a criterion (usually set by the researcher). If the criterion set for professionalism is nursing is belonging to nursing organizations and reading nursing journals, then couldn’t we just count memberships and subscriptions to come up with a professionalism score. Can you think of a simple criterion to measure leadership? Concurrent and predictive validity are often listed as forms of criterion related validity.

Reliability Homogeneity, equivalence and stability of a measure over time and subjects. The instrument yields the same results over repeated measures and subjects. Expressed as a correlation coefficient (degree of agreement between times and subjects) 0 to +1. Reliability coefficient expresses the relationship between error variance, true variance and the observed score. The higher the reliability coefficient, the lower the error variance. Hence, the higher the coefficient the more reliable the tool! .70 or higher acceptable.

Stability The same results are obtained over repeated administration of the instrument. Test-restest reliability parallel, equivalent or alternate forms

Test-Retest reliability The administration of the same instrument to the same subjects two or more times (under similar conditions--not before and after treatment) Scores are correlated and expressed as a Pearson r. (usually .70 acceptable)

Parallel or alternate forms reliability Parallel or alternate forms of a test are administered to the same individuals and scores are correlated. This is desirable when the researcher believes that repeated administration will result in “test-wiseness” Sample: ”I am able to tell my partner how I feel” “My partner tries to understand my feelings”

Homogeneity Internal consistency (unidimensional) Item-total correlations split-half reliability Kuder-Richardson coefficient Cronbach’s alpha

Item to total correlations Each item on an instrument is correlated to total score--an item with low correlation may be deleted. Highest and lowest correlations are usually reported. Only important if you desire homogeneity of items.

Spit Half reliability Items are divided into two halves and then compared. Odd, even items, or 1-50 and 51-100 are two ways to split items. Only important when homogenity and internal consistency is desirable.

Kuder-Richardson coefficient (KR-20) Estimate of homogeneity when items have a dichotomous response, e.g. “yes/no” items. Should be computed for a test on an initial reliability testing, and computed for the actual sample. Based on the consistency of responses to all of the items of a single form of a test.

Cronbach’s alpha Likert scale or linear graphic response format. Compares the consistency of response of all items on the scale. May need to be computed for each sample.

Equivalence Consistency of agreement of observers using the same measure or among alternate forms of a tool. Parallel or alternate forms (described under stability) Interrater reliability

Intertater reliability Used with observational data. Concordance between two or more observers scores of the same event or phenomenon.

Critiquing Was reliability and validity data presented and is it adequate? Was the appropriate method used? Was the reliability recalculated for the sample? Are the limitations of the tool discussed?