Reliability and Validity in Research Spring 2013 University of Missouri -St. Louis

Believing what you read? There is a need for reliable and valid data on student learning outcomes. Reliability: the extent to which an assessment tool is consistent or free from random error in measurement Validity concerns the degree to which inferences about students based on their test scores are warranted. Validity: the extent to which an assessment tool measures what it is intended to measure

Validity Validity has been defined as referring to the appropriateness, correctness, meaningfulness, and usefulness of the specific inferences researchers make based on the data they collect. It is the most important idea to consider when preparing or selecting an instrument. Validation is the process of collecting and analyzing evidence to support such inferences.

Evidence of Validity There are 3 types of evidence a researcher might collect: Content-related evidence of validity Content and format of the instrument Criterion-related evidence of validity Relationship between scores obtained using the instrument and scores obtained Construct-related evidence of validity Psychological construct being measured by the instrument

Content-related Evidence A key element is the adequacy of the sampling of the domain it is supposed to represent. The other aspect of content validation is the format of the instrument. Attempts to obtain evidence that the items measure what they are supposed to measure typify the process of content-related evidence.

Criterion-related Evidence A criterion is a second test presumed to measure the same variable. There are two forms of criterion-related validity: Predictive validity: time interval elapses between administering the instrument and obtaining criterion scores Concurrent validity: instrument data and criterion data are gathered and compared at the same time A Correlation Coefficient (r) indicates the degree of relationship that exists between the scores of individuals obtained by two instruments.

Construct-related Evidence Considered the broadest of the three categories. There is no single piece of evidence that satisfies construct-related validity. Researchers attempt to collect a variety of types of evidence, including both content-related and criterion-related evidence. The more evidence researchers have from different sources, the more confident they become about the interpretation of the instrument.

How can validity be established? Quantitative studies: measurements, scores, instruments used, research design Qualitative studies: ways that researchers have devised to establish credibility: member checking, triangulation, thick description, peer reviews, external audits

Reliability Refers to the consistency of scores or answers provided by an instrument. Scores obtained can be considered reliable but not valid. An instrument should be reliable and valid, depending on the context in which an instrument is used.

Reliability, continued In statistics or measurement theory, a measurement or test is considered reliable if it produces consistent results over repeated tests. Refers to how well we are measuring whatever it is that is being measured (regardless of whether or not it is the right quantity to measure).

Reliability, continued Unlike the common understanding, in these contexts “reliability” does not imply a value judgment Your car always starts/doesn’t start Your friend is always/ never late

Reliability of Measurement

Errors of Measurement Because errors of measurement are always present to some degree, variation in test scores are common. This is due to: Differences in motivation Energy Anxiety Different testing situation

Observational Studies Some characteristics cannot be measured through a test Unobtrusiveness Multiple sources of error Reliability depends on the extent to which observers agree

How can reliability be established? Quantitative studies? Assumption of repeatability Qualitative studies? Reframe as dependability and confirmability

Reliability and Validity

Reliability and Validity Why do we bother? Terms used in conjunction with one another Quantitative Research: R & V are treated as separate terms Qualitative Research: R & V are often all under another, all encompassing term Semi-reciprocal relationship