1. Measurement and Data Quality

2. Measurement
The assignment of numbers to represent the amount of an attribute present in an object or person, using specific rules
Advantages:
Removes guesswork
Provides precise information
Less vague than words

3. Levels of Measurement
There are four levels (classes) of measurement:
Nominal (assigning numbers to classify characteristics into categories) Gender, religion
Ordinal (ranking objects based on their relative standing on an attribute) "very dissatisfied," "somewhat dissatisfied," "somewhat satisfied," or "very satisfied."
Interval (objects ordered on a scale that has equal distances between points on the scale) Fahrenheit scale of temperature
Ratio (equal distances between score units; there is a rational, meaningful zero) amount of money you have in your pocket right now
A variable’s level of measurement determines what mathematic operations can be performed in a statistical analysis.

4. Errors of Measurement Obtained Score = True score ± Error
Obtained score: An actual data value for a participant (e.g., anxiety scale score)
True score: The score that would be obtained with an infallible measure
Error: The error of measurement, caused by factors that distort measurement

5. Factors That Contribute to Errors of Measurement
Situational contaminants
Transitory personal factors (e.g., fatigue)
Response-set biases
Administration variations
Item sampling

6. Question Is the following statement True or False?
The true score is data obtained from the actual research study.

7. Answer
False
The true score is the score that would be obtained with an infallible measure. The obtained score is an actual value (datum) for a participant.

8. Psychometric Assessments
A psychometric assessment is an evaluation of the quality of a measuring instrument.
Key criteria in a psychometric assessment:
Reliability
Validity

9. Reliability
The consistency and accuracy with which an instrument measures the target attribute
Reliability assessments involve computing a reliability coefficient.
Reliability coefficients can range from .00 to 1.00.
Coefficients below .70 are considered unsatisfactory.
Coefficients of .80 or higher are desirable.

10. Three Aspects of Reliability Can Be Evaluated
Stability
Internal consistency
Equivalence

11. Stability
The extent to which scores are similar on two separate administrations of an instrument
Evaluated by test–retest reliability
Requires participants to complete the same instrument on two occasions
Appropriate for relatively enduring attributes (e.g., creativity)

12. Internal Consistency
The extent to which all the items on an instrument are measuring the same unitary attribute
Evaluated by administering instrument on one occasion
Appropriate for most multi-item instruments
The most widely used approach to assessing reliability
Assessed by computing coefficient alpha (Cronbach’s alpha)
Alphas ≥.80 are highly desirable.

13. Question
When determining the reliability of a measurement tool, which value would indicate that the tool is most reliable?
0.50
0.70
0.90
1.10

14. Answer
c. 0.90
Reliability coefficients can range from 0.0 to Coefficients of 0.80 or higher are desirable. Thus, a coefficient of 0.90 would indicate that the tool is very reliable. A value greater than 1.00 for a coefficient would be an error.

15. Equivalence
The degree of similarity between alternative forms of an instrument or between multiple raters/observers using an instrument
Most relevant for structured observations
Assessed by comparing agreement between observations or ratings of two or more observers (interobserver/interrater reliability)

16. Reliability Principles
Low reliability can undermine adequate testing of hypotheses.
Reliability estimates vary depending on procedure used to obtain them.
Reliability is lower in homogeneous than heterogeneous samples.
Reliability is lower in shorter than longer multi-item scales.

17. Validity
The degree to which an instrument measures what it is supposed to measure
Four aspects of validity:
Face validity
Content validity
Criterion-related validity
Construct validity

18. Face Validity
Refers to whether the instrument looks as though it is an appropriate measure of the construct
Based on judgment; no objective criteria for assessment

19. Content Validity
The degree to which an instrument has an adequate sample of items for the construct being measured
Evaluated by expert evaluation, often via a quantitative measure—the content validity index (CVI)

20. Question Is the following statement True or False?
Face validity of an instrument is based on judgment.

21. Answer
True
Face validity refers to whether the instrument looks like it is an appropriate measure of the construct. There are no objective criteria for assessment; it is based on judgment.

22. Criterion-Related Validity
The degree to which the instrument is related to an external criterion
Validity coefficient is calculated by analyzing the relationship between scores on the instrument and the criterion.
Two types:
Predictive validity: the instrument’s ability to distinguish people whose performance differs on a future criterion
Concurrent validity: the instrument’s ability to distinguish individuals who differ on a present criterion

23. Construct Validity Concerned with these questions:
What is this instrument really measuring?
Does it adequately measure the construct of interest?

24. Some Methods of Assessing Construct Validity
Known-groups technique
Testing relationships based on theoretical predictions
Factor analysis

25. Criteria for Assessing Screening/Diagnostic Instruments
Sensitivity: the instruments’ ability to correctly identify a “case”—i.e., to diagnose a condition
Specificity: the instrument’s ability to correctly identify noncases, that is, to screen out those without the condition
Likelihood ratio: Summarizes the relationship between sensitivity and specificity in a single number
LR+: the ratio of true positives to false positives
LR-: the ratio of false negatives to true negatives