1. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 17 Assessing Measurement Quality in Quantitative Studies

2. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Measurement The assignment of numbers to represent the amount of an attribute present in an object or person, using specific rules Advantages: –Removes guesswork –Provides precise information –Less vague than words

3. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Errors of Measurement Obtained Score = True score + Error Obtained score: An actual data value for a participant (e.g., anxiety scale score) True score: The score that would be obtained with an infallible measure Error: The error of measurement, caused by factors that distort measurement

4. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Factors That Contribute to Errors of Measurement Situational contaminants Transitory personal factors Response-set biases Administration variations Problems with instrument clarity Item sampling Instrument format

5. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Key Criteria for Evaluating Quantitative Measures Reliability Validity

6. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Reliability The consistency and accuracy with which an instrument measures the target attribute Reliability assessments involve computing a reliability coefficient –most reliability coefficients are based on correlation coefficients

7. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Correlation Coefficients Correlation coefficients indicate direction and magnitude of relationships between variables Range  from –1.00 (perfect negative correlation)  through 0.00 (no correlation)  to +1.00 (perfect positive correlation)

8. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Three Aspects of Reliability Can Be Evaluated Stability Internal consistency Equivalence

9. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Stability The extent to which scores are similar on 2 separate administrations of an instrument Evaluated by test–retest reliability –Requires participants to complete the same instrument on two occasions –A correlation coefficient between scores on 1 st and 2 nd administration is computed –Appropriate for relatively enduring attributes (e.g., self-esteem)

10. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Internal Consistency The extent to which all the instrument’s items are measuring the same attribute Evaluated by administering instrument on one occasion Appropriate for most multi-item instruments Evaluation methods: –Split-half technique –Coefficient alpha

11. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Equivalence The degree of similarity between alternative forms of an instrument or between multiple raters/observers using an instrument Most relevant for structured observations Assessed by comparing observations or ratings of 2 or more observers (interobserver/interrater reliability) Numerous formula and assessment methods

12. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Reliability Coefficients Represent the proportion of true variability to obtained variability: r =V T V o Should be at least.70;.80 preferable Can be improved by making instrument longer (adding items) Are lower in homogeneous than in heterogeneous samples

13. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Validity The degree to which an instrument measures what it is supposed to measure Four aspects of validity: –Face validity –Content validity –Criterion-related validity –Construct validity

14. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Face Validity Refers to whether the instrument looks as though it is measuring the appropriate construct Based on judgment, no objective criteria for assessment

15. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Content Validity The degree to which an instrument has an appropriate sample of items for the construct being measured Evaluated by expert evaluation, via the content validity index (CVI)

16. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Criterion-Related Validity The degree to which the instrument correlates with an external criterion Validity coefficient is calculated by correlating scores on the instrument and the criterion

17. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Criterion-Related Validity (cont’d) Two types of criterion-related validity: Predictive validity: the instrument’s ability to distinguish people whose performance differs on a future criterion Concurrent validity: the instrument’s ability to distinguish individuals who differ on a present criterion

18. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Construct Validity Concerned with the questions: What is this instrument really measuring? Does it adequately measure the construct of interest?

19. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Methods of Assessing Construct Validity Known-groups technique Relationships based on theoretical predictions Multitrait-multimethod matrix method (MTMM) Factor analysis

20. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Multitrait-Multimethod Matrix Method Builds on two types of evidence: Convergence Discriminability

21. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Convergence Evidence that different methods of measuring a construct yield similar results Convergent validity comes from the correlations between two different methods measuring the same trait

22. Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Discriminabililty Evidence that the construct can be differentiated from other similar constructs Discriminant validity assesses the degree to which a single method of measuring two constructs yields different results