Copyright © Allyn & Bacon (2007) Data and the Nature of Measurement Graziano and Raulin Research Methods: Chapter 4 This multimedia product and its contents are protected under copyright law. The following are prohibited by law: (1) Any public performance or display, including transmission of any image over a network; (2) Preparation of any derivative work, including the extraction, in whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2007) Research Variables Variable: Any characteristic that can take on more than one value Variable: Any characteristic that can take on more than one value –Examples: speed, level of hostility, accuracy of feedback, reaction time Research is the study of the relationship among variables Research is the study of the relationship among variables –Therefore, there must be at least two variables in a research study (or there is no relationship to study)

Copyright © Allyn & Bacon (2007) Measuring Variables Measurement: Assigning numbers to indicate the level of a variable Measurement: Assigning numbers to indicate the level of a variable –Sometimes the number assignment is easy to understand (e.g., time measured in seconds) –Sometimes it is more arbitrary (e.g., 1 for male and 2 for female)

Copyright © Allyn & Bacon (2007) Scales of Measurement Based on how closely the measurement scale matches the real number system Based on how closely the measurement scale matches the real number system Scales of Measurement (Stevens, 1946, 1957) Scales of Measurement (Stevens, 1946, 1957) –Nominal –Ordinal –Interval –Ratio

Copyright © Allyn & Bacon (2007) Nominal Scales Naming scale Naming scale –Each number reflects a category –Examples: diagnostic categories, political affiliations Produces nominal or categorical data Produces nominal or categorical data Mathematical properties Mathematical properties –Identity

Copyright © Allyn & Bacon (2007) Ordinal Scales Scale indicating rank order Scale indicating rank order –Reflects the order, but not the amount Example: order of finish in a race, class rankings Produces ordered data Produces ordered data Mathematical properties Mathematical properties –Identity –Magnitude

Copyright © Allyn & Bacon (2007) Interval Scales Scale with equal intervals Scale with equal intervals –The scale indicates amount, but with no zero point –Examples: temperature on the Celsius scale, most psychological tests Produces score data Produces score data Mathematical properties Mathematical properties –Identity –Magnitude –Equal intervals

Copyright © Allyn & Bacon (2007) Ratio Scales Scale that fits the number system well Scale that fits the number system well –Includes equal intervals and a true zero –Examples: time, distance, frequency Produces score data Produces score data Mathematical properties Mathematical properties –Identity –Magnitude –Equal intervals –True zero

Copyright © Allyn & Bacon (2007) Psychological Tests Most psychological test fall somewhere between an ordinal and a ratio scale Most psychological test fall somewhere between an ordinal and a ratio scale –Ordinal in that the distance between scores may not be equal –Ratio in that one can view the test score as the number of correct items Norm is to assume such tests represent an interval scale Norm is to assume such tests represent an interval scale

Copyright © Allyn & Bacon (2007) Measurement Error Decreases the accuracy of measurement Decreases the accuracy of measurement Possible sources of measurement error Possible sources of measurement error –Response set biases –Inconsistent measurement procedures –Sloppy procedures –Unreliable measures

Copyright © Allyn & Bacon (2007) Operational Definitions Specific procedures for measuring and/or manipulating a variable Specific procedures for measuring and/or manipulating a variable Every variable should be operationally defined Every variable should be operationally defined The more careful and complete the operational definition, the more precise the measurement of the variable The more careful and complete the operational definition, the more precise the measurement of the variable

Copyright © Allyn & Bacon (2007) Reliability The consistency of measurement The consistency of measurement Consistency can be conceptualized in different ways Consistency can be conceptualized in different ways –Therefore, there are different types of reliability Usually measured with a correlation Usually measured with a correlation –Covered in Chapter 5 –Sensitive to the consistency of rank orderings of participants

Copyright © Allyn & Bacon (2007) Types of Reliability Interrater reliability: degree of agreement between two independent raters Interrater reliability: degree of agreement between two independent raters Test-retest reliability: degree of consistency over time Test-retest reliability: degree of consistency over time Internal consistency reliability: degree to which the items of a measure all measure the same thing Internal consistency reliability: degree to which the items of a measure all measure the same thing

Copyright © Allyn & Bacon (2007) Perfect Reliability Reliability is a measure of consistency. Reliability is a measure of consistency. Perfect reliability means that the scores are perfectly consistent. Perfect reliability means that the scores are perfectly consistent.

Copyright © Allyn & Bacon (2007) Good Reliability Reliability deteriorates when the consistency is lost. Reliability deteriorates when the consistency is lost. Here the rank orderings are still reasonably consistent. Here the rank orderings are still reasonably consistent.

Copyright © Allyn & Bacon (2007) Fair Reliability As reliability deteriorates further, the rank orderings from the two testings are less similar. As reliability deteriorates further, the rank orderings from the two testings are less similar.

Copyright © Allyn & Bacon (2007) Poor Reliability When you reach the point where the rank orderings have no relationship to one another, your reliability (i.e., consistency) is poor. When you reach the point where the rank orderings have no relationship to one another, your reliability (i.e., consistency) is poor.

Copyright © Allyn & Bacon (2007) Effective Range The range in which a measure gives an accurate indication of the level of the variable The range in which a measure gives an accurate indication of the level of the variable Critical to select measures with an effective range appropriate for Critical to select measures with an effective range appropriate for –Your sample –Your study –Example: Using a calculus test to measure math skills in second graders would NOT work

Copyright © Allyn & Bacon (2007) Scale Attenuation Effects Results from a restriction of the range of the measure Results from a restriction of the range of the measure –Therefore, people above or below the effective range are not measured accurately Two types of scale attenuation effects Two types of scale attenuation effects –Floor effects –Ceiling effects

Copyright © Allyn & Bacon (2007) Illustrating These Effects Situation Situation –Assume that we are measuring the weight of 9 men using a standard bathroom scale Floor Effect Floor Effect –The needle is stuck so that it never reads below 175 Ceiling Effect Ceiling Effect –The needle is stuck so that it never reads above 200

Copyright © Allyn & Bacon (2007) Weights for 9 Men

Copyright © Allyn & Bacon (2007) Floor Effect Range on the bottom of the scale is insufficient to measure people who score near the bottom Range on the bottom of the scale is insufficient to measure people who score near the bottom If the scale needle was stuck so that it never read below 175 pounds If the scale needle was stuck so that it never read below 175 pounds –There would be a floor at 175 Error due to this floor effect is shown in dark blue in the next slide Error due to this floor effect is shown in dark blue in the next slide –Weights recorded for these people are too high

Copyright © Allyn & Bacon (2007) Floor Effect

Copyright © Allyn & Bacon (2007) Ceiling Effect The range on the top of the scale is insufficient to measure people who score near the top The range on the top of the scale is insufficient to measure people who score near the top If the scale could not read above 200 pounds If the scale could not read above 200 pounds –There would be a ceiling at 200 (everyone above 200 would be reported as weighing 200) Error due to this ceiling effect is shown in dark blue in the next slide Error due to this ceiling effect is shown in dark blue in the next slide

Copyright © Allyn & Bacon (2007) Ceiling Effect

Copyright © Allyn & Bacon (2007) Source of These Effects Result of psychological scale with an insufficient range to measure the range of performance in the sample Result of psychological scale with an insufficient range to measure the range of performance in the sample –Example: a measure of memory that is either too easy or too hard

Copyright © Allyn & Bacon (2007) Validity A scale is valid if it measures what it is supposed to measure A scale is valid if it measures what it is supposed to measure Validity also refers to how well a scale predicts other variables Validity also refers to how well a scale predicts other variables –Example: An IQ test is likely to be a valid predictor of grades in school. –When used this way, the scale is called the predictor measure and the measure predicted is called the criterion

Copyright © Allyn & Bacon (2007) Measuring Validity Validity is usually measured with a correlation Validity is usually measured with a correlation –The correlation is between The measure and The measure and A SPECIFIED criterion measure A SPECIFIED criterion measure –Always list the criterion when reporting the level of validity For example, the IQ test is a valid predictor of school grades, but not a valid predictor of athletic ability. For example, the IQ test is a valid predictor of school grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2007) Perfect Validity Validity is the degree to which one measure predicts another. Validity is the degree to which one measure predicts another. With perfect validity, the rank orderings on the predictor and criterion measures are identical. With perfect validity, the rank orderings on the predictor and criterion measures are identical.

Copyright © Allyn & Bacon (2007) Good Validity If the rank orderings from the predictor and criterion measures are similar, you have good validity. If the rank orderings from the predictor and criterion measures are similar, you have good validity.

Copyright © Allyn & Bacon (2007) Fair Validity But the less the rank orderings from the predictor and criterion measures agree, the lower the validity. But the less the rank orderings from the predictor and criterion measures agree, the lower the validity.

Copyright © Allyn & Bacon (2007) Poor Validity When you reach the point that the rank orderings of predictor and criterion measures are unrelated, you have essentially zero validity. When you reach the point that the rank orderings of predictor and criterion measures are unrelated, you have essentially zero validity.

Copyright © Allyn & Bacon (2007) Objective Measurement The hallmark of science The hallmark of science –Objective measures produce the same result no matter who does the measuring –Therefore, scientific principles will apply no matter who tests these principles Objective measures reduce biases that could distort results (see Chapters 8 and 9) Objective measures reduce biases that could distort results (see Chapters 8 and 9)

Copyright © Allyn & Bacon (2007) Summary Measuring variables is central to research Measuring variables is central to research Several scales of measurement exist Several scales of measurement exist Reduce measurement error with carefully developed operational definitions Reduce measurement error with carefully developed operational definitions Enhance reliability and validity of your measures Enhance reliability and validity of your measures The goal is to produce objective, accurate measures of your variables The goal is to produce objective, accurate measures of your variables