2 CHAPTER OBJECTIVES STUDENTS SHOULD BE ABLE TO: Explain why measurement is important to the research process.Discuss the four levels of measurement and provide an example of each.Explain the concept of reliability in terms of observed score, true score, and error.Describe the two elements that can make up an error score.List methods for increasing reliability.Discuss four ways in which reliability can be examined.Provide a conceptual definition of validity.List the three traditional types of validity.Explain the relationship between reliability and validity.
3 CHAPTER OVERVIEW The Measurement Process Levels of Measurement Reliability and Validity: Why They Are Very, Very ImportantValidityThe Relationship Between Reliability and ValidityClosing (and Very Important) Thoughts
5 THE MEASUREMENT PROCESS Two definitionsStevens—“assignment of numerals to objects or events according to rules.”“…the assignment of values to outcomes.”Chapter fociLevels of measurementReliability and validity
6 LEVELS OF MEASUREMENTLevel of MeasurementFor ExampleQuality of LevelRatioRachael is 5’ 10” and Gregory is 5’ 5”Absolute zeroIntervalRachael is 5” taller than GregoryAn inch is an inch is an inchOrdinalRachael is taller than GregoryGreater thanNominalRachael is tall and Gregory is shortDifferent fromVariables are measured at one of these four levelsQualities of one level are characteristic of the next level upThe more precise (higher) the level of measurement, the more accurate is the measurement process
7 NOMINAL SCALE Qualities Example What You Can Say What You Can’t Say NOMINAL SCALEQualitiesExampleWhat You Can SayWhat You Can’t SayAssignment of labelsGender—(male orfemale)Preference—(like or dislike)Voting record—(for oragainst)Each observation belongs in its own categoryAn observation represents “more” or “less” than another observation
8 ORDINAL SCALE Qualities Example What You Can Say What You Can’t Say Assignment of values along some underlying dimensionRank in collegeOrder of finishing a raceOne observation is ranked above or below another.The amount that one variable is more or less than another
9 INTERVAL SCALE Qualities Example What You Can Say What You Can’t Say Equal distances between pointsNumber of words spelled correctlyIntelligence test scoresTemperatureOne score differs from another on some measure that has equally appearing intervalsThe amount of difference is an exact representation of differences of the variable being studied
10 RATIO SCALE Qualities Example What You Can Say What You Can’t Say Meaningful and non-arbitrary zeroAgeWeightTimeOne value is twice as much as another or no quantity of that variable can existNot much!
11 CONTINUOUS VERSUS DISCRETE VARIABLES Continuous variablesValues can range along a continuumE.g., heightDiscrete variables (categorical)Values are defined by category boundariesE.g., gender
12 WHAT IS ALL THE FUSS? Measurement should be as precise as possible In psychology, most variables are probably measured at the nominal or ordinal levelBut—how a variable is measured can determine the level of precision
13 RELIABILITY AND VALIDITY: WHY THEY ARE VERY, VERY IMPORTANT
14 RELIABILITY AND VALIDITY Reliability—tool is consistentValidity—tool measures “what-it-should”Good assessment tools Rejection of Null hypothesesORAcceptance of Research hypotheses
15 A CONCEPTUAL DEFINITION OF RELIABILITY Method ErrorObserved Score = True Score + Error ScoreTrait Error
16 A CONCEPTUAL DEFINITION OF RELIABILITY Method ErrorObserved Score = True Score + Error ScoreTrait ErrorObserved scoreScore actually observedConsists of two componentsTrue ScoreError Score
17 A CONCEPTUAL DEFINITION OF RELIABILITY Method ErrorObserved Score = True Score + Error ScoreTrait ErrorTrue scorePerfect reflection of true value for individualTheoretical score
18 A CONCEPTUAL DEFINITION OF RELIABILITY Method ErrorObserved Score = True Score + Error ScoreTrait ErrorError scoreDifference between observed and true score
19 A CONCEPTUAL DEFINITION OF RELIABILITY Method ErrorObserved Score = True Score + Error ScoreTrait ErrorMethod error is due to characteristics of the test or testing situationTrait error is due to individual characteristicsConceptually, reliability =Reliability of the observed score becomes higher if error is reduced!!True ScoreTrue Score + Error Score
20 INCREASING RELIABILITY Decreasing Error Increase sample sizeEliminate unclear questionsStandardize testing conditionsModerate the degree of difficulty of the testsMinimize the effects of external eventsStandardize instructionsMaintain consistent scoring procedures
21 HOW RELIABILITY IS MEASURED Reliability is measured using aCorrelation coefficientr test1•test2Reliability coefficientsIndicate how scores on one test change relative to scores on a second testCan range from -1.0 to +1.0+1.00 = perfect reliability0.00 = no reliability
22 TYPES OF RELIABILITY Type of Reliability What It Is How You Do It What the Reliability Coefficient Looks LikeTest-RetestA measure of stabilityAdminister the same test/measure at two different times to the same group of participantsrtest1•test1Parallel FormsA measure of equivalenceAdminister two different forms of the same test to the same group of participantsrform1•form2Inter-RaterA measure of agreementHave two raters rate behaviors and then determine the amount of agreement between themPercentage of agreementsInternal ConsistencyA measure of how consistently each item measures the same underlying constructCorrelate performance on each item with overall performance across participantsCronbach’s alphaKuder-Richardson
24 VALIDITY A valid test does what it was designed to do A valid test measures what it was designed to measure
25 A CONCEPTUAL DEFINITION OF VALIDITY Validity refers to the test’s results, not to the test itselfValidity ranges from low to high, it is not “either/or”Validity must be interpreted within the testing context
26 TYPES OF VALIDITY Type of Validity What Is It? How Do You Establish It?ContentA measure of how well the items represent the entire universe of itemsAsk an expert if the items assess what you want them toCriterionConcurrentA measure of how well a test estimates a criterionSelect a criterion and correlate scores on the test with scores on the criterion in the presentPredictiveA measure of how well a test predicts a criterionSelect a criterion and correlate scores on the test with scores on the criterion in the futureConstructA measure of how well a test assesses some underlying constructAssess the underlying construct on which the test is based and correlate these scores with the test scores
27 HOW TO ESTABLISH CONSTRUCT VALIDITY OF A NEW TEST Correlate new test with an established testShow that people with and without certain traits score differentlyDetermine whether tasks required on test are consistent with theory guiding test development
28 MULTITRAIT-MULTIMETHOD MATRIX ImpulsivityTrait 2Activity LevelMethod 1Paper and PencilMethod 2Activity Level MonitorTrait 1ModerateLowImpulsivityTrait 2Activity LevelConvergent validity—different methods yield similar resultsDiscriminant validity—different methods yield different results
29 THE RELATIONSHIP BETWEEN RELIABILITY AND VALIDITY
30 THE RELATIONSHIP BETWEEN RELIABILITY AND VALIDITY A valid test must be reliableButA reliable test need not be valid
32 CLOSING (AND VERY IMPORTANT) THOUGHTS You must define a reliable and valid dependent variable or you will not know whether or not there truly is no difference between groups!Use a test with established and acceptable levels of reliability and validity.If you cannot do this, develop such a test for your thesis or dissertation (and do no more than that) OR change what you are measuring.
33 HAVE WE MET THE OBJECTIVES? CAN YOU: Explain why measurement is important to the research process?Discuss the four levels of measurement and provide an example of each?Explain the concept of reliability in terms of observed score, true score, and error?Describe the two elements that can make up an error score?List methods for increasing reliability?Discuss four ways in which reliability can be examined?Provide a conceptual definition of validity?List the three traditional types of validity?Explain the relationship between reliability and validity?