Measuring Research Variables KNES 510 Research Methods in Kinesiology 1

Evidence of Validity The American Educational Research Association and American Psychological Association agree on the definition of four types of validity: Logical ValidityLogical Validity Content ValidityContent Validity Criterion ValidityCriterion Validity –Concurrent Validity –Predictive Validity Construct ValidityConstruct Validity 2

Logical or Face Validity What is logical or face validity?What is logical or face validity? Logical validity is determined subjectivelyLogical validity is determined subjectively What are some examples of logical validity?What are some examples of logical validity? 3

Content Validity What is content validity?What is content validity? A test which has content validity should adequately measure the skills and/or material which has been presented in classA test which has content validity should adequately measure the skills and/or material which has been presented in class 4

Concurrent Validity What is concurrent validity?What is concurrent validity? Determined by correlating scores on a test with scores on a criterion measureDetermined by correlating scores on a test with scores on a criterion measure The resulting correlation coefficient is called a validity coefficientThe resulting correlation coefficient is called a validity coefficient 5

Examples of Concurrent Validity VO 2 max (criterion: oxygen consumption)VO 2 max (criterion: oxygen consumption) –Distance runs (e.g., 1.0-mile, 1.5-mile, 9- minute, 12-minute, 20-minute shuttle) –Submaximal (e.g., cycle, treadmill, swimming) –Nonexercise models (e.g. self-reported physical activity) 6

Body fat (criterion: hydrostatically determined body fat)Body fat (criterion: hydrostatically determined body fat) –Skinfolds –Anthropometric measures Sport skills (criterion: game performance, expert ratings)Sport skills (criterion: game performance, expert ratings) –Sport skills tests 7

Predictive Validity What is predictive validity?What is predictive validity? When, and why, are we interested in determining predictive validity?When, and why, are we interested in determining predictive validity? 8

Examples of Predictive Validity Heart disease (criterion: heart disease developed in later life)Heart disease (criterion: heart disease developed in later life) –Present diet, exercise behaviors, blood pressure, family history Success in graduate school (criterion: grade-point average or graduation status)Success in graduate school (criterion: grade-point average or graduation status) –Graduate Record Examination scores –Undergraduate grade-point average 9

Job capabilities (criterion: successful job performance)Job capabilities (criterion: successful job performance) –Physical abilities –Cognitive abilities Predictive validity is established by correlating the scores on a test with scores on another test in the futurePredictive validity is established by correlating the scores on a test with scores on another test in the future 10

Construct Validity What is construct validity?What is construct validity? Construct validity is used with abstract rather than concrete testsConstruct validity is used with abstract rather than concrete tests An abstract test measures something that is not directly observableAn abstract test measures something that is not directly observable 11

Examples of abstract measures:Examples of abstract measures: –Attitudes –Personality characteristics –Other unobservable yet theoretically existing traits 12

Construct validity is established by finding two groups known to differ on the variable, or construct, being testedConstruct validity is established by finding two groups known to differ on the variable, or construct, being tested The test is then administered to both groups to determine if there is a significant difference between the scores for the two groupsThe test is then administered to both groups to determine if there is a significant difference between the scores for the two groups This is the known group difference methodThis is the known group difference method 13

Reliability Reliability – the degree of consistency with which a test measures what it measuresReliability – the degree of consistency with which a test measures what it measures In order to be valid, a test must also be reliableIn order to be valid, a test must also be reliable Observed score = True score + Error scoreObserved score = True score + Error score 14

Types of Reliability Stability Reliability – the degree to which an individual's scores are unchanged from day to dayStability Reliability – the degree to which an individual's scores are unchanged from day to day We use the test-retest method to obtain the stability reliability coefficientWe use the test-retest method to obtain the stability reliability coefficient 15

Each person is measured with the same test or instrument on several (usually 2) different days (Day 1, Day 2, and so on)Each person is measured with the same test or instrument on several (usually 2) different days (Day 1, Day 2, and so on) The correlation between the two sets of scores is the stability reliability coefficientThe correlation between the two sets of scores is the stability reliability coefficient The closer this coefficient is to positive one (+1.0), the more stable and reliable the scoresThe closer this coefficient is to positive one (+1.0), the more stable and reliable the scores 16

Three factors can contribute to poor score stability (a low stability reliability coefficient): 1.the people tested may perform differently 2.the measuring instrument may operate or be applied differently 3.the person administering the measurement may change 17

Internal-Consistency Reliability – the degree to which an individual's scores are unchanged within a dayInternal-Consistency Reliability – the degree to which an individual's scores are unchanged within a day We use the multiple trials within a day method to obtain the internal-consistency reliability coefficientWe use the multiple trials within a day method to obtain the internal-consistency reliability coefficient 18

To obtain an internal-consistency reliability coefficient, the evaluator must give at least 2 trials of the test within a single dayTo obtain an internal-consistency reliability coefficient, the evaluator must give at least 2 trials of the test within a single day Change in the scores of the people being tested from trial to trial indicates a lack of test reliabilityChange in the scores of the people being tested from trial to trial indicates a lack of test reliability 19

The correlation among the trial scores is the internal-consistency reliability coefficientThe correlation among the trial scores is the internal-consistency reliability coefficient What types of tests should not be evaluated for reliability using this method?What types of tests should not be evaluated for reliability using this method? 20

Stability versus Internal Consistency The internal-consistency reliability coefficient is usually higher than the stability reliability coefficientThe internal-consistency reliability coefficient is usually higher than the stability reliability coefficient With the test-retest method some learning or increase in performance will usually occur, despite the fact that it is presumed that ability will not changeWith the test-retest method some learning or increase in performance will usually occur, despite the fact that it is presumed that ability will not change 21

After completing a test for the first time, subjects will often perform better on the second administrationAfter completing a test for the first time, subjects will often perform better on the second administration This improvement can be referred to as a learning effectThis improvement can be referred to as a learning effect The learning effect is a threat to reliabilityThe learning effect is a threat to reliability How do we avoid this problem?How do we avoid this problem? 22

Methods of Calculating a Reliability Coefficient Pearson’s rPearson’s r Intraclass R from One-Way ANOVAIntraclass R from One-Way ANOVA Cronbach’s AlphaCronbach’s Alpha 23

SPSS Interclass Reliability Output 24

SPSS Output for Intraclass Reliability Analysis (Cronbach’s Alpha) Here is the SPPS output with the value for Cronbach’s alpha:Here is the SPPS output with the value for Cronbach’s alpha: 25

SPSS Output for Intraclass R from One-Way ANOVA Table This ANOVA table may be used to calculate the intraclass RThis ANOVA table may be used to calculate the intraclass R R = ( – 0.333) / = 0.989R = ( – 0.333) / =

Acceptable Reliability R = is below-averageR = is below-average R = is averageR = is average R = is above-averageR = is above-average 27

Intertester Reliability (Objectivity) Objectivity (rater reliability) – the degree to which multiple scorers agree on the magnitude of scoresObjectivity (rater reliability) – the degree to which multiple scorers agree on the magnitude of scores How can the objectivity of a test be improved?How can the objectivity of a test be improved? 28

Next Class Mock proposals 29