Measurement in Exercise and Sport Psychology Research EPHE 348

Measurement We measure performance on a variable – an attribute that different people possess in different amounts Measurement is not specifically defined but has levels and properties

Levels of Measurement ( from least rigorous to most rigorous ) Nominal – numbers are used to classify (gender, eye color) Ordinal – numbers have an order (ranking in finishing a race) Interval – equal differences in numbers imply equal differences in attributes (calendar time) Ratio – interval with a zero point reflecting the absence of the attribute (height)

Basic Measurement Theory An observed score on a measure is made up of two components: –1) true score –2) error Error may have a systematic component and a random component –examples of systematic error: social desirability, gender bias, cultural bias etc.

Thinking more about Error....In measurement theory we assume: -1) Error is random across items -2) Error is independent across items -3) Error has a normal distribution with a mean of 0 (cancels itself out)

Key Point in Scale Construction Multiple items generally are needed in a measure to approximate the “true score” and reduce random error towards 0

Most common Types of Reliability Assessment 1) Internal consistency of a scale –Purpose: to try to identify the commonality of variability in the scale items and interpret it as “true score variance” Examples: –split half –Cronbach’s 

Most common Types of Reliability Assessment 2) Comparison between time and testers –Purpose: To identify consistency across time, and across testers Examples: -test-retest reliability -inter-rater reliability

Factors that Influence Reliability 1) Heterogeneity of the construct 2) Method of estimation 3) Number of items 4) Variability in the participants that answer the measure

Validity Are we even measuring what we want to measure? If you do not know what your measurement means, you do not know anything (Gulliksen, 1950) Validity is a matter of degree and we must gather multiple lines of evidence

The Holy Trinity of Validity -1) Content validity – based on professional judgements about the relevance of the item content to the content of a particular domain of interest and about the representativeness with which the item covers that domain -2) Criterion validity – based on the degree of empirical relationships, usually correlations between the scale and criterions

The Holy Trinity of Validity -3) Construct validity – evaluated by investigating what qualities a scale measures All evidence for and against a measure is actually construct validity (Rogers, 1994)

Threats to Construct Validity 1) Construct-irrelevant variance –including something that should not have been included 2) Construct under representation –leaving out something that should be included according to the theory surrounding the construct of interest

Constructing A Measure Phase One: Item Construction –strive for representativeness and relevance to the domain of interest –items should be clear, short and simple –items should not have double meaning (conjunctions) –avoid items that are endorsed by no one or everyone

Constructing a Measure –items should be balanced in positive and negative wording –arrange items in random order General problems: – Social desirability – tendency to always answer favorably –Acquiescence - tendency to agree

Constructing a Measure Phase 2: Judges Analysis –Items are sent to expert judges in the domain and evaluated for relevance and representativness –The more judges the better –Evaluation is rated objectively and with room for comments

Constructing a Measure Phase 3: Protocol Analysis –Items are examined by a sample of participants in a think out loud procedure and focus group –Helps identify any differences in meaning between the experts and the population sample

Constructing a Measure Phase 3: Structure Analysis –The measure is administered to a population with very similar and very different measures –Can then examine: 1) structure of the measure 2) Divergence from the very different measures 3) Convergence with the similar measure

Exercise In a group of five, develop a five item measure Consider –Representativeness & relevance –Phrase simplicity & variance –Type of scaling