1. Chapter 5: Developing a Measurement Strategy
Scales of Measurement
Reliability and Validity
Modalities of Measurement
Locating and Evaluating Measures

2. Reliability & Validity
Why is reliability important?
Theory can’t do without it
Constructs can’t be valid unless they are reliable
A phenomenon must be reliably demonstrated before it can have construct status
E.g. ESP, Loch Ness Monster
What are some examples of phenomena important to:
Counselors?
I/O psychologists?

3. Reliability & Validity
Manifest variables:
Directly observable
Hypothetical Constructs:
Not directly observed (inferred)
What are some examples?
Behavioral?
Psychological constructs?
Which types (manifest/hypothetical) are:
Operational definitions?
Hypothetical constructs?
How are Operational definitions and Hypothetical constructs related?
Operational definitions represent hypothetical constructs

4. Reliability & Validity and measurement error
Reliability – consistency
Validity – accuracy
Polygraph
Reliable and valid?
Can something be reliable but not valid?
Can something be valid but not reliable?
Measurement Error
Xo= T+E (Observed score = True score + Error)
Randon v. systematic error
What kind of error is introduced when an applicant fakes on a personality measure? Other examples?

5. Assessing Reliability
Forms
Test-retest
Alternate forms (equivalent forms)
Interrater reliability
Internal consistency
Split half
Cronbach’s alpha (coefficient alpha)
mean r among all items
Which one to use?
Give some examples
Standards for reliability
What’s acceptable?

6. Assessing Validity Validity Inferred
Depends on different types of evidence
Varies in how much (degree)
Trichotomized into: high, moderate, low
Specific to what it’s used for (valid for what?)
refers to inferences drawn, not the measure itself
A unitary construct (with three types of evidence)
Content,
criterion-related (predictive, concurrent)
Construct

7. Validity Convergent Divergent
Related to other constructs it should be related to
Divergent
Not related to other constructs it should not be related to
which should be convergent and which divergent?
IQ and Depression
Work motivation and conscientiousness
Happiness and wealth
Need for approval and caring about how one looks
Ability to sell and friendliness
Aggression and frustration level

8. Validity: Determining Degree of Validity
Measure validation process
See figure 5-4
Theory of trait
Test hypotheses
Confirm/disconfirm
Do it again
Never ending (seemingly)
Eg. CPI, WGCTA, WPT, NEO, Beck DI

9. Validity: Differential
Valid for specific population, group?
Moral reasoning?
Different cultures, Gender (Gilligan, ’82)?
Math tests?
Gender?
PONS test?
Personality inventories?
Fakers v. honest?
Job setting v. home setting?

10. Reliability & Validity: Modalities of Measurement
Self-Report Measures
Advantages
Limitations
Behavioral Measures
Physiological Measures
Choosing a Measurement Modality

11. Modalities: Self-Report Advantages Limitations Cognitive Affective
Retrospective v. Hypothetical (behavioral intention)
Kinesthetic
Advantages
Easy to collect
Easy to administer
Feelings (cannot observe)
Thought process (Policy capturing)
May be more accurate than observation. Why?
Limitations
Accuracy of recall
Willingness to report
Verbal skills needed

12. Self Report Modalities:
How to
Ask about actual previous behaviors
Use dichotomous or categorical for or behavior or behavioral intentions
Use Likert (5, 7 or 9) for attitudes
Include both positively and negatively scaled items
Use graphic labels or benchmarks
Use multiple items to capture a concept
Avoid leading double barreled questions (usually)

13. Modalities: Behavioral Measures
Uses
Behavior is object of study
Operational definitions
Nonverbal may be clue to feelings/ physio state
Types of measures
Frequency (bar presses)
Rate
Duration
Intensity
Accuracy
Persistence
Examples?

14. Behavioral Measures Advantages Limitations Surreptitious (sneaky)
Avoids evaluation apprehension
Capture automatic behaviors
Often not premeditated
More accurate than behavioral intention
Limitations
“what you see is what you get” What limitation is implied here?
Interpretation is inferred by observer
Highly situation-specific
That’s why R. Hogan likes to use personality trait as predictor
Need trained observers

15. Modalities: Physiological Measures
Purposes
Of interest in and of itself
As op definition of psychological state
E.g. anxiety, arousal, lying
Advantages
Most direct (no intervening human observer
Highly precise
Limitations
Need equipment, trained administrators
Obtrusive
Constrain freedom
Source of unreliability (testing effects)
E.g. polygraph

16. Modalities: Choosing & Locating Measurement Modality
Choosing: Self-report, behavioral, physiological
Rank them for level of validity
Which one should be used?
See table 5-2 relative advantages and limitations
Multiple operationism
Categories of Measures (figure 5-7, p. 147)
Type: manifest v. hypothetical
Hypothetical construct:
Psychometric (used for individual scores)
research measures (used for mean scores; norms)
Developed (use when it fits procedure)
Ad hoc (when you have to develop it)

17. Modalities: Locating & Evlauting Measures
Locating Measures
MMY
(Buros)
Tests in Print
Directory of Unpublished Experimental Measures
(Goldman & Sanders, 1997)
Measures of Personality and Social Psychological Attitudes
(Robinson, Shaver,& Wrightsman, ’91)
Organizational Measures
Warr, Cook and Wall??

18. Modalities: Evaluating Them
Theoretical Background
How much construct standing?
Quality of Development
Participant samples, norms
Reliability & Validity
What are good coefficients for each?
Valid for what?
Freedom from Response Bias
Social desirability
Acquiescence bias

19. Developing a Measurement Strategy Summary
Scales of Measurement
Reliability and Validity
Modalities of Measurement
Locating and Evaluating Measures