1. The end of construct validity
Denny Borsboom
University of Amsterdam

2. Two kinds of validity
The working researcher’s idea: Validity concerns the question of whether a test measures what it should measure
The construct validity idea: Validity is an evaluative, integrated judgement of the degree to which test score interpretations are justified in the light of empirical evidence and theoretical rationales (and, possibly, social consequences that follow from test use)

3. What I will argue
The working researchers’ conception is theoretically and practically superior
The construct validity position has some sophication but that is mainly windowdressing; in general, it precisely misses the point of what validity is

4. The pillars of construct validity
Construct validity is
an evaluative judgement
about ‘test score interpretations’
in terms of ‘constructs’
that is a function of evidence
and a matter of degree
I will argue that this view
does not align with the working researcher’s view at all
has quite unreasonable consequences that one should not be comfortable with

6. Why construct validity theory is dysfunctional

7. The social consequences of construct validity theory

8. The social consequences of construct validity theory

9. A black hole that traps all psychometric problems

10. Why construct validity has nothing to do with tests (and why this is wrong)

11. Every interpretation can have construct validity

12. There as as many ‘construct validities’ as there are judges

13. Measurement instruments can ‘become valid’

14. Some measurement instruments ‘were valid’...

15. ...but then ‘ceased to be’ valid...

16. Reference is unimportant
‘Aether’
‘DNA’
‘Phlogiston’
‘Black hole’

17. Validity depends on the presence of ‘interpreters’

18. How construct validity is sold
Construct validity is an evaluative, integrated judgement of the degree to which test score interpretations are justified in the light of empirical evidence and theoretical rationales (and, possibly, social consequences that follow from test use)

19. What construct validity really is
Somebody’s evaluative, integrated and fluctuating judgement of the degree to which test score interpretations, that may have nothing to do with measurement, are justified in the light of time-dependent empirical evidence and that person’s theoretical rationales (and, possibly, that person’s guesses about social consequences that follow from test use as well as his or her valuation of these outcomes)

20. Why all this sophistication misses the point

21. Construct validity is an evaluative, integrated judgement of the degree to which test score interpretations are justified in the light of empirical evidence and theoretical rationales (and, possibly, social consequences that follow from test use)
However, validity is...
a property, not a judgment
a property of instruments, not of inferences
a function of truth, not of evidence
the object of validation research, not its result

22. VALIDITY

23. A simple alternative:
A test is valid for measuring an attribute if and only if variation in the attribute causally produces variation in the measurement outcomes

26. Attribute
structure

27. Attribute
structure

28. Score
structure
Attribute
structure

29. Score
structure
Response
process
Attribute
structure

30. IQ-scores 82
134 70
115 99
Response
process g

31. IQ-scores X 82
134 70
115 99
Response
process
f(X| ) g 

32. Substantive theory Formal model IQ-score X patterns Response f(X| )
process
f(X| ) g 

33. ? ? ? Substantive theory Formal model IQ-score X patterns Response
process
f(X| ) ? g  ?

34. Where to look for validity
Traditionally, evidence for validity is sought in external relations: relations between test scores and other test scores
In criterion validity the evidence comes from correlations with a criterion (or with the criterion)
In construct validity, the evidence comes from correlations with lots of other variables (MTMMs)

35. .09
.15
.56
Attractiveness
Extraversion
Working memory
.55
Masculinity
.40
Race
Visual memory
.35
Annual income
IQ-scores
Job performance
.30
.37
.41
Annual income
Sex
Numerical ability
.50
SES
.20
.78
Physique
Length
Genetic differences
But even if we knew all correlations between all conceivable tests, the validity problem would remain

36. Where to look for validity
Validity is not a matter of external relations between the test scores and other test scores
It is a matter of which processes take attribute differences into response differences
For many tests we have no idea of what happens between item administration and item response
This is the reason that the validity problem has proven hard to crack

37. Where to look for validity
Ingredients for validity:
A theory on the structure of the attribute
A theory on the processes that take levels of the attribute into observed score patterns
A formal model to test the theory against data
The question of validity then becomes: is this theory true?

38. Example: The balance scale test
Weight item
Distance item
What happens when the blocks are removed ?
Conflict Weight item

39. Example: The balance scale test
Theory on the structure of the attribute:
Cognitive development involves an ordered series of discrete transitions between stages
Theory on the processes that take levels of the attribute into observed score patterns:
Children in different developmental stages use different cognitive rules to solve balance scale items, which results in different response patterns
Statistical model to test the theory against data
Developmental stages are conceptualized as latent classes with theoretically driven response vectors

40. Balance scale
Test scores X
001100
111100
001100
110011
110011
Response
process
Rule 1
Rule 2
Rule 3
P(X=x| )
Developmental
stages
Latent classes

41. The question of validity: Is this theory of response behavior correct?

42. How does this relate to other issues?
The validity concept is usually applied to many questions simultaneously:
Does the test measure the intended attribute?
How well do the test scores predict other attributes?
Is the use of the test legally defensible?
Will using the test improve the human condition?
which are put under one umbrella; I only deal with (1)
(2-...) are better left to psychometrics, law, politics, etc.

44. Does this mean that other issues are unimportant?
No. Interpretations, uses, and consequences matter a great deal
But they are not thereby issues of validity
Moreover, they usually belong in the public sphere, not in the domain of validity theory

45. Bottom line
To find out what you measure, you have to find out how your instrument works - there is no other way
If you know how your instrument is supposed to work, and you know how it works, you have a definite answer to the validity problem
However, if don’t know how your instrument is supposed to work, and you don’t know how it works, you are in trouble

47. Validity is...
...measuring the right thing