1. MANA 5341 Dr. George Benson benson@uta.edu
Measurement
MANA 5341
Dr. George Benson 1

2. Testing: Basic Concepts
The Normal Curve
Many people taking a test
One person taking the test many times
Central tendency (Mean)
Variability (Standard Deviation)

3. The Normal Curve Rounded Percentiles
Note: Not to Scale
Rounded Percentiles
Standard Deviations or
Z Scores
.1% % 15.9% % % % %

4. Variability How did an individual score compared to others?
How to compare scores across different tests?
Test 1
Test 2
Bob
Jim
Sue
Linda
Raw Score 49 47

5. Variability How did an individual score compared to others?
How to compare scores across different tests?
Test 1
Test 2
Bob
Jim
Sue
Linda
Raw Score 49 47
Mean 48 46

6. Variability How did an individual score compared to others?
How to compare scores across different tests?
Test 1
Test 2
Bob
Jim
Sue
Linda
Raw Score 49 47
Mean 48 46
Std. Dev
2.5
.80

7. Z Score or “Standard” Score
Score – Mean
Z Score =
Std. Dev
Test 1
Test 2
Bob
Jim
Sue
Linda
Raw Score 49 47
Mean 48 46
Std. Dev
2.5
.80
Z score .4
-.4
3.75
1.25

8. The Normal Curve
Note: Not to Scale
Jim Bob Linda Sue

9. Converting Z scores to Percentiles
Look up z scores on a “standard normal table”
Corresponds to proportion of area under normal curve
Move decimal 2 places and you have a percentage
Linda has z score of 1.25
Standard normal table = .9265
Percentile score of 92.65%
Linda scored better than 92.65% of test takers
Z score
Percentile
3.0
99.9%
2.0
97.7%
1.0
84.1%
0.0
50.0%
-1.0
15.9%
-2.0
2.3%
-3.0
.1%

10. Proportion Under the Normal Curve
Note: Not to Scale
Jim Bob Linda Sue

11. Correlation How strongly are two variables related?
Correlation coefficient (r)
Ranges from to 1.00
Shared variation = r2
If two variables are correlated at r =.6 then they share .62 or 36% of the total variance.
Illustrated using scatter plots
Used to test consistency and accuracy of measure

12. Correlation Scatterplots

13. Reliability: Basic Concepts
Observed score = true score + error
Error is anything that impacts test scores that is not the characteristic being measured
Reliability measures error
Lower the error the better the measure
Things that can be observed are easier to measure than things that are inferred

14. Reliability Consistency of the measure Potential contaminations
If the same person takes the test again will he/she earn the same score?
Potential contaminations
Test takers physical or mental state
Environmental factors
Test forms
Multiple raters

15. Reliability of Measures
Visual acuity High
Hearing
Dexterity
Mathematical ability
Verbal ability
Intelligence
Clerical skills
Mechanical aptitudes
Sociability
Cooperativeness
Tolerance
Emotional stability Low

16. Reliability Test Methods
Test – retest
Alternate or parallel form
Internal consistency
Inter-rater
Methods of calculating correlations between test items, administrations, or scoring.

17. Summary of Types of Reliability
Compare scores within T1
Compare Scores across T1 and T2
Objective Measures
(Test items)
Internal Consistency or
Alternate Form
Test-retest
Subjective Ratings
Interrater –
Compare different Raters
Intrarater –
Compare same Rater different times

18. Reliability Test Methods
Test – retest
Pearson product moment correlation
(the “Correlation” coefficient)
Memory and learning as potential contaminations
Good for single item measures
Alternate or parallel form
Correlation coefficient becomes the “Coefficient of Equivalence”
Testing Form “equivalency”

19. Reliability Test Methods
Internal consistency
Extent to which measures are similar
Single measure use “Split-half” reliability
Spearman Brown Correction since you use only half
Multiple measures use “Chronbach’s Alpha”
Average correlation of measures with one another
Inter-rater
Used to test reliability of subjective measures
Consistency across two raters level of agreement
80% is the target
Cohen’s Kappa or Kendal’s Tau

20. How high should a reliability coefficient be?
Test-Retest (immediate) r = .90
Split-half reliability r = .85
Test-Retest (long-term) r = .80
Chronbach’s Alpha r = .75

21. Standard Error of Measure (SEM)
Estimate of the error for an individual test score
Uses variability AND reliability to establish a confidence interval around a score
95% Confidence Interval (CI) means if one person took the test 100 times, 95 of the scores will fall within the upper and lower bounds.
SEM = SD * √ (1- reliability)
There is a 5% chance that scores observed outside the CI are due to chance, therefore the differences are “significant”.

22. The Normal Curve Rounded Percentiles
Note: Not to Scale
Rounded Percentiles
Standard Deviations or
Z Scores
.1% % 15.9% % % % %

23. Standard Error of Measure (SEM)
Assume a mathematical ability test has a reliability of .9 and a standard deviation of 10.
SEM = 10 * √ (1- .9) = 3.16
If an applicant scores a 50, the SEM is the degree to which the score would vary if she were retested on another day.
Plus or minus 2 SEM gives you a ~95% confidence interval.
50 + 2(3.16) = 56.32
50 – 2(3.16) = 43.68

24. Standard Error of Measure
The difference between two scores should not be considered significant unless the difference is twice the standard error.
If an applicant scores 2 points above a passing score and the SEM is 3.16 – then there is a good chance of making a bad selection choice.
If two applicants score within 2 points of one another and the SEM is 3.16 then it is possible that the difference is due to chance.

25. Standard Error of Measure
The higher the reliability, the lower the SEM
Std. Dev. r
SEM 10
.96 2
.84 4
.75 5
.51 7

26. Confidence Intervals Jim -- 40 Mary -- 50 Jen -- 60 SEM -2 SEM +2 SEM36 44 46 54 56 64 4 32 48 42 58 52 68
Do the applicants differ when SEM = 2?
Do the applicants differ when SEM = 4?

27. Validation: Basic Concepts
Validity is the degree to which research supports inferences made from selection test scores
Accuracy of the measure
Are you measuring what you intend to measure? OR
Does the test measure a characteristic related to job performance?

28. Methods to Test Validity
Criterion – test predicts job performance
Criterion (criteria) vs. predictors
Content – test representative of the job
Construct – test of abstract trait is accurate and predictive of job performance

29. Types of Validity Criterion-Related Content-Related Job Performance
Job Duties
Selection Tests
KSA’s

30. Conducting a Validation Study
Conduct a job analysis
Identify critical KSA’s
Choose (or develop) predictors of the KSA’s
Choose job performance criteria
Administer to incumbents or applicants
Correlate two administrations

31. Criterion-Related Validity
Deficiency
Validity
Contamination
Job
Performance
Test
Performance

32. Tests of Criterion-Related Validity
Predictive validity
“Future Employee or Follow-up Method”
Test Applicants Performance of Hires
Time mos. Time 2
Concurrent validity
“Present Employee Method”
Test Existing Employee AND Measure Performance
Time 1

33. Assumptions Time between measures is appropriate.
Job (and measures of performance) is stable over time.
Sample of applicants or incumbents is representative.
Large enough samples (hundreds of observations).

34. Content-Related Validity
Deficiency
Validity
Contamination
Job
Content
Test
Content

35. Reliability vs. Validity

36. Validation in Practice
Small business applications
Validation studies require large samples and expertise
Use off-the-shelf tests for common jobs
Increase number of observations by broadening categories of jobs or look at KSA’s across jobs
Utility Analysis
Estimates the economic value of selection test decisions
Need to put a dollar value on productivity
Seldom used in practice

37. Reliability vs. Validity
Validity Coefficients
Reject below .11
Very useful above .21
Rarely exceed .40
Reliability Coefficients
Reject below .70
Very useful above .90
Rarely approaches 1.00
Why the difference?

38. Principles of Assessment
Uniform Guidelines For Employee Selection Procedures
Don’t rely on a single method.
Use only fair and unbiased instruments.
Use only reliable and valid instruments.
Use only tools designed for a specific group.
Use instruments with understandable instructions.
Ensure test administration staff are properly trained.
Ensure test conditions are suitable for all test takers.
Provide reasonable accommodation.
Maintain confidentiality of results.
Ensure proper interpretation of results.

39. Selection Tests and Litigation
Unstructured interviews
Cognitive ability tests
Physical ability tests
Structured interviews
Work sample tests
Assessment Centers
More likely to be challenged in court

40. Where to get validated measures....
Buros’ Mental Measurements Yearbook
Pro-Ed Consumer’s Guide to Tests in Print