MANA 5341 Dr. George Benson benson@uta.edu Measurement MANA 5341 Dr. George Benson benson@uta.edu 1

Testing: Basic Concepts The Normal Curve Many people taking a test One person taking the test many times Central tendency (Mean) Variability (Standard Deviation)

The Normal Curve Rounded Percentiles Note: Not to Scale Rounded Percentiles Standard Deviations or Z Scores .1% 2.3% 15.9% 50% 84.1% 97.7% 99.9% -3 -2 -1 0 +1 +2 +3

Variability How did an individual score compared to others? How to compare scores across different tests? Test 1 Test 2 Bob Jim Sue Linda Raw Score 49 47

Variability How did an individual score compared to others? How to compare scores across different tests? Test 1 Test 2 Bob Jim Sue Linda Raw Score 49 47 Mean 48 46

Variability How did an individual score compared to others? How to compare scores across different tests? Test 1 Test 2 Bob Jim Sue Linda Raw Score 49 47 Mean 48 46 Std. Dev 2.5 .80

Z Score or “Standard” Score Score – Mean Z Score = Std. Dev Test 1 Test 2 Bob Jim Sue Linda Raw Score 49 47 Mean 48 46 Std. Dev 2.5 .80 Z score .4 -.4 3.75 1.25

The Normal Curve Note: Not to Scale Jim Bob Linda Sue

Converting Z scores to Percentiles Look up z scores on a “standard normal table” Corresponds to proportion of area under normal curve Move decimal 2 places and you have a percentage Linda has z score of 1.25 Standard normal table = .9265 Percentile score of 92.65% Linda scored better than 92.65% of test takers Z score Percentile 3.0 99.9% 2.0 97.7% 1.0 84.1% 0.0 50.0% -1.0 15.9% -2.0 2.3% -3.0 .1%

Proportion Under the Normal Curve Note: Not to Scale Jim Bob Linda Sue

Correlation How strongly are two variables related? Correlation coefficient (r) Ranges from -1.00 to 1.00 Shared variation = r2 If two variables are correlated at r =.6 then they share .62 or 36% of the total variance. Illustrated using scatter plots Used to test consistency and accuracy of measure

Correlation Scatterplots

Reliability: Basic Concepts Observed score = true score + error Error is anything that impacts test scores that is not the characteristic being measured Reliability measures error Lower the error the better the measure Things that can be observed are easier to measure than things that are inferred

Reliability Consistency of the measure Potential contaminations If the same person takes the test again will he/she earn the same score? Potential contaminations Test takers physical or mental state Environmental factors Test forms Multiple raters

Reliability of Measures Visual acuity High Hearing Dexterity Mathematical ability Verbal ability Intelligence Clerical skills Mechanical aptitudes Sociability Cooperativeness Tolerance Emotional stability Low

Reliability Test Methods Test – retest Alternate or parallel form Internal consistency Inter-rater Methods of calculating correlations between test items, administrations, or scoring.

Summary of Types of Reliability Compare scores within T1 Compare Scores across T1 and T2 Objective Measures (Test items) Internal Consistency or Alternate Form Test-retest Subjective Ratings Interrater – Compare different Raters Intrarater – Compare same Rater different times

Reliability Test Methods Test – retest Pearson product moment correlation (the “Correlation” coefficient) Memory and learning as potential contaminations Good for single item measures Alternate or parallel form Correlation coefficient becomes the “Coefficient of Equivalence” Testing Form “equivalency”

Reliability Test Methods Internal consistency Extent to which measures are similar Single measure use “Split-half” reliability Spearman Brown Correction since you use only half Multiple measures use “Chronbach’s Alpha” Average correlation of measures with one another Inter-rater Used to test reliability of subjective measures Consistency across two raters level of agreement 80% is the target Cohen’s Kappa or Kendal’s Tau

How high should a reliability coefficient be? Test-Retest (immediate) r = .90 Split-half reliability r = .85 Test-Retest (long-term) r = .80 Chronbach’s Alpha r = .75

Standard Error of Measure (SEM) Estimate of the error for an individual test score Uses variability AND reliability to establish a confidence interval around a score 95% Confidence Interval (CI) means if one person took the test 100 times, 95 of the scores will fall within the upper and lower bounds. SEM = SD * √ (1- reliability) There is a 5% chance that scores observed outside the CI are due to chance, therefore the differences are “significant”.

The Normal Curve Rounded Percentiles Note: Not to Scale Rounded Percentiles Standard Deviations or Z Scores .1% 2.3% 15.9% 50% 84.1% 97.7% 99.9% -3 -2 -1 0 +1 +2 +3

Standard Error of Measure (SEM) Assume a mathematical ability test has a reliability of .9 and a standard deviation of 10. SEM = 10 * √ (1- .9) = 3.16 If an applicant scores a 50, the SEM is the degree to which the score would vary if she were retested on another day. Plus or minus 2 SEM gives you a ~95% confidence interval. 50 + 2(3.16) = 56.32 50 – 2(3.16) = 43.68

Standard Error of Measure The difference between two scores should not be considered significant unless the difference is twice the standard error. If an applicant scores 2 points above a passing score and the SEM is 3.16 – then there is a good chance of making a bad selection choice. If two applicants score within 2 points of one another and the SEM is 3.16 then it is possible that the difference is due to chance.

Standard Error of Measure The higher the reliability, the lower the SEM Std. Dev. r SEM 10 .96 2 .84 4 .75 5 .51 7

Confidence Intervals Jim -- 40 Mary -- 50 Jen -- 60 SEM -2 SEM +2 SEM 36 44 46 54 56 64 4 32 48 42 58 52 68 Do the applicants differ when SEM = 2? Do the applicants differ when SEM = 4?

Validation: Basic Concepts Validity is the degree to which research supports inferences made from selection test scores Accuracy of the measure Are you measuring what you intend to measure? OR Does the test measure a characteristic related to job performance?

Methods to Test Validity Criterion – test predicts job performance Criterion (criteria) vs. predictors Content – test representative of the job Construct – test of abstract trait is accurate and predictive of job performance

Types of Validity Criterion-Related Content-Related Job Performance Job Duties Selection Tests KSA’s

Conducting a Validation Study Conduct a job analysis Identify critical KSA’s Choose (or develop) predictors of the KSA’s Choose job performance criteria Administer to incumbents or applicants Correlate two administrations

Criterion-Related Validity Deficiency Validity Contamination Job Performance Test Performance

Tests of Criterion-Related Validity Predictive validity “Future Employee or Follow-up Method” Test Applicants Performance of Hires Time 1 6-12 mos. Time 2 Concurrent validity “Present Employee Method” Test Existing Employee AND Measure Performance Time 1

Assumptions Time between measures is appropriate. Job (and measures of performance) is stable over time. Sample of applicants or incumbents is representative. Large enough samples (hundreds of observations).

Content-Related Validity Deficiency Validity Contamination Job Content Test Content

Reliability vs. Validity

Validation in Practice Small business applications Validation studies require large samples and expertise Use off-the-shelf tests for common jobs Increase number of observations by broadening categories of jobs or look at KSA’s across jobs Utility Analysis Estimates the economic value of selection test decisions Need to put a dollar value on productivity Seldom used in practice

Reliability vs. Validity Validity Coefficients Reject below .11 Very useful above .21 Rarely exceed .40 Reliability Coefficients Reject below .70 Very useful above .90 Rarely approaches 1.00 Why the difference?

Principles of Assessment Uniform Guidelines For Employee Selection Procedures http://www.dol.gov/dol/allcfr/ESA/Title_41/Part_60-3/toc.htm Don’t rely on a single method. Use only fair and unbiased instruments. Use only reliable and valid instruments. Use only tools designed for a specific group. Use instruments with understandable instructions. Ensure test administration staff are properly trained. Ensure test conditions are suitable for all test takers. Provide reasonable accommodation. Maintain confidentiality of results. Ensure proper interpretation of results.

Selection Tests and Litigation Unstructured interviews Cognitive ability tests Physical ability tests Structured interviews Work sample tests Assessment Centers More likely to be challenged in court

Where to get validated measures.... Buros’ Mental Measurements Yearbook Pro-Ed Consumer’s Guide to Tests in Print http://www.shrm.org/assessment/index.asp