1. Questionnaire Development
Measuring Validity & Reliability
James A. Pershing, Ph.D.
Indiana University

2. Definition of Validity
Instrument measures what it is intended to measure:
Appropriate
Meaningful
Useful
Enables a performance analyst or evaluator to draw correct conclusions

3. Types of Validity Face Content Criterion Construct Concurrent
Predictive
Construct

4. Face Validity It looks OK
Looks to measure what it is supposed to measure
Look at items for appropriateness
Client
Sample respondents
Least scientific validity measure
Looks Good To Me

5. Content-Related Validity
Organized review of format and content of instrument
Comprehensiveness
Adequate number of questions per objective
No voids in content
By subject matter experts
Balance
Definition
Sample
Content
Format

6. Criterion-Related Validity
How one measure stacks-up against another
Concurrent = at same time
Predictive = now and future
Independent sources that measure same phenomena
Seeking a high correlation
Subject Instrument A Instrument B
Task Observation
Inventory Checklist
John yes no
Mary no no
Lee yes no
Pat no no
Jim yes yes
Scott yes yes
Jill no yes
Usually expressed as a correlation coefficient (0.70 or higher is generally accepted as representing good validity)

7. Construct-Related Validity
A theory exists explaining how the concept being measured relates to other concepts
Look for positive or negative correlation
Often over time and in multiple settings
Usually expressed as a correlation coefficient (0.70 or higher is generally accepted as representing good validity)
Prediction 1 - Confirmed
THEORY
Prediction 2 - Confirmed
Prediction 3 - Confirmed
Prediction n - Confirmed

8. Definition of Reliability
The degree to which measures obtained with an instrument are consistent measures of what the instrument is intended to measure
Sources of error
Random error = unpredictable error which is primarily affected by sampling techniques
Select more representative samples
Select larger samples
Measurement error = performance of instrument

9. Types of Reliability Test-Retest Equivalent Forms Internal Consistency
Split-Half Approach
Kuder-Richardson Approach
Cronbach Alpha Approach

10. Test-Retest Reliability
Administer the same instrument twice to the same exact group after a time interval has elapsed.
Calculate a reliability coefficient (r) to indicate the relationship between the two sets of scores.
r of+.51 to +.75 moderate to good
r over +.75 = very good to excellent
T I M E

11. Equivalent Forms Reliability
Also called alternate or parallel forms
Instruments administered to same group at same time
Vary:
Calculate a reliability coefficient (r) to indicate the relationship between the two sets of scores.
r of+.51 to +.75 moderate to good
r over +.75 = very good to excellent
Stem:
-- Order
-- Wording
Response Set:
-- Order
-- Wording

12. Internal Consistency Reliability
Split-Half
Break instrument or sub-parts in ½ -- like two instruments
Correlate scores on the two halves
Best to consult statistics book and consultant and use computer software to do the calculations for these tests
Kuder-Richardson (KR)
Treats instrument as whole
Compares variance of total scores and sum of item variances
Cronbach Alpha
Like KR approach
Data scaled or ranked

13. Reliability and Validity
So unreliable as to be invalid
Fair reliability and fair validity
Fair reliability but invalid
Good reliability but invalid
Good reliability and good validity
The bulls-eye in each target represents the information that is desired. Each dot represents a separate score obtained with the instrument. A dot in the bulls-eye indicates that the information obtained (the score) is the information the analyst or evaluator desires.

14. Comments and Questions