1. MGTO 324 Recruitment and Selections
Scale and Test Construction
Kin Fai Ellick Wong Ph.D.
Department of Management of Organizations
Hong Kong University of Science & Technology

2. Prologue
In the last lesson, I have discussed the scientific elements in testing
Today, we focus on how a test can be constructed
In particular, you are expected to understand the following
The concepts of “item”, “scale”, and “test”
Different item formats
How to write a set of good items
Multiple-item scaling

3. Outline

4. Outline

5. Part I: Basic Concepts What is a (psychological) test?
Measurement device or technique
To quantify behavior or aid in the understanding and prediction of behavior
A set of items designed to measure characteristics of human beings that pertain to behavior

6. Part I: Basic Concepts What is an item?
A specific stimulus to which a person responds overtly
What is the English word of “邂逅”
You can manage well in interpersonal relationship
Overt behaviors (Scientific Standard)
Observable
Measurable
Can be replicated 1 2 3 4 5
Strongly disagree
Strongly agree

7. Part I: Basic Concepts What is a scale?
The quantified scores obtained from a test
The raw scores are related to some defined theoretical or empirical distribution
The matching between the raw score and the theoretical meaning of that score
E.g., 0oC = freezing point; 100oC = boiling point
The same theoretical meaning could be represented by different scales
Temperature: Degree Celsius vs. Degree Fahrenheit
Length: Meter vs. Feet; Kilometer vs. Mile
Weight: lb vs. kg
Wealth: HK$ vs. US$

8. Part I: Basic Concepts What is a scale? Examples Thermometer HKCEE
Raw scores = 2.4 cm;
Degree Celsius = 100
Theoretically (empirically) = boiling point
HKCEE
Raw scores = 87
Grade = A
Theoretically (empirically) = Excellent students
IQ test
Raw scores = 2400
IQ = 130
Theoretically (empirically) = Gifted individual

9. Part I: Basic Concepts Essential steps in scale and test construction
Have a clear definition of what (i.e., the psychological construct) the test is supposed to measure
E.g., Locus of control; self-efficacy
Generating a set of items that at least seems to capture the construct
Determining the scale format
Pilot tests to assess the face validity, reliability, construct validity, and criterion validity
Deleting or revising less useful items
Assessing reliability and validity again
Revising again
Assessing again
Shorten the scale
Assessing again….

10. Outline

11. Part II: Item Formats

12. Part II: Item Formats Dichotomous formats
Offers two alternatives for each item
True/False; or Select a more appropriate statement
“You often spend more than three hours on typing every day”
Agree vs. Disagree
“Generally speaking, the salary for me accurately represents my contribution to the organization”
“Which job, technical or administrative, do you prefer?”
Technical vs. Administrative
Scores
Simply count the number of items a person endorse
Commonly used in both educational and personality tests

13. Part II: Item Formats Dichotomous formats
Some famous scales used dichotomous formats
Locus of control (Rotter, 1966)
“the degree to which people believe they are masters of their own fates” from the OB textbook (Robin, 2003)
High in externality
Less satisfied with jobs; Higher absenteeism rates; More alienated from the work; Less involved in their jobs
Choose one
A. Many of the unhappy things in people’s lives are partly due to bad luck
B. People’s misfortunes result from the mistakes they make
Heredity plays the major role in determining one’s personality
It is one’s experiences in life which determine what one is like

14. Part II: Item Formats Dichotomous formats Advantages Disadvantages
Simple, easy to administer and score
Absolute judgment; people should declare one of the two alternatives
Disadvantages
Effect of memorizing materials
Requires numerous items to produce reliable results (chance = 50%)

15. Part II: Item Formats

16. Part II: Item Formats Polytomous formats
Offers more than two alternatives (e.g., usually 4 to 5)
One correct choice and other distractors
Multiple-choice
Scores
Number of items correctly answered
How many distractors?
More distractors may not be better than less distractors (Sidick, Barrett, & Doverspike, 1994)
Less influenced memory effects (relative to dichotomous format)
Mainly used in educational tests

17. Part II: Item Formats

18. Part II: Item Formats Likert-type formats
Usually, people are required to express their degree of dimension on a statement
I can manage well in interpersonal relationship
Monotone items
A higher score suggests higher agreement 1 2 3 4 5
Strongly disagree
disagree
neutral
agree
Strongly agree

19. Part II: Item Formats Likert-type formats Number of alternatives
Odd (5, 7, or 9)
Even (6, or 10): avoid midpoint
Popular in psychological tests
It can be subject to various psychometric analyses, such as factor analysis
For example, the General Self-efficacy Scale
Self-efficacy
“The individual’s belief that he or she is capable of performing a task” from OB textbook (Robin, 2003)
Higher self-efficacy
Less likely to give up under difficult situations
Usually perform better than

20. Part II: Item Formats

21. Part II: Item Formats Cumulative (Guttman) formats
Items on the same dimension are set up in ascending order
Subject with a particular attitude will agree with all items on one side of that position and disagree with other items
Example Addition, long-division, and calculus
Monotone items

22. Part II: Item Formats
Cumulative (Guttman) formats

23. Part II: Item Formats
Cumulative (Guttman) formats

24. Part II: Item Formats

25. Part II: Item Formats Cumulative (Guttman) formats Advantages
A single score carries complete information about the response patterns
Calculus OK = division OK = addition OK
Marriage OK = being neighbor OK
When there is no random error
Provides a test of the unidimensionality of what are to be tested
Cumulative response pattern will not be obtained when the items do not measure only one dimension
Disadvantages
Problems resulted from random error
Difficult to find domains that are unidimensional
Less popular than Likert-type format

26. Outline

27. Part III: Writing good items
Define clearly
Clearly define what you want to measure
Check the face and content validity
Is the items out of syllabus
How many psychological factor I want to measure? List them all
I want to develop a test that helps me hire employees who have strong self-learning tendency
What do you want to measure from the test?
What is “self-learning tendency”?
Give a clear definition before moving to the next step

28. Part III: Writing good items
Clearly think about the item formats
Think carefully what type of tests as well as what statistical analyses you want to use
Some statistical analyses may not be appropriate for certain item formats
Rank order is not appropriate for t test
Polytomous responses may not be able to be analyzed by Factor Analysis
Likert-type or dichotomous format seems to be a good choice as the default

29. Part III: Writing good items
Sources of items
Discourse and text
From brainstorming or informal conversation
Asking others the meaning of “self-learning”
Use qualitative method is a more systematic method
Open-ended interview or question
Focus group
Content analysis
Newspaper
Classic literature
The goal is to generate a pool of items that seems to measure what we want to measure
E.g., “When I have problems, I’ll seek help from books prior to people.”

30. Part III: Writing good items
Nature of items
The meanings of items should be clear, straightforward, and can be easily understood
Avoid exceptionally long items
Avoid reading difficulty
Don’t use jargon….
Social desirability
“I like self-learning”
Offensiveness
Pay attention to the problems of sexism and racism
Avoid double-barreled items
“seeking help from people is a better method than seeking help from books because people are more accessible.
Reverse items
“Most knowledge could not be learned without the help from others”
“I can learn almost all knowledge by myself”

31. Part III: Writing good items
Generate an item pool
You first need to generate a set of item pool
The numbers of items in this pool are usually much more than the final version of the scale
Do a preliminary test
Select useful items
Discard or revise other items (you will learn these skills later in this course)
You may need several rounds of revision before the scale becomes reliable and valid

32. Part III: Writing good items
What should be included in a test apart from the basic items?
Clear instruction for subjects
Give examples whenever possible
Questions for demographic information
Age, gender, education level, etc.
Declare that how the collected information will be used
Is it confidential? The purpose of collecting the data; who will assess the data, etc.

33. Outline

34. Part IV: Multiple-item scaling
True score theory
Due to the matter of precision, measures may vary from time to time
The true scores can hardly be obtained by only one measure
Sometimes it may overestimate or underestimate the true score, the errors are assumed to be randomly distributed
The true scores can be obtained by averaging multiple responses

35. Part IV: Multiple-item scaling
The concepts of multiple-item scaling
I want to measure one’s attitude toward whether secondary students can have romantic relationships
Single item
I encourage secondary students to develop romantic relationship
Two items
I encourage secondary students to develop romantic relationships
We should respect the secondary students’ freedom of engaging in romantic relationships
The issues of reliability and validity

36. Part IV: Multiple-item scaling
Advantages multiple-item scaling
It allows testing the nature of a construct
A set of items may capture more than one dimension
Intelligence:
Memory span, verbal ability, and visual-spatial ability
Items measuring same dimension or same construct should be highly inter-correlated
Items measuring different dimensions are supposed to have relatively low inter-item correlation
It allows us to check scale validity through Factor analysis
Discuss again later
Increase reliability and validity
Remember the concept of true score theory
Random errors may exert impacts on a particular single response
Effects of random errors can be “neutralized” by measuring multiple responses