1. Questionnaire Construction: Eight Principles
SARS=Satisfaction And Reference Service
Dr. John V. Richardson Jr.
Professor of Information Studies
UCLA Department of Information Studies

2. Presentation Outline Question Design Validity Concerns
Methodological Issues
Mini Case Studies
Question Design and Validity Concerns:
Intent of the Question
Clarity of the Question
Unidimensionality
Scaling
Number of Questions
Method Issues: Timing of Administration
Question Order
Sample Sizes

3. Question Design and Validity Concerns
Eight issues which must be addressed to insure validity of survey results:
Intent of the question
Clarity of the question
Unidimensionality
Scaling
Number of questions to include
Timing of administration
Question order
Sample sizes
A well designed study takes into consideration these nine points

4. 1. Intent of the Question RUSA Behavioral Guidelines (1996)
Approachability
Interest in the query, and
Active listening skills
UniFocus (300 factor analyses of the hospitality industry)
Friendliness
Helpfulness or accuracy
Promptness of service
What does your library care about? What do you want to know about?
Customer satisfaction versus customer loyalty?

5. 2. Clarity of the Question
Data from unclear questions
may be invalid
Use instructions
to enhance question clarity
Editing, rewriting, and pre-testing are invaluable

6. Mini Case Study
What is the literal correct answer to the question posed?
Yes or No is the correct answer rather than a scaled response…
Could it be rewritten? “How are we providing…”?

7. 3. Unidimensionality
Unidimensionality is a statistical concept that describes the extent to which a set of questions all measure the same topic
How many questions do you need?
Do they all measure the same thing?

8. Constellation of Attitudes
Satisfaction
Delight
Intent to Return
Feelings about experiences
Value
Loyalty
“Satisfaction researchers in marketing are in general agreement that the emotion of delight is comprised of joy and surprise. This study reviews the relevant emotions literature in psychology, the neurosciences and philosophy to show that there may be two different kinds of delight - one with surprise and one without surprise. The work of Plutchik (1980) is often cited as the basis for conceptualizing delight as being comprised of joy and surprise. We replicated Plutchik's two studies using more positive complex emotion terms than the original study. It was found that subjects could feel delighted without being surprised and that there were different emotion terms that were considered by subjects to be comprised of joy and surprise. These results were validated in a second study in which consumer emotions and other responses were captured in a live setting during the intermission of an upbeat, fast tempo Irish Dance concert. The results show that consumers could be delighted even when they were not surprised. We show how these findings clarify and explain some unexpected results obtained in past research on customer delight. The implications of these findings for both theory and practice are also discussed."

9. 4. Scaling Three key characteristics:
Does the scale have the right number of points (called response options)?
Are the words used to describe the scale points appropriate?
Is there a midpoint or neutral point on the scale?
Take a look at your examples…

10. A. Response Options A common four point scale:
Very good, good, fair, and poor
Distance between very good and good is not the same as the distance between fair and poor
Numeric values associated with these options:
4, 3, 2, and 1 may lead to invalid results…
Four point is familiar, but…

11. Mini Case Study
What is the distance between each of these response options?
Technically, the question is framed to be answered as Yes or No.

12. B1. Scale Anchors VERY… VERY… Satisfied Dissatisfied
Much Agree Much Disagree
Positive Negative
Valuable Costly
Enjoyable Unpleasant
Friendly Unfriendly
Here are some good examples of scale anchors which are balanced…

13. Mini Case Study What are the scale anchors here?
It is not 1; three point scale where the value of 1 is for N/A
4 and 2 are the anchors—doing well and needs improvement.
Parallel? Not okay is….better than ok is….
What if this question is skipped?

14. B2. Seven Point Scales Scale A: Very good Very Poor N/A
Scale B:
Excellent Very Poor N/A
Scale C:
Outstanding Disappointing N/A
Seven point scales are better due to the possibility of range restriction;
Is there any reason to believe that scores will cluster together? Then use seven points, but look closely at the anchors in this slide.

15. C. Wording of Options
The only difference in the preceding slide are the response anchors…
Is very good a rigorous enough expectation?
Would excellent be better?
What about outstanding?
What are your library’s aspirations?

16. Mini Case Study How many response points are there?
What is the level of expectation?
It’s not a trick question—there are two correct answers, but the better answer may be 8 because 0 would be the missing value if the question is skipped.
Is “very satisfied” a rigorous enough expectation? Would delighted be better?

17. D. Midpoint or Neutral Point
The rate of skipped questions increases when a neutral response is not included
Use an odd number of response points
Also, a neutral response provides a way to treat missing data
If they skip it, then one make the assumption that they are Neutral on this question

18. Mini Case Study
What’s the midpoint?

19. 5. Number of Questions Short enough Long enough
So that users will answer all the questions
Long enough
So that enough information is gathered for decision making purposes
How many questions should we ask our users?
Short surveys mean less than ten minutes

20. A. Longer Surveys
Take more time and effort on the part of the respondent
High perceived “cost of completion” results in partially or completely unanswered questions in surveys
Longer means up to one hour

21. B. Likelihood of Complete Responses
Higher salience or more important the topic to the user, the greater the likelihood that they will complete a longer survey
Multiple questions measuring a single attitude make for longer surveys, although
They also aid in evaluating user attitudes
Remember: what’s interesting to us, may not be interesting to them
Role of premiums; rewards

22. 6. Timing and Ease During or immediately following
Blurring together?
Cards or mail method (IVR=interactive voice response)
Delay seems to cause more positive results
Electronic reference allows for ease of administration (more on PaSS™ later)
Immediacy versus longer-term utility
In an academic library, the end of quarter or end of the year might not be the best time.

23. 7. Question Order Specific questions first More general second
Technology, resources, or staffing
More general second
Value, overall satisfaction, intent to return
Halo Effect
Four question survey: one overall and three specific questions
Asking general question last produces better data
What difference does it make?

24. Mini Case Study
Specific questions first.
More general second.

25. 8. Sample Sizes Depends upon population size
Error rate
Confidence
Consult a table of sample sizes
Please, please work on getting larger sample sizes.
No creditability when read outside of our field.

26. A. Error Rate Defined as the precision of measurement
Accurate to plus or minus some figure
Has to be precise enough to know which direction service quality is going (i.e., up or down)

27. B. Confidence Refers to the overall confidence in the results:
.99 confidence level means that one can be relatively certain that the results are within that range 99% of the time
.95 confidence level is common
.90 confidence level is less common, but…a 90 CL requires fewer respondents, but will result in a less accurate survey

28. C. Population and Sample
Population (N) refers to the people of interest
Sample (n) refers to the people measured to represent the population
Response rate is the proportion of the population who respond to the survey

29. D. Population & Sample Size
N= n=
SOURCE: Robert V. Krejcie and Daryle W. Morgan, “Determining sample size for research activities," Educational and Psychological Measurement 30 (Autumn 1970):
Note that the required sample size does not increase at the same rate as the population.
Several hundred should be your goal in most library settings…
Larger the population—say 10K graduate students versus 20K undergraduates

30. Appropriate Sample Sizes