1. DEVELOPING A QUESTIONNAIRE FOR USE IN OUTCOMES ASSESSMENT
Kathleen A. Martin, D.P.E.
Office of Institutional Research & Assessment
Bucknell University

2. POINT OF CLARIFICATION
Terms “survey” and “questionnaire” are often used interchangeably
“Survey” refers to a descriptive research technique or methodology (also includes interviews, and focus groups)
“Questionnaire” refers to a paper and pencil instrument through which information is obtained
For the purposes of outcomes assessment, questionnaires are frequently developed to survey a population

3. QUESTIONNAIRES & ASSESSMENT
Useful for surveying groups of any size
Typically designed to determine opinions, attitudes or present practices
Limitation: rely on self report information v. observed behavior
Challenges to validity become a concern

4. VALIDITY CHALLENGES
Validity: the extent to which the test measures what it purports to measure
Internal Validity: Relates to instrument (questionnaire) quality. Are the appropriate questions asked clearly and logically?
External validity: can the results be generalized?
Measurement error: difference between the characteristics of your sample and the characteristics of the population
Non-response bias: answers of respondents do not mirror the characteristics of those who refused participation.

5. MAXIMIZING VALIDITY Create a questionnaire that is clear and logical
Support content relevance and content representativeness
Content Relevance: are the questions relevant to the purpose of the questionnaire? Unimportant content should be eliminated.
Content Representativeness: Are the questions an adequate representation of the universe of possible questions?
Solicit the input of experts in scale development and the content of the questionnaire for an objective evaluation
Select a large, representative random sample from the population of interest or census the finite population
Increase response rate using formal follow up procedures

6. INITIAL CONSIDERATIONS
Is the development of a new questionnaire . . .
Necessary?
Feasible?
Select mode of administration
Web Based?
Paper and pencil?
Combination?

7. QUESTIONNAIRE PLANNING
1. Determine the Objective
What do you want to ask and of whom?
Consider how the data will be analyzed. What information is needed?
2. Determine the Mode of Administration
Paper and pencil
Cost effective, but low response rate
Web / based
Cost effective, higher response rate, some suspicion of the medium
Combination
Can be administratively cumbersome

8. QUESTIONNAIRE PLANNING
Determine the Sampling Methodology
To whom will you send the questionnaire? Who can supply the information?
Will everyone be surveyed?
ALL current students? ALL alumni?
If yes, this is a census of the finite population
The selection of a representative sample of the population may be more effective based on time and cost of administration
Decision depends on the size of the finite population

9. CONSTRUCTING the QUESTIONNAIRE
Consider what objective each question measures
Three to five questions per objective is recommended
How many questions?
Find a balance: long enough to obtain the necessary information but short enough so respondents won’t lose interest.
Questionnaires that are too long won’t be completed
Goal: minutes MAX

10. DESIGNING THE QUESTIONNAIRE
Question Formats: Open Ended Questions
Allow respondents to express feelings and expand on ideas.
Question should be phrased to avoid a one word response.
“What aspects of the Sociology program best prepared you for your current position?”
Open ended questions at the beginning of the questionnaire may be discouraging to respondent – include one or two at the end
Analysis of responses can be challenging

11. QUESTIONNAIRE DEVELOPMENT
Question Formats: Closed Questions
Ranking
Forces the respondent to rank order responses based on a value judgment.
Rank the following items with regard to how you spend time at Bucknell from 1 (spend the most time) to 5 (spend the least time).
_____ Studying
_____ Socializing
_____ Volunteering
_____ Working at a part time job
_____ Exercising

12. QUESTIONNAIRE DEVELOPMENT
Question Formats: Closed Questions
Checklist
A number of possible answers are provided and the respondent is asked to choose one or choose all that apply.
How did you finance your Bucknell University education? Choose all that apply.
_____ Parental contribution
_____ Student Loan
_____ Work Study
_____ Off campus employment
_____ Other (please specify): _____________

13. QUESTIONNAIRE DEVELOPMENT
Question Formats: Closed Questions
Scaled items
Very common
Indicate strength of agreement or disagreement with a statement.
Numbers are assigned to each response on the continuum
The English curriculum at Bucknell University adequately prepared me for my current position
Strongly
Disagree
Neutral / No Opinion
Agree

14. QUESTIONNAIRE DEVELOPMENT
Question Formats: Closed Questions
Suggestions for writing scaled items
Use positively worded or negatively worded items consistently
Use an odd number of responses along the continuum to allow for a neutral response
Anchors may be different based on focus of the item (Strongly Agree – Strongly Disagree; Frequently – Never; Very Good – Very Poor)
Keep items short

15. QUESTIONNAIRE DEVELOPMENT
Question Formats: Categorical
Typical demographic questions:
Yes/No/?, Male/Female, F/S/J/S/G
Answers fall into certain categories – respondents pick the one that applies to them
Analyzed using frequency, proportion, percentages
2 considerations:
Questions on a continuum should stay on a continuum if they will be directly analyzed
Example: Age – allow respondent to write in their age rather than check off a box for their age range
Categories must be collectively exhaustive & mutually exclusive (next slide)

16. QUESTIONNAIRE DEVELOPMENT
Problematic:
Number of hours spent studying per week:
18 – 21
21 – 24
24 – 27
Categories are not mutually exclusive
Corrected:
Number of hours spent studying per week:
18 – 21
22 – 25
26 – 29

17. IMPORTANT CONSIDERATIONS
APPEARANCE AND DESIGN
Have clear directions that are age appropriate & easy to follow
Format, size, reproduction should facilitate completion
No less than 12 pt
Times, Arial, Courier
PILOT STUDY
Give the questionnaire to a few people to try it out
Have them evaluate readability and clarity of questions
Estimate time required to complete the questionnaire
Suggestions for revisions

18. SENDING IT OUT INITIAL CONTACT / COVER LETTER Short & to the point
Identify person by name
Convince respondent that participation is important
Ensure confidentiality
Deadline for return of questionnaire
SASE for return (mailed questionnaires only)

19. Institutional approval
Institutional Review Board
Research involving human subjects must be approved by the Bucknell University IRB
For more information, visit the IRB website at
Campus Survey Coordination
The timing of all university-wide surveys must be coordinated through the Campus Survey Coordination Group.
For more information, visit the CSCG website at

20. FINAL THOUGHTS FOLLOW UP PROCEDURES
Send to all participants to preserve confidentiality (“If you haven’t already returned the questionnaire…..”)
May include a second copy if available
Usually done in 2 week intervals
RESPONSE RATE FOR MAILED QUESTIONNAIRES
Typically low - 30% is considered good
Higher response rate with web-based questionnaires
Good follow-up increases response rate

21. SUMMARY OF STEPS Determine the Objective (consider analyses)
Determine the Mode of Administration
Determine the Sampling Methodology
Construct the Questionnaire
Institutional Approval
Conduct the Pilot Study
Write the Initial Communication
Send the Questionnaire
Follow up
Analyze the Results

22. Contact Kathy Martin @ 71960 or kmartin@bucknell.edu
FOR MORE INFORMATION
Contact Kathy or