1. SURVEY RESEARCH Created by: Nancy Whelchel
Assistant Director for Survey Research
University Planning and Analysis
NCSU
Adapted for LITRE use by;
Geetanjali Soni
LITRE Assessment Coordinator

2. SURVEY RESEARCH What is a Survey?
A scientific social research method that involves:
Selecting a random sample of people to answer some questions
Designing & administering a standardized questionnaire to get information about the research questions
Analyzing the results to provide descriptions about the people in the sample and find relationships between different responses
Generalizing the results to the population from which the sample was drawn

3. Basic survey formats Telephone interview Face-to-face interview
Computer assisted interview
Self-administered “paper & pencil” survey (scannable or not)
Self-administered Web survey
Self-administered survey
Format determines much of research design (e.g., sampling, questionnaire design, administration, data analysis, reporting, etc.)
Each has implications for cost, timing, etc.
Today’s talk applicable to most methods…

4. Why do a survey?
To collect information not available from other sources
To have comparable information
To be able to generalize results to the population in which you’re interested
Info not available = Attitudes and opinions, Values, Experiences and behaviors, Expectations, Needs,
Demographics, Descriptive information
Comparable info = standardized instrument allows for comparisons between groups, between surveys, benchmarking, etc
Generalize = w/ random sample it’s not just people speaking up

5. Before you plan to do a your own survey…
Think about alternative sources
Information from non-survey sources
Information from existing survey data
Non-Survey sources = At col/univ IR office has access to lots of data - - maybe don’t need ‘opinion’ info
Existing surveys = Other surveys done on similar topic. May not be perfect but is close enough

6. Still think you need to do a survey?
You will have to
Identify the goals and objectives
Develop the research design
Design the questionnaire
Identify the population and select the sample
Pre-test the survey
Administer the survey
Analyze and interpret the results
Write up the results
Present the findings
Takes a lot of skilled help - -

7. Still think you need to do a survey?
Think about:
Resources in terms of TIME (Time up front in planning stage is especially important) and MONEY(Labor costs, but also, in some cases, material costs (e.g., stamps, envelopes, scannable forms, etc
How will you USE the results?

8. And if you still want to do a survey Helpful hints
Familiarize yourself with survey research ‘best practices’ and code of ethics (see
Get approval from your Institutional Review Board if needed
Organize a survey team. Include all those working on any part of the survey project. Get input from all team members at each step along the way

9. So you still want to do a survey …
Develop a detailed overall research design
What?
Why?
When?
How?
Who?
Each decision has implications for other steps!

10. What is the purpose of the survey?
Explicitly state your objectives
Example:
Improve effectiveness of simulation module developed for modeling DNA.
Identify and list factors related to objectives
Examples of factors related to “effectiveness”
Student learning outcomes
How students feel about using the simulation
What they don’t use, etc.
Make an exhaustive list of everything you want to be able to measure related to your objective.
Okay to include multiple objectives

11. Who do you need to survey?
Identify your population(s)
Survey everyone in a specific population.
If you
Have a relatively small population
Have reason to expect a relatively low response rate
Want to have enough respondents to do sub-group analyses for small groups

12. Who do you need to survey?
A sample of the population
If you
Understand sampling procedures
Can convince others that it’s okay to sample
Want/need to limit time in the field
Want to enhance ability for quality control over entire process
Want to keep costs down

13. Select the sample Probability sampling methods Simple random sample
List elements in the sampling frame and use random number chart or computer program to select elements
Systematic sampling
List elements in the sampling frame and take every Nth element, based on how many needed in sample
Warning: Be attentive to any inherent patterns in how the sampling frame is ordered.
Stratified sampling
Draw appropriate proportion of elements from homogenous groups you are interested in

14. Select the sample Sample size? Depends on: Cost
Anticipated response rate
Number of responses needed per subgroup interested in
Level of precision want (“margin of error”)
See Appendix A for more information on sampling.

15. Select the sample Sample Warnings BAD SAMPLE = BAD DATA
Survey results are only representative of the sampling frame from which the sample was selected.
Sample survey results are generalizable only if respondents are randomly selected.
Problem = sampling frame is rarely perfect…
Out of date
Incomplete
Etc.
“Probability sample” = Everyone in sampling frame has an equal chance of being selected.

16. A Warning About Response Rates! are ‘biased’ in some way
Select the sample
A Warning About Response Rates!
A high response rate
does NOT mean good data
if those who responded
are ‘biased’ in some way

17. Select the sample A Warning about Web Surveys!
If you can not control who
accesses and responds to a web survey
the results are NOT
generalizable to the population.

18. Design the questionnaire
REMEMBER YOUR OBJECTIVES!
What do you want to learn?
How will you use the information?

19. Design the questionnaire
Don’t reinvent the wheel!!
Develop (or borrow*) questions/items to measure your specific outcomes
*In general it is perfectly acceptable to borrow questions from other surveys, but always get permission and/or check for copyright restrictions.

20. Design the questionnaire
To have confidence in your survey results your questions must be
Reliable
Questions consistently convey the same meaning to all people in the population being surveyed.
Valid
Questions measure what they are intended to measure

21. Design the questionnaire
Warning!
Bad Question = Bad Data
The way a question is worded
and the response options offered
determine the nature of the data received.

22. Design the questionnaire
Open-end questions
Pros
Stimulate free thought
Solicit suggestions
Clarify positions
Richer information
Cons
Respondent burden
Incomplete, irrelevant, uninterpretable responses
Coding, analyzing, reporting

23. Design the questionnaire
Suggestion for
open-end questions!
Limit the number of open-end questions
BUT always include
at least one
for additional comments.

24. Design the questionnaire
“Forced-choice” (closed-ended) questions
Pros
Less demanding on respondent
Determine levels of intensity, frequency of participation, etc.
Use to form scales
Easy to analyze and make comparisons
Cons
Limits richness of data

25. Good Questions… Are clear and use simple language Are concise
Are specific
Are possible to answer
Don’t overly tax the respondent’s memory
Are not overly sensitive
Are relevant to the respondent
Do not use double negatives
Avoid biased terms
Have only 1 part (not “double-barrel”)

26. Good Questions… Most importantly… Can be connected to your objectives
and
Provide usable information

27. Good Questions… Good Questions: Use simple language...
My professor gives me clear feedback on my written work.
Is better than :
My professor provides lucid feedback when transmitting her thoughts regarding my written work.

28. Good Questions… Good Questions: Are concise...
How interested would you be in a flexible study program in which you could take some courses at home, online?
Is better than:
How interested would you be in a flexible study program that would still require you to complete the same credit hours, but for some courses you could not physically come to the campus but work from home online?

29. Good Questions… Good Questions: Are specific...
In which semester did you first enroll as a full-time student at NC State University?
Is better than:
When did you start studying here?

30. Good Questions… Good Questions:
Are possible for the respondent to answer...
NO:
The time I spend in the library has a direct effect on my grade.

31. Good Questions… Good Questions:
Do not overly tax the respondent’s memory...
NO:
How many of your peers said hello to you on the first day of class?

32. Good Questions… Good Questions: Are not overly sensitive… NO:
How often do you take time out to pray during the course of a normal day?

33. Good Questions… Good Questions: Are relevant... NO:
My professor should get a better haircut.

34. Good Questions… Good Questions: Do not use double negatives...
The lack of flexibility in my school hours makes it difficult for me to spend the amount of time I would like with my family.
Is better than:
The inflexibility in my school hours does not make it difficult for me to spend the amount of time I would like with my family.

35. Good Questions… Good Questions:
Do not use biased terms or suggestions...
To what extent do you agree or disagree that all students need to be on campus during regular hours?
Is better than:
Don’t you agree with the Chancellor when he says that all students need to be on campus during regular hours?

36. Good Questions… Good Questions:
Do not ask two questions in one (“double- barrel”)...
My professor is knowledgeable about xxxxx
My professor takes the time to make sure I understand xxxxxx
Is better than:
My professor is knowledgeable about xxxx and takes the time to make sure I understand them as well.

37. Response Options Make sure your response options:
Reflect the concepts you are trying to measure
Fit with the question wording
Get at level of precision/intensity needed
Allow for comparisons between items
Use “undecided/don’t know’ sparingly
Are used consistently
Use balanced scales
Are mutually exclusive
Are exhaustive

38. Response Options Most importantly Relate back to your objectives
Correspond to how want to use the results

39. Response Options Must use Balanced scales Example: NOT balanced...
How would your rate your professor’s lab classes?
Outstanding
Excellent
Very good
Good
Example: balanced...
Fair
Poor

40. Response Options Must be:
Mutually exclusive (can select only one appropriate answer)
Example: NOT mutually exclusive...
Q. On most days, what time do you leave your house for school?
6:00 AM or earlier :30-8:00 AM
6:00-6:30 AM :00-8:30 AM
6:30-7:00 AM :30-9:00 AM
7:00-7:30 AM :00 AM or later

41. Response Options Mutually exclusive continued
Example: IS mutually exclusive...
Q. On most days, what time do you leave your house for school?
Before 6:00 AM :30-7:59 AM
6:00-6:29 AM :00-8:29 AM
6:30-6:59 AM :30-8:59 AM
7:00-7:29 AM :00 AM or later

42. Response Options Exhaustive Must be:
all possible answers are listed (including e.g., “other, “ “don’t know,” etc.)
Example: NOT exhaustive
Q. On most days, what time do you leave your house for campus?
7:00-7:29 AM
7:30-7:59 AM
8:00-8:29 AM
8:30-8:59 AM

43. Response Options Exhaustive continued Example: IS exhaustive
Q. On most days, what time do you leave your house for campus?
Before 7:00 AM
7:00-7:29 AM
7:30-7:59 AM
8:00-8:29 AM
8:30-8:59 AM
9:00 AM or later

44. Response Options Some common scales: Excellent, Good, Fair, Poor
Strongly agree, Agree, Disagree, Strongly disagree
Very useful, Somewhat useful, Not very useful, Not at all useful
Definitely, Probably, Probably not, Definitely Not
All of the time, Most of the time, Some of the time, Seldom, Never
Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree

45. Questionnaire format As short as possible Spread out & uncluttered
Attractive & professional looking
Broken into logical sections
Have a good flow
Uncomplicated & easy to follow
Include simple instructions for answering questions
Should have clear skip patterns for contingency questions
Length – keep objectives in mind. Don’t ask for info won’t use. Don’t ask for info can get elsewhere.
Uncluttered – better to have more pages than cluttered pages
Attractive – makes a huge difference
Logical sections/flow – easier for respondent
Easy to follow – have clear skip patterns (use arrows, shading, etc)

46. Questionnaire format Question order Remember –
Start with interesting, easy, non-threatening questions
Remember –
‘Question order effects’
Earlier questions provide information and context that influences responses to later questions

47. Pilot A pilot or pre-test is important! Test for: Question clarity
Questionnaire format
Variance in responses
Etc.
In a perfect world, pre-test your sample design, data collection methods, data processing, and data analysis if possible in addition to questionnaire and cover letter.
Use respondents similar to those who will be in survey sample
Question clarity: failure to answer? multiple answers? “other” answers? qualified answers? understanding of content?
Questionnaire format: instructions? flow? layout? Variance in responses (do most respondents give the same answer to a question?)

48. Data collection Include (in a perfect worlds) Preliminary announcement
Cover letter, questionnaire w/ return envelope and/or survey URL
Reminder (with copy of questionnaire and return envelope and/or survey URL)
2nd reminder (with contact info to request copy of questionnaire and/or URL)
Thank you card/page

49. Data Collection Instructions should
Emphasize confidentiality (if applicable)
Provide contact information
Briefly explain
purpose of survey
how results will be used
why respondent was selected
why participation is important
how to access questionnaire
THANK respondent for participating
Incentives are good!
Confidentiality is critical
Confidentiality is NOT the same as anonymous

50. Clean, analyze, & interpret the data
Remember your objectives
Remember your report outline
Remember your audience
Keep it simple…
You can always do more analyses…

51. Interpreting the results
Everything is relative!
Suggestions for what to look for:
Do responses/ratings meet pre-defined goals?
(e.g., % giving a positive rating)
How do responses/ratings for individual items compare to each other?
What factors/characteristics are related to ratings?
How do responses/ratings from one group of respondents compare to another?
(How have ratings changed over time?)

52. Prepare written report(s)
Generally include
What you did (methods)
Why you did it
What you learned
How it can be used

53. Want to learn more about surveys?
American Association of Public Opinion Research, “Best Practices for Survey and Public Opinion Research.”
Babbie, Earl (2002), The Basics of Social Research 2nd ed. Wadsworth Thomson Learning: CA.
Dillman, D.A. (2000), Mail and Internet Surveys 2nd ed. Wiley: NY.
Fink, Arlene and Jacqueline Kosecoff (1998), How to Conduct Surveys. Sage: CA.
Fowler, Floyd J. (1993), Survey Research Methods. Sage: CA.
Fowler, Floyd J. (1995), Improving Survey Questions. Sage: CA.
National Council on Public Polls, “20 Questions a Journalist Should Ask About Poll Results.”
Sudman, Seymour and Bradburn, Norman (1982), A Practical Guide to Questionnaire Design. Jossey-Bass: San Francisco.
Tourangeau, R., and Smith, T.W. (1996), “Asking Senstive Questions: The Impact of Data Collection Mode, Question Format, and Question Context, “ Public Opinion Quarterly, 60:

54. Appendix A Response Rate # surveys completed and returned
net sample size*
*net sample size = sample size - undeliverable surveys

55. Appendix A Margin of Error Based on
# people in the sample (or population)
# people who respond
the response distribution
e.g., 75% said yes, 25% said no
how sure you want to be of you data (“confidence interval”)
e.g., a 95% confidence interval means that you can be sure that 95 out of 100 times the responses to a given question would be within some calculable percentage points of the actual number you got.

56. Appendix A Margin of Error Examples (at 95% confidence interval)
Population Size # Respondents Margin of Error
E.G. If 50% of the 7 respondents (from your population of 10) said “yes,” you can be sure that 95 out of 100 times between 38.9% and 61.1% of those in your population would say “yes.”