1. A Hands-On Approach to Developing Survey Tools
CBR 303:
A Hands-On Approach to Developing Survey Tools
in Community-Based Research

2. Welcome & Introductions
Name & affiliation
Experience with CBR and/or survey tools
What you hope to learn today

3. Objectives Upon completion of this workshop you will be able to:
Differentiate between what you need to know and what is nice to know
Use various strategies for asking survey questions
Apply different types of response options
Recognize various approaches for administering surveys
Describe how to increase response-rates
Develop a strategy to manage data
Identify various strategies for presenting survey results

4. Agenda Review of CBR Definition and purposes of a survey
Identify features of ‘best’ surveys
Steps in the survey process
Establish research objectives; identify what you need to know vs. what is nice to know
Begin to develop research questions
Refine research questions

5. What is CBR?
When you hear “community-based research,” what words or phrases come to mind?
How would you define it?

6. CBR definition Minkler & Wallerstein (2003)
CBR is a collaborative approach to research that equitably involves all partners in the research process and recognizes the unique strengths that each brings. CBR begins with a research topic of importance to the community with the aim of combining knowledge and action for social change to improve community health and eliminate health disparities.
Minkler & Wallerstein (2003)
Community-Based Participatory Research for Health
Jossey-Bass (2004)

7. Principles of CBR

8. Introduction to surveys
Why do we do surveys?
What questions can they answer?
What questions can’t they answer?
When is doing a survey a bad idea?

9. What is a survey?
A process for gathering information, without detailed verification, on the activity being examined.
The systematic collection, analysis and interpretation of information about some area of study.
Information is collected primarily by means of sampling and interviews with the selected individuals.

10. Survey Goals To produce detailed data that will allow researchers to:|
describe the characteristics of the group under study,
test theories about the group, and
generalize results beyond just those people who responded to the survey

11. Purpose for Surveys Understand the specific activity under review
Identify significant areas warranting special emphasis
Obtain information for use in making organizational changes
Identify future changes/goals for the organization

12. Features of the “BEST” Surveys *
Specific objectives
Straightforward questions
Sound research design
Sound choice of population or sample
Reliable & valid instruments
Appropriate analysis
Accurate reporting of survey results
Reasonable resources
* From The Survey Handbook, Arlene Fink, 1995

13. Survey process Assemble your team
Establish your research objectives and questions
Determine your sample
Choose interview methodology
Design questionnaire
Pre-test the questionnaire
Administer survey
Collect and enter data
Analyze data
Present data

14. 1. Assemble your team
Statistician +/ Social Scientist +/Epidemiologist
Community members +/ Community leaders +/ Service providers
Remember to consult your team early and often during planning, implementation analysis and dissemination.

15. 2. Establish Research Objectives & Questions
What do you want to know?
What do you need to know?
What will you measure? Why this factor?
How will the information be used?
Is a survey the best way to get the information you need?
What kinds of resources (time, staff, money) do you have in place to make this happen?
Try not to mix too many objectives into a single survey. Keep it focused.

16. Example – Research Objective
To identify the factors that lead to early school withdrawal for lower income youth in the community

17. Don’t Reinvent the Wheel
Before you develop your own tools see what other people have done.
Is it worth redoing a study that has been done 100 times?
What makes your research new?
How do others ask similar types of questions? (check the literature)
Can you use or adapt a pre-existing instrument to meet your needs?

18. Standardized vs. Homemade Instruments
can compare your results
hard to compare
less costly
more costly
may not be totally appropriate
can tailor to your needs, may be more relevant
higher credibility
credibility depends of investigators
may not exist!
always creatable!

19. Next … Hypothesis
X  Y
Lower Socio-economic Status  Early School Withdrawal

20. ‘Operationalizing’ Variables
Lower Socio-economic Status
Actual Income
Occupation
Work Shift/Conditions
Time Spent with children
Ability to provide educational supports
Early School Withdrawal
Withdrawal from school prior to completion of Grade 12
Marketable skills
Relationship with school staff
Relationship with peers
Social attitudes

21. From Concept to Questions Example: Socio-economic Status
Construct
Socio-economic status
Indicator
Income level
Operationalized Variables
What was your total family income during the past 12 months?
Measurement
Under $10,000
$10,001-$25,000
c) $25, $50,000
d) $50, ,000
e) more than $100,001

22. From Concept to Questions Example: Socio-economic Status
Construct
Socio-economic status
Indicators
Educational attainment
Operationalized Variables
What is the highest level of school you have completed?
Measurement
Primary School
Some High School
c) Completed High School
d) Some College
Completed College
Graduate/Professional Degree

23. Exercise: From concept to questions
On your Survey Development Template:
Identify the hypothesis (e.g. X causes Y) underlying your objective.
List the related factors for each variable.
Develop a related research question, operationalizing your variables.
Develop a specific measurement for the question.

24. Important Tips!
Each question on your survey needs to be JUSTIFIED and LINKED to your research objectives
Obtain ALL of the ESSENTIAL information
Obtain ONLY the ESSENTIAL information

25. Example: Possible question topics
Neighborhood
What their parents do
Extracurricular activities
Hobbies
Relationships with teachers
Relationship with peers
Relationship with parents
Part-time employment
Age
Sex
Sexuality
Experiences in school
Level of academic completion
Experiences of success
SES
School

26. Triaging Questions High: Absolutely essential
Medium: Very valuable for decision making
Low: Supportive data to enhance understanding

27. Strategies for asking questions
Different Types of Questions

28. Open-Ended The respondent answers in their own words
More demanding and time consuming for respondents
Lower response rate
Harder to categorize, analyze and interpret
Good when you don’t know how people will respond, useful in the preliminary pilot stage
E.g. Please describe any barriers that you face accessing our service?

29. Closed/Structured The respondent answers from a range of choices
Less time consuming for respondents
Higher response rate
Easier to categorize, analyze and interpret
Requires more front-end effort (designing the questions)
Can oversimplify or make assumptions
Response categories must be inclusive
Limits answers to options presented
Can compromise with ‘other:____________’ category

30. Types of questions: Two choices
Do you like ice cream?
Yes No

31. Types of questions: Multiple choice
Which is your favourite ice cream flavour?
Chocolate
Vanilla
Strawberry
Banana
Other: __________ (please specify)

32. Types of questions: Checklist
Please select all of the ice cream flavours that you have eaten in the last month?
Chocolate
Vanilla
Strawberry
Banana
Other: __________ (please specify)

33. Types of questions: Ranking
Please order the following ice cream flavours by preference (where 1=your favourite & 4 = your least favourite)?
Chocolate
Vanilla
Strawberry
Banana

34. Types of questions: Rating/Likert Scale
Please rate the degree to which you agree/disagree with the following statements:
1. Ice cream is my favourite dessert.
Strongly agree Agree Disagree Strongly Disagree
2. Ice cream's only redeeming feature is that it can have chocolate.
Strongly agree Agree Disagree Strongly Disagree

35. Helpful options Excellent – good – fair – poor
Approve/Disapprove, Favour/Oppose,
Too many – about right – not enough
Often – Regularly – Seldom
Always – most of the time- some of the time- rarely – never
More likely – no difference – less likely
(Agree) (Disagree)

36. Evens or Odds Even numbered scales force a choice
Odd numbered scales allow for a neutral stance
Beware of the forced choice: research shows that those who would have gone for neutral are more likely to go for positive choices when forced!
Beware of neutral categories: research shows that people ‘revert to the mean’ (ie they are most likely to choose neutral).

37. Types of questions: Numerical
How many ice cream cones have you eaten in the last week? _____
How old are you? ______
What year were you born?_____

38. Tips for Question Design
Don’t ask questions that you would not answer
Focus on behaviours not labels
Keep your values/assumptions out of questions
Make sure that questions are not too demanding
Avoid abbreviation or professional jargon
Use simple, direct and familiar words/terms (slang can be OK)

39. Tips for question design cont’d …
Make questions clear, specific and as short as possible
Make sure everyone will interpret the question the same way
Specify the frame of reference (e.g. time, place)
Avoid leading questions
Be aware of ‘social desirability’
Make sure that questions are applicable to all respondents

40. Ordering your questions
Always give an intro (title, sponsor, purpose etc)
Start with easy questions that everyone can answer (builds trust & confidence)
Introduce sensitive stuff gradually

41. Ordering your questions
There should be a logic to sequence and flow, use transition statements or subheadings
Move from general to specific
Ask about things in chronological order
Always say thank you at the end and leave room for questions/comments/feedback

42. Beware!
Common Survey Errors!

43. Problem questions: The double barrel
Beware of the “ANDs”
Do you like to watch TV and eat potato chips?
How would you rewrite?

44. Problem Questions: The Incompetent Experts
Sometimes people don’t know enough to answer the questions or may not be able to estimate the answers
To what extent do you agree with the economic foreign policy of Latvia?
How many cans of soup have you bought in the last year?

45. Problem questions: The long list
Sometimes having questions with too many response categories can be overwhelming, especially when you are asking folks to rank order them:
Please rank order these TV shows (where 1 is your favourite and 17 is your least favourite)…

46. Problem questions: The double negative
I disagree that the statement is false…

47. Problem questions: Not mutually exclusive
How old are you?
20-25
25-30
30-35
35-45

48. Problem questions: Vague wording
Do you have a partner?
Yes No

49. Problem questions: The mismatch
Do you work out?
Strongly agree
Agree
Disagree
Strongly disagree

50. The Vague Response
When is your birthday? __________
Answer: tomorrow

51. Sex, drugs and all the good stuff
How do we Ask about Sensitive Topics?
Sex, drugs and all the good stuff

52. Asking sensitive questions
Don’t ask what you would not answer!
Consider social desirability bias
Some strategies for asking sensitive questions include:
Self-administered questionnaires
Ensuring anonymity or confidentiality
Careful wording
Providing appropriate response ranges

53. Ways of Asking Threatening Questions
Various techniques have been developed to ask embarrassing or sensitive questions in non-threatening ways.
The following are examples of several such techniques as applied to the question:
"Did you kill your boss?"

54. Ways to ask: “Did you kill your boss?”
The Casual Approach
"Did you happen to have killed your boss?“
The Numbered Card
“Please read the number which corresponds with what happened to your boss.“
1. Natural death
2. I killed him/her
3. Other (please specify)

55. “Did you kill your boss?”
The “Everybody’s Doing It” Approach
"As you may be aware, many people have been killing their bosses these days. Would you happen to have killed yours?"
The “Sandwich” Method
"Have you ever done any of the following?"
a. Fire-bombed the Parliament buildings
b. Murdered your boss
c. Detonated a nuclear device

56. “Did you kill your boss?”
The Assumption Method
"How many of your bosses have you killed in the last ten years of work?“
The “Other People” Approach
a. "Do you know of other people who have killed their bosses?"
b. “Have you done that yourself?“

57. “Did you kill your boss?”
The Illustrative Technique
“Which of the following pictures most accurately depicts what happened to your boss?"

58. Designing a good survey is hard work
Give Yourself Plenty of Time!
Designing a good survey is hard work

59. Pilot Testing TEST! TEST! TEST! TEST! TEST!
Test with colleagues & participants – the more the better.
This will save you time in the long run!
Discover
Poor question wording or ordering or instructions
Suggest additional response categories
Time issues

60. Getting your questions answered….
Who are you going to talk to?
Getting your questions answered….

61. 3. Determine Your Sample Who will you interview?
Who is your clientele?
Will this group provide you with the necessary data to meet the goals of the project?
Keep the sample large enough for a sufficient amount of data but small enough so that it is manageable.

62. Sampling Eligibility or inclusion/exclusion criteria
Who do you want to talk to?
Who do you not want to hear from?

63. Inclusion & Exclusion Research Question Target Population
How does socio-economic factors impact on students’ ability to remain in school?
Target Population
High School Students
Inclusion Criteria
13-18 years old
Students in school & Early school leavers
Parental and youth consent
Exclusion Criteria
Use alcohol, marijuana or other drugs more than once a week

64. Sampling Probability Sampling (ideal) Simple random sampling
Stratified random sampling
Systematic random sampling
Cluster random sampling
Non-random sampling (hard-to-reach, pilot)
Whole group
Convenience
Snowball
Quota

65. Cost Effect Size Response rate
Sample Size
Best way to determine this is to talk to a statistician!
Considerations include:
Cost
Effect
Size
Response rate

66. Getting your questions answered….
How are you going to do it?
Getting your questions answered….

67. 4. Choose an Interview Methodology
How will you conduct the interview?
mail out
Questionnaires
individual interviews
case study
focus group
on-line surveys
Consider the make up of your clientele: language; accessibility options; age; education

68. Types: Face to Face (door to door)
BENEFITS
Interviewer administers survey
Ensures consistency & complete response
Allows for clarification & probing
Good if there are lots of ‘skips’
High response rate
Takes care of literacy issues
CONSIDERATIONS
Expensive
Confidentiality & privacy concerns
Interviewer bias
Social desirability bias may be stronger

69. Types: Telephone CONSIDERATIONS BENEFITS
Less response rate than face to face
Miss those without a telephone
Hard in this climate of ‘telemarketers – must be SHORT!
BENEFITS
Interviewer administers survey
Ensures consistency & complete response
Allows for clarification & probing
Good if there are lots of ‘skips’
Less expensive than face to face

70. Types: Self-completed (mail, web)
BENEFITS
Survey completed by respondent
Low-cost
Can be good if there is lots of ‘confidential’ or ‘embarrassing’ questions
Increase anonymity of responses
CONSIDERATIONS
Requires literacy
Variable completeness of answers
Bad if there are lots of ‘skips’ or confusing parts
Lower response rate

71. Factors to Consider when Choosing a Method
Cost, budget
Human resources
Equipment
Expertise
Who is likely to be reached/not reached
What is your target population most likely to respond to?
Bias
Response rate
Participant burden
Follow-up necessary?
Timelines
Sample size

72. Getting your questions answered….
But nobody wants to talk to me?
Getting your questions answered….

73. Improving Response Rate
Keep things short & simple
Ensure confidentiality
Target the appropriate population
Make it easy/convenient
Clearly communicate purpose, importance & relevance
Give advance notice
If you can – provide rewards or incentives

74. Increasing response rate: Layout matters!
Create a booklet
Print on white paper, don’t look like an ad
No questions on 1st page – just title
No questions on last page – room for comments
Order fun/easy questions 1st, harder in the middle and demographics at the end
The first question matters: it should be easily answerable!
The layout should be neat and clean and easy to follow
From Dillman handout

75. Getting your questions answered….
What do I with all these surveys?
Getting your questions answered….

76. 8. Manage Your Data
Expect to collect data 2-3 weeks after distributing
Stay organized
Remember confidentiality
Determine ahead of time (before or while survey is being developed) data management program (e.g. NVIVO or NuDist for qualitative; SPSS, SAS, Stata for quantative)
Determine ahead of time who will enter data; provide them with adequate training
Talk to your statistician!

77. 9. Analyze Data Identify themes & patterns
Frequencies (counts of things, how many times things happen; get mean, median, mode)
Descriptives (percentages or raw numbers of demographics and other factors that would be nonsensical to take an average such as gender or ethnicity)
Outliers (things that fall outside the expected values)

78. My questions are answered….
Presenting Results
My questions are answered….
Numbers can be deceiving…

79. ALWAYS CONTEXTUALIZE YOUR DATA
10. Report Results
Make sure to acknowledge both the strengths & limitations of your data beginning with strengths!
ALWAYS CONTEXTUALIZE YOUR DATA
How many people did you talk to?
How did you decide who to talk to? (random, convenience)
Who did you talk to?
How easy is it to generalize your results?

80. Presentation Matters
Tables
Graphs
Pictures
Words

81. Features of the ‘best’ Surveys
Specific objectives
Straightforward questions
Sound research design
Sound choice of population or sample
Reliable & valid instruments
Appropriate analysis
Accurate reporting of survey results
Reasonable resources
From The Survey Handbook, Arlene Fink, 1995

82. Reflection & Next Steps
Reflect on and record 1-3 things you learned today.
Record 1-3 related goals you have going forward.
Record 1-3 related next steps you plan to take upon leaving the workshop.

83. Objectives Having completed this workshop you should now be able to:
Differentiate between what you need to know and what is nice to know
Use various strategies for asking survey questions
Apply different types of response options
Recognize various approaches for administering surveys
Describe how to increase response-rates
Develop a strategy to manage data
Identify various strategies for presenting survey results

84. Acknowledgements
Several of the slides presented here today were developed by San Patten, MSc., Community-Based Research Coordinator for the Alberta Community Council on HIV

85. A Hands-On Approach to Developing Survey Tools
CBR 303:
A Hands-On Approach to Developing Survey Tools
in Community-Based Research