Presentation on theme: "A Hands-On Approach to Developing Survey Tools"— Presentation transcript:
1 A Hands-On Approach to Developing Survey Tools CBR 303:A Hands-On Approach to Developing Survey Toolsin Community-Based Research
2 Welcome & Introductions Name & affiliationExperience with CBR and/or survey toolsWhat you hope to learn today
3 Objectives Upon completion of this workshop you will be able to: Differentiate between what you need to know and what is nice to knowUse various strategies for asking survey questionsApply different types of response optionsRecognize various approaches for administering surveysDescribe how to increase response-ratesDevelop a strategy to manage dataIdentify various strategies for presenting survey results
4 Agenda Review of CBR Definition and purposes of a survey Identify features of ‘best’ surveysSteps in the survey processEstablish research objectives; identify what you need to know vs. what is nice to knowBegin to develop research questionsRefine research questions
5 What is CBR?When you hear “community-based research,” what words or phrases come to mind?How would you define it?
6 CBR definition Minkler & Wallerstein (2003) CBR is a collaborative approach to research that equitably involves all partners in the research process and recognizes the unique strengths that each brings. CBR begins with a research topic of importance to the community with the aim of combining knowledge and action for social change to improve community health and eliminate health disparities.Minkler & Wallerstein (2003)Community-Based Participatory Research for HealthJossey-Bass (2004)
8 Introduction to surveys Why do we do surveys?What questions can they answer?What questions can’t they answer?When is doing a survey a bad idea?
9 What is a survey?A process for gathering information, without detailed verification, on the activity being examined.The systematic collection, analysis and interpretation of information about some area of study.Information is collected primarily by means of sampling and interviews with the selected individuals.
10 Survey Goals To produce detailed data that will allow researchers to:| describe the characteristics of the group under study,test theories about the group, andgeneralize results beyond just those people who responded to the survey
11 Purpose for Surveys Understand the specific activity under review Identify significant areas warranting special emphasisObtain information for use in making organizational changesIdentify future changes/goals for the organization
12 Features of the “BEST” Surveys * Specific objectivesStraightforward questionsSound research designSound choice of population or sampleReliable & valid instrumentsAppropriate analysisAccurate reporting of survey resultsReasonable resources* From The Survey Handbook, Arlene Fink, 1995
13 Survey process Assemble your team Establish your research objectives and questionsDetermine your sampleChoose interview methodologyDesign questionnairePre-test the questionnaireAdminister surveyCollect and enter dataAnalyze dataPresent data
14 1. Assemble your teamStatistician +/ Social Scientist +/EpidemiologistCommunity members +/ Community leaders +/ Service providersRemember to consult your team early and often during planning, implementation analysis and dissemination.
15 2. Establish Research Objectives & Questions What do you want to know?What do you need to know?What will you measure? Why this factor?How will the information be used?Is a survey the best way to get the information you need?What kinds of resources (time, staff, money) do you have in place to make this happen?Try not to mix too many objectives into a single survey. Keep it focused.
16 Example – Research Objective To identify the factors that lead to early school withdrawal for lower income youth in the community
17 Don’t Reinvent the Wheel Before you develop your own tools see what other people have done.Is it worth redoing a study that has been done 100 times?What makes your research new?How do others ask similar types of questions? (check the literature)Can you use or adapt a pre-existing instrument to meet your needs?
18 Standardized vs. Homemade Instruments can compare your resultshard to compareless costlymore costlymay not be totally appropriatecan tailor to your needs, may be more relevanthigher credibilitycredibility depends of investigatorsmay not exist!always creatable!
19 Next … HypothesisX YLower Socio-economic Status Early School Withdrawal
20 ‘Operationalizing’ Variables Lower Socio-economic StatusActual IncomeOccupationWork Shift/ConditionsTime Spent with childrenAbility to provide educational supportsEarly School WithdrawalWithdrawal from school prior to completion of Grade 12Marketable skillsRelationship with school staffRelationship with peersSocial attitudes
21 From Concept to Questions Example: Socio-economic Status ConstructSocio-economic statusIndicatorIncome levelOperationalized VariablesWhat was your total family income during the past 12 months?MeasurementUnder $10,000$10,001-$25,000c) $25, $50,000d) $50, ,000e) more than $100,001
22 From Concept to Questions Example: Socio-economic Status ConstructSocio-economic statusIndicatorsEducational attainmentOperationalized VariablesWhat is the highest level of school you have completed?MeasurementPrimary SchoolSome High Schoolc) Completed High Schoold) Some CollegeCompleted CollegeGraduate/Professional Degree
23 Exercise: From concept to questions On your Survey Development Template:Identify the hypothesis (e.g. X causes Y) underlying your objective.List the related factors for each variable.Develop a related research question, operationalizing your variables.Develop a specific measurement for the question.
24 Important Tips!Each question on your survey needs to be JUSTIFIED and LINKED to your research objectivesObtain ALL of the ESSENTIAL informationObtain ONLY the ESSENTIAL information
25 Example: Possible question topics NeighborhoodWhat their parents doExtracurricular activitiesHobbiesRelationships with teachersRelationship with peersRelationship with parentsPart-time employmentAgeSexSexualityExperiences in schoolLevel of academic completionExperiences of successSESSchool
26 Triaging Questions High: Absolutely essential Medium: Very valuable for decision makingLow: Supportive data to enhance understanding
27 Strategies for asking questions Different Types of Questions
28 Open-Ended The respondent answers in their own words More demanding and time consuming for respondentsLower response rateHarder to categorize, analyze and interpretGood when you don’t know how people will respond, useful in the preliminary pilot stageE.g. Please describe any barriers that you face accessing our service?
29 Closed/Structured The respondent answers from a range of choices Less time consuming for respondentsHigher response rateEasier to categorize, analyze and interpretRequires more front-end effort (designing the questions)Can oversimplify or make assumptionsResponse categories must be inclusiveLimits answers to options presentedCan compromise with ‘other:____________’ category
30 Types of questions: Two choices Do you like ice cream?YesNo
31 Types of questions: Multiple choice Which is your favourite ice cream flavour?ChocolateVanillaStrawberryBananaOther: __________ (please specify)
32 Types of questions: Checklist Please select all of the ice cream flavours that you have eaten in the last month?ChocolateVanillaStrawberryBananaOther: __________ (please specify)
33 Types of questions: Ranking Please order the following ice cream flavours by preference (where 1=your favourite & 4 = your least favourite)?ChocolateVanillaStrawberryBanana
34 Types of questions: Rating/Likert Scale Please rate the degree to which you agree/disagree with the following statements:1. Ice cream is my favourite dessert.Strongly agree Agree Disagree Strongly Disagree2. Ice cream's only redeeming feature is that it can have chocolate.Strongly agree Agree Disagree Strongly Disagree
35 Helpful options Excellent – good – fair – poor Approve/Disapprove, Favour/Oppose,Too many – about right – not enoughOften – Regularly – SeldomAlways – most of the time- some of the time- rarely – neverMore likely – no difference – less likely(Agree) (Disagree)
36 Evens or Odds Even numbered scales force a choice Odd numbered scales allow for a neutral stanceBeware of the forced choice: research shows that those who would have gone for neutral are more likely to go for positive choices when forced!Beware of neutral categories: research shows that people ‘revert to the mean’ (ie they are most likely to choose neutral).
37 Types of questions: Numerical How many ice cream cones have you eaten in the last week? _____How old are you? ______What year were you born?_____
38 Tips for Question Design Don’t ask questions that you would not answerFocus on behaviours not labelsKeep your values/assumptions out of questionsMake sure that questions are not too demandingAvoid abbreviation or professional jargonUse simple, direct and familiar words/terms (slang can be OK)
39 Tips for question design cont’d … Make questions clear, specific and as short as possibleMake sure everyone will interpret the question the same waySpecify the frame of reference (e.g. time, place)Avoid leading questionsBe aware of ‘social desirability’Make sure that questions are applicable to all respondents
40 Ordering your questions Always give an intro (title, sponsor, purpose etc)Start with easy questions that everyone can answer (builds trust & confidence)Introduce sensitive stuff gradually
41 Ordering your questions There should be a logic to sequence and flow, use transition statements or subheadingsMove from general to specificAsk about things in chronological orderAlways say thank you at the end and leave room for questions/comments/feedback
43 Problem questions: The double barrel Beware of the “ANDs”Do you like to watch TV and eat potato chips?How would you rewrite?
44 Problem Questions: The Incompetent Experts Sometimes people don’t know enough to answer the questions or may not be able to estimate the answersTo what extent do you agree with the economic foreign policy of Latvia?How many cans of soup have you bought in the last year?
45 Problem questions: The long list Sometimes having questions with too many response categories can be overwhelming, especially when you are asking folks to rank order them:Please rank order these TV shows (where 1 is your favourite and 17 is your least favourite)…
46 Problem questions: The double negative I disagree that the statement is false…
47 Problem questions: Not mutually exclusive How old are you?20-2525-3030-3535-45
48 Problem questions: Vague wording Do you have a partner?YesNo
49 Problem questions: The mismatch Do you work out?Strongly agreeAgreeDisagreeStrongly disagree
50 The Vague ResponseWhen is your birthday? __________Answer: tomorrow
51 Sex, drugs and all the good stuff How do we Ask about Sensitive Topics?Sex, drugs and all the good stuff
52 Asking sensitive questions Don’t ask what you would not answer!Consider social desirability biasSome strategies for asking sensitive questions include:Self-administered questionnairesEnsuring anonymity or confidentialityCareful wordingProviding appropriate response ranges
53 Ways of Asking Threatening Questions Various techniques have been developed to ask embarrassing or sensitive questions in non-threatening ways.The following are examples of several such techniques as applied to the question:"Did you kill your boss?"
54 Ways to ask: “Did you kill your boss?” The Casual Approach"Did you happen to have killed your boss?“The Numbered Card“Please read the number which corresponds with what happened to your boss.“1. Natural death2. I killed him/her3. Other (please specify)
55 “Did you kill your boss?” The “Everybody’s Doing It” Approach"As you may be aware, many people have been killing their bosses these days. Would you happen to have killed yours?"The “Sandwich” Method"Have you ever done any of the following?"a. Fire-bombed the Parliament buildingsb. Murdered your bossc. Detonated a nuclear device
56 “Did you kill your boss?” The Assumption Method"How many of your bosses have you killed in the last ten years of work?“The “Other People” Approacha. "Do you know of other people who have killed their bosses?"b. “Have you done that yourself?“
57 “Did you kill your boss?” The Illustrative Technique“Which of the following pictures most accurately depicts what happened to your boss?"
58 Designing a good survey is hard work Give Yourself Plenty of Time!Designing a good survey is hard work
59 Pilot Testing TEST! TEST! TEST! TEST! TEST! Test with colleagues & participants – the more the better.This will save you time in the long run!DiscoverPoor question wording or ordering or instructionsSuggest additional response categoriesTime issues
60 Getting your questions answered…. Who are you going to talk to?Getting your questions answered….
61 3. Determine Your Sample Who will you interview? Who is your clientele?Will this group provide you with the necessary data to meet the goals of the project?Keep the sample large enough for a sufficient amount of data but small enough so that it is manageable.
62 Sampling Eligibility or inclusion/exclusion criteria Who do you want to talk to?Who do you not want to hear from?
63 Inclusion & Exclusion Research Question Target Population How does socio-economic factors impact on students’ ability to remain in school?Target PopulationHigh School StudentsInclusion Criteria13-18 years oldStudents in school & Early school leaversParental and youth consentExclusion CriteriaUse alcohol, marijuana or other drugs more than once a week
64 Sampling Probability Sampling (ideal) Simple random sampling Stratified random samplingSystematic random samplingCluster random samplingNon-random sampling (hard-to-reach, pilot)Whole groupConvenienceSnowballQuota
65 Cost Effect Size Response rate Sample SizeBest way to determine this is to talk to a statistician!Considerations include:CostEffectSizeResponse rate
66 Getting your questions answered…. How are you going to do it?Getting your questions answered….
67 4. Choose an Interview Methodology How will you conduct the interview?mail outQuestionnairesindividual interviewscase studyfocus groupon-line surveysConsider the make up of your clientele: language; accessibility options; age; education
68 Types: Face to Face (door to door) BENEFITSInterviewer administers surveyEnsures consistency & complete responseAllows for clarification & probingGood if there are lots of ‘skips’High response rateTakes care of literacy issuesCONSIDERATIONSExpensiveConfidentiality & privacy concernsInterviewer biasSocial desirability bias may be stronger
69 Types: Telephone CONSIDERATIONS BENEFITS Less response rate than face to faceMiss those without a telephoneHard in this climate of ‘telemarketers – must be SHORT!BENEFITSInterviewer administers surveyEnsures consistency & complete responseAllows for clarification & probingGood if there are lots of ‘skips’Less expensive than face to face
70 Types: Self-completed (mail, web) BENEFITSSurvey completed by respondentLow-costCan be good if there is lots of ‘confidential’ or ‘embarrassing’ questionsIncrease anonymity of responsesCONSIDERATIONSRequires literacyVariable completeness of answersBad if there are lots of ‘skips’ or confusing partsLower response rate
71 Factors to Consider when Choosing a Method Cost, budgetHuman resourcesEquipmentExpertiseWho is likely to be reached/not reachedWhat is your target population most likely to respond to?BiasResponse rateParticipant burdenFollow-up necessary?TimelinesSample size
72 Getting your questions answered…. But nobody wants to talk to me?Getting your questions answered….
73 Improving Response Rate Keep things short & simpleEnsure confidentialityTarget the appropriate populationMake it easy/convenientClearly communicate purpose, importance & relevanceGive advance noticeIf you can – provide rewards or incentives
74 Increasing response rate: Layout matters! Create a bookletPrint on white paper, don’t look like an adNo questions on 1st page – just titleNo questions on last page – room for commentsOrder fun/easy questions 1st, harder in the middle and demographics at the endThe first question matters: it should be easily answerable!The layout should be neat and clean and easy to followFrom Dillman handout
75 Getting your questions answered…. What do I with all these surveys?Getting your questions answered….
76 8. Manage Your DataExpect to collect data 2-3 weeks after distributingStay organizedRemember confidentialityDetermine ahead of time (before or while survey is being developed) data management program (e.g. NVIVO or NuDist for qualitative; SPSS, SAS, Stata for quantative)Determine ahead of time who will enter data; provide them with adequate trainingTalk to your statistician!
77 9. Analyze Data Identify themes & patterns Frequencies (counts of things, how many times things happen; get mean, median, mode)Descriptives (percentages or raw numbers of demographics and other factors that would be nonsensical to take an average such as gender or ethnicity)Outliers (things that fall outside the expected values)
78 My questions are answered…. Presenting ResultsMy questions are answered….Numbers can be deceiving…
79 ALWAYS CONTEXTUALIZE YOUR DATA 10. Report ResultsMake sure to acknowledge both the strengths & limitations of your data beginning with strengths!ALWAYS CONTEXTUALIZE YOUR DATAHow many people did you talk to?How did you decide who to talk to? (random, convenience)Who did you talk to?How easy is it to generalize your results?
81 Features of the ‘best’ Surveys Specific objectivesStraightforward questionsSound research designSound choice of population or sampleReliable & valid instrumentsAppropriate analysisAccurate reporting of survey resultsReasonable resourcesFrom The Survey Handbook, Arlene Fink, 1995
82 Reflection & Next Steps Reflect on and record 1-3 things you learned today.Record 1-3 related goals you have going forward.Record 1-3 related next steps you plan to take upon leaving the workshop.
83 Objectives Having completed this workshop you should now be able to: Differentiate between what you need to know and what is nice to knowUse various strategies for asking survey questionsApply different types of response optionsRecognize various approaches for administering surveysDescribe how to increase response-ratesDevelop a strategy to manage dataIdentify various strategies for presenting survey results
84 AcknowledgementsSeveral of the slides presented here today were developed by San Patten, MSc., Community-Based Research Coordinator for the Alberta Community Council on HIV
85 A Hands-On Approach to Developing Survey Tools CBR 303:A Hands-On Approach to Developing Survey Toolsin Community-Based Research