Presentation on theme: "**This instructor is slightly customized for each lesson."— Presentation transcript:
1 **This instructor is slightly customized for each lesson. Designing Your Survey
2 By the end of this lesson you will be able to: Define the 6 goals of the questionnaire survey.Differentiate between a biased and an unbiased survey question.Develop survey questions from preliminary objectives.
3 Design a survey that:has appropriate sample sizes and strategymeasures the impact of your Pride campaign on SMART objectives for each target audiencemeasures exposure to your Pride campaign activities
4 SESSION TOPICS AND TIMING Learning to Detect Bias and Poorly written questionsReading ActivityWriting the best possible survey questionsThursCreating Questions from Preliminary ObjectivesDrafting Your Stages of Behavior QuestionsDetermining Sample Sizes and CalculatingYour Sites Sample SizeAssignment, Rubrics and Research Plan ReviewFridaySESSION TOPICS AND TIMING
5 Activity 1: Learning to Detect Bias and Poorly Written Questions (Basics) PURPOSE:To practice recognizing “bias” on questions and identify what makes a question biased45 minutesDefining impact is key here – provide example tied to smart objective
6 Revisit the Theory of Change Defining impact is key here – provide example tied to smart objective
7 INSTRUCTIONS: ACTIVITY 2: READING Read the Designing Your Survey Lesson Content (pp ) – 20 minutesProceed to your workshop group (3-4 persons per groups) and answer the guiding questions in pages – 10 minutesRecord your groups responses and report your results (5 minutes)ACTIVITY 2: READING
8 Why Conduct an Interview Survey Surveys are research tools that provide quantitative information about people in your target area.The systematic sampling will make your respondents “representative” of the population of interest.Surveys are ideal tools to measure change in your target audiences because you can measure the “baseline” and post-campaign levels.The difference between baseline and post-campaign survey results provides an estimate of the impact of your campaign on that variable.
9 Questionnaires and your Pride Campaign You will implement 2 surveys during your campaign:During the planning phase (baseline)At the conclusion of your campimpact.
10 Goals for your survey1. Describe your Target Audience(s) “ 90 % of all fishers surveyed are male and 60 % of those are over the age of 40”
11 Goals for your survey– pre campaign 2. Assign Respondents to their Appropriate Stage-of-Behavior-Change “Among fishermen surveyed, 5 percent are currently setting rattraps on their boats, 5 percent are currently thinking about setting rattraps...”
12 Goals for your survey– pre campaign 3. Establish a Baseline for your SMART Objectives “30% of Fishers know that rats get to Serena on fishing boats” can be the basis for setting the SMART Objectives: “By 2013, the percentage of Fishers who know that rats travel to Serena on their boats will increase from 30% to 50%Insert an associated knowledge or attitude objective appropriate for your site
13 Goals for your survey– post campaign 4. Measure the impact of your PRIDE campaign in achieving your SMART ObjectivesControl siteICInterpersonal communicationsAAttitudepowerful way to do this, is to use a control group will be discussed in detail below.KKnowledge
14 Goals for your survey– pre campaign CARE FOR THE LASTING PRIDE5. Measure the Audience Exposure to your Pride Campaign and Attribute Effects to your Pride CampaignA campaign to protect Tagbilaran City’s underwater kingdom
15 Goals for your survey– pre campaign 6. Ground Truth – test your assumptions about community, strategy, and ToC"99% of community members know about the importance of keeping rats off of the island”…what does this tell you about your ToC?If you thought knowledge was low in your target audience about rats, but you find that it is very very high as in the above example, then this means your campaign may need to be more focused on attitude or IC changes and not knowledge.
17 A Note of Caution…Developing and implementing quantitative surveys is a lot of workbias questionsask questions of the wrong people, or at least not enough of the “right” ones, of your critical target audience(s)ask too many questionsmake mistakes in choosing your samples - your baseline and post-campaign samples are not comparablemake mistakes during data entry you enter the wrong data into the wrong variable and end up with meaningless resultsWe are going to provide the tools you need to design but there are a number of key challenges. Will practice these challenges so you feel comfortable dealing with thembias questions—a great deal of care is needed when crafting questions and when asking themask questions of the wrong people, or at least not enough of the “right” ones, of your critical target audience(s)ask too many questionsmake mistakes in choosing your samples, so that your baseline and post-campaign samples are not comparable to each othermake mistakes during data entry so that you enter the wrong data into the wrong variable and end up with meaningless results
18 Biased QuestionsThe most important challenge of survey questions is that they can be BIASED:Biased questions mean: they make respondents tend to pick one response over another because of the way the question is worded or the way the response options are offered.Refer to the lesson plan for examples and talk through each of the questions and the alternatives for each bold faced section on the slide in the lesson content.
19 Biased questions: Biased Questions Try to educate Assume what they ask (are leading)Are double-barreled, asking 2 questions at onceAre ambiguous or confusingRefer to the lesson plan for examples and talk through each of the questions and the alternatives for each bold faced section on the slide in the lesson content.
20 Key Elements of Good Survey Questions? Unbiased – not leading or pushing respondents towards a specific answerTied to ObjectivesTime boundClearNot JudgementalRefer to the lesson plan for examples and talk through each of the questions.
21 Writing good survey questions Other important criteria for good survey questions:Include a relevant time frameUse filter questions as neededDon’t overlap (e.g. age ranges)Use appropriate language for the respondentsRefer to the lesson plan for examples and talk through each of the questions.Include a relevant time frame – “In the last six months, have you seen anyone fishing inside the MPA?”Use filter questions as needed -Don’t overlap (e.g. age ranges)Use appropriate language for the respondents
22 Activity 3: Writing the Best possible Survey Questions Purpose:To Practice writing “good” survey questions and appropriate response choicesInstructions: (Turn to pages )Work in pairsRead through each of the survey questions assigned to you and the corresponding responses foundFor each of the question, do the following:Identify what makes this as poorly written questions
23 Survey Questionnaire #1 Original Survey Question/Poorly Written QuestionDo you know what the Ministry’s law on legal size of fish that can be killed is about?YESNONot SureRewritten Survey QuestionDoes the Ministry have a law about the legal size of fish catch? If yes, please tell me what you think the legal size of fish catch is?YESNONot SureThe question is biased and educates the respondent as to what the law is about. The “good” survey question is rewritten so it is unbiased. It is clear with either a target audience response being acceptable if they response “yes” or “no”. Finally, the rewirtten question does not educate the respondent
24 Survey Questionnaire #2 Original Survey Question/Poorly Written QuestionDo you agree that people should no longer slaughter elephants just for their tusks?YESNONot SureRewritten Survey QuestionI am going to read you a statement and I would like for you to decide if you strongly agree, agree, disagree, strongly disagree or if you have no opinion: “Should people be allowed to kill elephants for their tusks”?Strongly AgreeAgreeNo OpinionDisagreeStrongly DisagreeThe poorly written question is biased and (1) suggests that the target audience response should agree with the statement, it also uses the emotional language of “slaughter” and “just”.The rewritten question is unabiased and does not lead the respondent to answer. It is also void of emotional language.
25 Survey Questionnaire #3 Original Survey Question/Poorly Written QuestionFor what reasons have you gone into the Bungo Protected Area? Tell me all that ApplyHuntingNTFPFarmingOtherRewritten Survey QuestionI am going to read you a list of possible reasons that people sometimes go into Bungo Protected area. I would like you to tell me for each reason whether you have gone into the Bungo PA for that reason during the past 6 months.ReasonsYesNoNot SureHuntingGathering plants for medicines or foodFarmingAny other reason, please specifyThe poorly written question is biased in three ways (1) it is not time bounded, (2) assumes that they have gone into the protected area, and finally with the target audiences responses, the survey will (3) likely end up with a lot of “other” responses. As a result, it introduces bias because if they don’t mention one of the reasons you are interested in, you don’t know if they didn’t do it, or just didn’t think to mention it.The rewritten question is unbiased, time bounded (in the past six months) provides an option for respondents to say they have not gone into the Bungo PA for any reason.
26 Survey Questionnaire #5 Original Survey Question/Poorly Written QuestionHave you purchased bushmeat?YESNONot SureRewritten Survey QuestionHave you purchased or bartered for bushmeat, or meat from the wild animals, at any time in the past 6 months?YESNONot SureThe poorly written question is not time bound, so you would not be able to distinguish between someone who purchased it a year ago, but not from someone who purchased it recently (or during the timeframe of your campaign). It is also not very specific, as bushmeat can mean different things to different people and can be be bartered.The rewritten question is unbiased, precise and timebound.
27 Survey Questionnaire #6 Original Survey Question/Poorly Written QuestionHow many times would you say that you have gone into the Jamu Protected Area?Never1 to 4 times4 to 10 times10 t0 20 timesRewritten QuestionIn the last six months, how many times would you say that you have been in this area here (show them the map of the protected area)Never1 to 4 times5 to 10 timesMore than 10 timesThere are three elements of this question that can be biased. First is that it is not bound and so you wont be able to distinguish between someone who entered the protected area 5 years ago vs. last month (or within the timeframe of the campaign).Second, the protected area language can make people like they area not supposed to be in the area and then may feel not comfortable saying they had been in there.And third, the responses overlap so if they had been in the protected area 4 times, which category would they go in?The rewritten survey is unbiased, time boun, and provided a map of the protected area instead of just saying “protected area”. It also provides mutually exclusive categories for the target audiences responses. It does, however, assume that in this case, the respondents area able to read maps, which may not always be the case.
28 Survey Questionnaire #7 Original Survey Question/Poorly Written QuestionHave you talked to anyone about Jamu?YESNONot SureRewritten Survey QuestionIn the last 6 months, have you spoken to anyone in your community about the importance of protected Jamu for the health of the people who live here?YESNONot SureThe poorly written question is biased. It is not time bound and it is not clear what you want to test regarding interpersonal communication. They could have talked to someone about Jamu, and said that they hate it!The rewritten question is unbiased, time bound, and specifically asks about the type of communication regarding Jamu. In the survey, if this type of question is asked, you may want to follow up with a question about whom in the community they spoke to.
29 Survey Questionnaire #8 Original Survey Question/Poorly Written QuestionWhat are the threats to the Jamu Ecosystem?FireDroughtPoachingDevelop-mentRewritten QuestionWhat types of activities and conditions threaten the health and sustainability of important wildlife in Jamu?FireDroughtPoachingDevelop-mentOtherThe poorly written question is biased because it uses language that may not translate to people in rural areas (e.g. ecosystem). It also doesn’t include an “Other” column in the target audiences responses to capture threats that may be included in the list.The rewritten question is unbiased and uses clear language that is relevant for the target audience.
30 Survey Questionnaire #9 Original Survey Question/Poorly Written QuestionIn the past 6 months, have you ever fished illegally?YESNONot SureRewritten Survey QuestionIn the last 6 months, have you ever used dynamite fishing?YESNONot SureThe poorly written question is biased and will be sensitive to target audiences as it assumes “illegal” fishing practices. Not many fisherman will agree to tell you that they are fishing illegal for fear of retribution.The rewritten question is unbiased and still asks about illegal fishing behavior but does not identify it as illegal. However, using this same language will be subject on a case to case basis. If the community or respondents have a high sense of dynamite fishing as illegal, it might be good to rephrase the question as to “In the past six months, have you ever seen anyone fishing using dynamite” inside or outside the marine sanctuary?
31 Survey Questionnaire #10 Original Survey Question/Poorly Written QuestionDo you think that rats can spread the disease typhoid fever to humans and cause damage to nets?YESNONot SureRewritten Survey QuestionI am going to read you to you a series of statements about rats. Please tell me if you think they area true, false or you are unsure:Rat spread typhoid fever to humans.Rats cause damage to fishing netsTrueFalseNot SureThe poorly written question is a biased question because it is considered a “double barreled” question, meaning it asks two questions at the same time – typhoid fever and damage to nets.The revised version of the question is unbiased, specific and clear. It also provides an opportunity for the target audience respondent to answer only one question at a time.
32 Survey Questionnaire #11 Original Survey Question/Poorly Written QuestionDo you agree that overfishing is a problem and that something needs to be done about it?YESNONot SureRewritten Survey QuestionI am going to read you to you a series of statements and I would like you to tell me whether or not you strongly agree, agree, disagree, strongly disagree or have no opinion:Overfishing is a problemThe government should create laws to prevent overfishingThe Banga community should support a ban on overfishingStronglyAgreeSAADisagreeDStrongly DisagreeSDNo OpinionNOThe poorly written question is a biased question that forces the target audience respondent to feel like they have to agree (do you agree) and also is doubled barreled because it asks whether something should be done about it.The revised version of the question is unbiased and provides all potential target audiences responses, (SA, A, D, SD, NO). It separates the questions about “knowing” whether there is a problem from what should be done about it.
34 READING ASSIGNMENTS FOR TOMORROW Read and review the lesson content especially on Biased and Unbiased questions (pages )2. Read Lesson Content pages Write in a piece of paper 2-3 questions (per participant) that you would want to be clarified or know more.READING ASSIGNMENTS FOR TOMORROW
35 DESIGNING YOUR QUESTIONNAIRE REQUIREMENTS:Identified Preliminary Target AudiencePreliminary SMART ObjectivesDESIGNING YOUR QUESTIONNAIRE
36 How Many QUESTIONNAIRES? ONLY ONE QUESTIONNAIRE (Pre and Post KAP Survey) for all target audiencesSome questions though may not be applicable to some audienceCreate a system where some questions are answered by certain target audience while others are not.How Many QUESTIONNAIRES?
37 How Many QUESTIONS should I ask? Factors that can affect your survey are:the number of respondentslength of questionswho will administerData entryData analysis and writingCostInclude questions that will answer only the six goals of surveyHow Many QUESTIONS should I ask?
38 Where do the survey questions come from? KEY CHARACTERISTICSDateEnumerator NameForm #Geographic Area/ Sampling locationRespondent Information: Age, gender, education level, income sources, religion?Knowledge QuestionAttitude QuestionInterpersonal Communication QuestionBehavior Change QuestionStages of Behavior ChangeBarrier Removal (As Appropriate)Exposure to MaterialsIt is only after you know who you are going to interview and the preliminary objectives for your survey that you should begin to write survey questions.
39 SMART ExamplesREVISIT SMART OBJECTIVES SLIDE FROM PREVIOUS ABOUT DRAFTING SURVEY QUESTIONS FROM THE SMART OBJECTIVES… ADD CUSTOMIZED SLIDES HERE DEVELOPED FROM DRAFT OBJECTIVES
40 Behavior Change and Questionnaire Surveys Statement from stage of behavior change question in surveyStage of behavior change it should correspond toI have never considered using rat traps in my fishing boat.Validation stageI am considering using rat traps in my fishing boat but haven’t decided yet.Pre-contemplation stageI intend to use rat traps in my fishing boat in the near future, but I haven’t talked to anyone about it yet.Maintenance stageI have talked to someone about my intention to use rat traps in my fishing boat.Action StageI used rat traps in my fishing boat at least once.Contemplation stageI always use rat traps in my fishing boat.Preparation stagethe stages of change after the survey, but it is simpler for the respondent.
41 Sample Behavior Change Questions Clearly define the “desired behavior change” of your campaignMake one question each along the Stages of Behavior (Behavior Change continuum)Example:..\..\EXAMPLE behavior change questions.docx
42 Exposure QuestionsMake at least 8-10 Examples of potential materials that you will be producing in your campaign and ask the participants if they have seeSee example ..\..\EXAMPLE media exposure questions.docx
43 TOP TIPS Complete each interview in about 30 minutes or less The categories should also be “exhaustive;” that is, there is no other possible response that a respondent wants to give but cannot.Be sure that your questions directly relate to what it is you are studying, especially for those questions intended to measure SMART objectives.
44 Top Tips…..4. Be very aware of the fact that people in your target communities may not have the same vocabulary as you.5. If you want to force people to choose from a short list of possible responses, use the closed-ended format.6. If there are some questions that you feel might be sensitive you should put these at the end of the survey so that respondents don’t get upset at the beginning.
45 Top Tips7. Have one person do the first translation from the language the survey was drafted in into the second language. Then, have a different person translate the survey back into the original language (back translation). 8. “Administer the survey to a small number of respondents (five is usually enough) similar to your target population to make sure that categories are clear and complete”.
46 Key Elements of Good Survey Questions? Unbiased – not leading or pushing respondents towards a specific answerTied to ObjectivesTime boundClearNot JudgementalRefer to the lesson plan for examples and talk through each of the questions.
47 Writing good survey questions Other important criteria for good survey questions:Include a relevant time frameUse filter questions as neededDon’t overlap (e.g. age ranges)Use appropriate language for the respondentsRefer to the lesson plan for examples and talk through each of the questions.Include a relevant time frame – “In the last six months, have you seen anyone fishing inside the MPA?”Use filter questions as needed -Don’t overlap (e.g. age ranges)Use appropriate language for the respondents
48 Writing good survey questions FILTER QUESTIONSIn the past 6 months, have you talked to anyone about benefits to the community of a well enforced sanctuary? YES NOIf you have, please tell me all of the people with whom you have talked to about this.[ ] Have not talked to anyone[ ] Talked to spouse/partner[ ] Talked to parents, or in-laws[ ] Talked to your children aged 16 or older[ ] Talked to your children aged 15 or younger[ ] Talked to friend or neighbour[ ] Talked to village elder or local authority[ ] Talked to National government environmental officials[ ] Talked to Non-government environment officials[ ] Talked to Local government environmental officials[ ] Talked to a Local XXXX Fisher or Gleaner[ ] Other ________________
49 Writing good survey questions RESPONSE QUESTIONS TO CATEGORIES SHOULD NOT OVERLAPPROVIDE ACCURATE TRANSLATIONCLOSE VERSUS OPEN-ENDED QUESTIONS
50 ORDER OF QUESTIONSUse transitions between modules – use to switch from one topic to anotherProvide Accurate translationReview and Pretest your Questionnaire
51 Criteria for a Good Survey Questionnaire Uses the Rare template as a starting point, and asks the socioeconomic and demographic questions exactly as are in template.There is a clear link to the campaign’s objectives so that the data can help in campaign design, test assumptions and be used to measuring its impact across all stages of the Theory of Change.Questions cover most important factors such as:Socioeconomic and Demographic profilesStages of behavior change questionSMART objectives measurementExposure to Campaign Activities & Messages
52 Criteria ….Questions are clear, unbiased and appropriate level of language and vocabulary for the target audiences.Interpersonal communication, behavior change, and exposure to Pride activity questions are time-bound, if appropriate, so you can measure change, normally this means they state a time of less than 1 year.Questionnaire did not touch upon socially sensitive issues which introduces bias.Questionnaire takes not more than 30 minutes to administer to a respondent.Questionnaire was pretested.
53 INDIVIDUAL/ GROUP WORKTIME Drafting your Survey Questions Your Draft Questions must follow the template provided to you earlierAdd:Behavior Change QuestionsMedia Exposure Questions
55 Guide to Individual/Group Work (60 MINUTES) – 10:00-11:00Review the Serena Island Research Plan (tables 5&6) and pages in your packetUsing your preliminary Results Chains and preliminary SMART Objectives begin to develop your site's survey questions. Take the time to develop survey questionsIncorporate the newly developed survey questions into the Quantitative Data Collection section of your site's Research Plan. (Section 5.4 & 5.5)(30 MINUTES) 11:00-11:30Share with another CF within your groupConsult/ Share with your PPM on your Survey Questions
56 Other Considerations Filter Questions Response Categories for Questions Should not OverlapClosed versus Open-ended QuestionsOrder of questionsProvide accurate translationReview and pretest your questionnaire
57 Selecting Your Sample Representative sample Sample size Identify your random sampling procedure
58 ASSIGNMENT ASSIGNMENT #7, Draft Quantitative Survey (ComX 5337) Points: 30 pointsDATE ASSIGNED: 21 September 2012DATE DUE: 24 September, Monday, 9:00PMWhat to Submit:Draft Survey Questions for K, A, IC, BC, Stage of Behavior Questions and Media Exposure
59 Grading CriteriaPointsTied to SMART objectivesAlthough there are a lot of questions that you might be interested in asking your respondents, the goal of your questionnaire survey is to understand your impact and exposure in relation to your objectives. Ensuring that the survey question is clearly tied to your objectives will make understanding your impact transparent.10UnbiasedThe questions are not leading or pushing the respondent towards a specific answer.5Follows logical orderThe survey should be organized so that simple and non-sensitive questions come first with any sensitive questions coming last. Also, survey should test knowledge of topics before describing what those topics are. Questions should flow in a logical order.ClearIf the questions are complex or unclear, your responses will be as well! Simple questions that are not overly complicated by jargon will make responses to your survey precise and accurate.Not judgmental or emotionalOverly emotional survey questions will make respondents feel uncomfortable and may answer in a way that doesn’t accurately reflect how they feel.Total 30Penalty for late submission (-1.5)
60 Revisiting Your Research Plan Template SECTION 5.0 QUANTITATIVE DATA COLLECTION 5.1 KAP Survey 5.2 Sampling Design 5.3 Survey Planning Table 5.4 Draft Survey Questions 5.5 Draft Stage-of-Behavior Questions
61 By the end of this lesson you will be able to: Define the 6 goals of the questionnaire survey.Differentiate between a biased and an unbiased survey question.Develop survey questions from preliminary objectives.
62 Design a survey that:has appropriate sample sizes and strategymeasures the impact of your Pride campaign on SMART objectives for each target audiencemeasures exposure to your Pride campaign activities