Presentation on theme: "CBR 303: A Hands-On Approach to Developing Survey Tools in Community-Based Research."— Presentation transcript:
CBR 303: A Hands-On Approach to Developing Survey Tools in Community-Based Research
Welcome & Introductions Name & affiliation Experience with CBR and/or survey tools What you hope to learn today
Objectives Upon completion of this workshop you will be able to: Differentiate between what you need to know and what is nice to know Use various strategies for asking survey questions Apply different types of response options Recognize various approaches for administering surveys Describe how to increase response-rates Develop a strategy to manage data Identify various strategies for presenting survey results
Agenda Review of CBR Definition and purposes of a survey Identify features of best surveys Steps in the survey process Establish research objectives; identify what you need to know vs. what is nice to know Begin to develop research questions Refine research questions
What is CBR? When you hear community-based research, what words or phrases come to mind? How would you define it?
CBR definition CBR is a collaborative approach to research that equitably involves all partners in the research process and recognizes the unique strengths that each brings. CBR begins with a research topic of importance to the community with the aim of combining knowledge and action for social change to improve community health and eliminate health disparities. Minkler & Wallerstein (2003) Community-Based Participatory Research for Health Jossey-Bass (2004)
Principles of CBR CBR community initiation capacity building varied methods joint data ownership social action outcomes community relevance process oriented ethical review
Introduction to surveys Why do we do surveys? What questions can they answer? What questions cant they answer? When is doing a survey a bad idea?
What is a survey? A process for gathering information, without detailed verification, on the activity being examined. The systematic collection, analysis and interpretation of information about some area of study. Information is collected primarily by means of sampling and interviews with the selected individuals.
Survey Goals To produce detailed data that will allow researchers to:| describe the characteristics of the group under study, test theories about the group, and generalize results beyond just those people who responded to the survey
Purpose for Surveys Understand the specific activity under review Identify significant areas warranting special emphasis Obtain information for use in making organizational changes Identify future changes/goals for the organization
Features of the BEST Surveys * Specific objectives Straightforward questions Sound research design Sound choice of population or sample Reliable & valid instruments Appropriate analysis Accurate reporting of survey results Reasonable resources * From The Survey Handbook, Arlene Fink, 1995
Survey process Assemble your team Establish your research objectives and questions Determine your sample Choose interview methodology Design questionnaire Pre-test the questionnaire Administer survey Collect and enter data Analyze data Present data
1. Assemble your team Statistician +/ Social Scientist +/Epidemiologist Community members +/ Community leaders +/ Service providers Remember to consult your team early and often during planning, implementation analysis and dissemination.
2. Establish Research Objectives & Questions What do you want to know? What do you need to know? What will you measure? Why this factor? How will the information be used? Is a survey the best way to get the information you need? What kinds of resources (time, staff, money) do you have in place to make this happen? Try not to mix too many objectives into a single survey. Keep it focused.
Example – Research Objective To identify the factors that lead to early school withdrawal for lower income youth in the community
Dont Reinvent the Wheel Before you develop your own tools see what other people have done. Is it worth redoing a study that has been done 100 times? What makes your research new? How do others ask similar types of questions? (check the literature) Can you use or adapt a pre-existing instrument to meet your needs?
Standardized vs. Homemade Instruments StandardizedHomemade can compare your resultshard to compare less costlymore costly may not be totally appropriatecan tailor to your needs, may be more relevant higher credibilitycredibility depends of investigators may not exist!always creatable!
Next … Hypothesis X Y Lower Socio-economic Status Early School Withdrawal
Operationalizing Variables Lower Socio-economic Status Actual Income Occupation Work Shift/Conditions Time Spent with children Ability to provide educational supports Early School Withdrawal Withdrawal from school prior to completion of Grade 12 Marketable skills Relationship with school staff Relationship with peers Social attitudes
From Concept to Questions Example: Socio-economic Status ConstructSocio-economic status IndicatorIncome level Operationalized Variables What was your total family income during the past 12 months? Measurementa)Under $10,000 b)$10,001-$25,000 c) $25, $50,000 d) $50, ,000 e) more than $100,001
From Concept to Questions Example: Socio-economic Status ConstructSocio-economic status IndicatorsEducational attainment Operationalized Variables What is the highest level of school you have completed? Measurementa)Primary School b)Some High School c) Completed High School d) Some College e)Completed College f)Graduate/Professional Degree
Exercise: From concept to questions On your Survey Development Template: 1.Identify the hypothesis (e.g. X causes Y) underlying your objective. 2.List the related factors for each variable. 3.Develop a related research question, operationalizing your variables. 4.Develop a specific measurement for the question.
Important Tips! Each question on your survey needs to be JUSTIFIED and LINKED to your research objectives Obtain ALL of the ESSENTIAL information Obtain ONLY the ESSENTIAL information
Example: Possible question topics Age Sex Sexuality Experiences in school Level of academic completion Experiences of success SES School Neighborhood What their parents do Extracurricular activities Hobbies Relationships with teachers Relationship with peers Relationship with parents Part-time employment
Triaging Questions High: Absolutely essential Medium: Very valuable for decision making Low: Supportive data to enhance understanding
Strategies for asking questions Different Types of Questions
Open-Ended The respondent answers in their own words More demanding and time consuming for respondents Lower response rate Harder to categorize, analyze and interpret Good when you dont know how people will respond, useful in the preliminary pilot stage E.g. Please describe any barriers that you face accessing our service?
Closed/Structured The respondent answers from a range of choices Less time consuming for respondents Higher response rate Easier to categorize, analyze and interpret Requires more front-end effort (designing the questions) Can oversimplify or make assumptions Response categories must be inclusive Limits answers to options presented Can compromise with other:____________ category
Types of questions: Two choices Do you like ice cream? Yes No
Types of questions: Multiple choice Which is your favourite ice cream flavour? Chocolate Vanilla Strawberry Banana Other: __________ (please specify)
Types of questions: Checklist Please select all of the ice cream flavours that you have eaten in the last month? Chocolate Vanilla Strawberry Banana Other: __________ (please specify)
Types of questions: Ranking Please order the following ice cream flavours by preference (where 1=your favourite & 4 = your least favourite)? Chocolate Vanilla Strawberry Banana
Types of questions: Rating/Likert Scale Please rate the degree to which you agree/disagree with the following statements: 1. Ice cream is my favourite dessert. Strongly agree Agree DisagreeStrongly Disagree 2. Ice cream's only redeeming feature is that it can have chocolate. Strongly agree Agree Disagree Strongly Disagree
Helpful options Excellent – good – fair – poor Approve/Disapprove, Favour/Oppose, Too many – about right – not enough Often – Regularly – Seldom Always – most of the time- some of the time- rarely – never More likely – no difference – less likely (Agree) (Disagree)
Evens or Odds Even numbered scales force a choice Odd numbered scales allow for a neutral stance Beware of the forced choice: research shows that those who would have gone for neutral are more likely to go for positive choices when forced! Beware of neutral categories: research shows that people revert to the mean (ie they are most likely to choose neutral).
Types of questions: Numerical How many ice cream cones have you eaten in the last week? _____ How old are you? ______ What year were you born?_____
Tips for Question Design Dont ask questions that you would not answer Focus on behaviours not labels Keep your values/assumptions out of questions Make sure that questions are not too demanding Avoid abbreviation or professional jargon Use simple, direct and familiar words/terms (slang can be OK)
Tips for question design contd … Make questions clear, specific and as short as possible Make sure everyone will interpret the question the same way Specify the frame of reference (e.g. time, place) Avoid leading questions Be aware of social desirability Make sure that questions are applicable to all respondents
Ordering your questions Always give an intro (title, sponsor, purpose etc) Start with easy questions that everyone can answer (builds trust & confidence) Introduce sensitive stuff gradually
Ordering your questions There should be a logic to sequence and flow, use transition statements or subheadings Move from general to specific Ask about things in chronological order Always say thank you at the end and leave room for questions/comments/feedback
Common Survey Errors! Beware!
Problem questions: The double barrel Beware of the ANDs and Do you like to watch TV and eat potato chips? How would you rewrite?
Problem Questions: The Incompetent Experts Sometimes people dont know enough to answer the questions or may not be able to estimate the answers To what extent do you agree with the economic foreign policy of Latvia? How many cans of soup have you bought in the last year?
Problem questions: The long list Sometimes having questions with too many response categories can be overwhelming, especially when you are asking folks to rank order them: 1 17 Please rank order these TV shows (where 1 is your favourite and 17 is your least favourite)…
Problem questions: The double negative I disagree that the statement is false…
Problem questions: Not mutually exclusive How old are you?
Problem questions: Vague wording Do you have a partner? Yes No
Problem questions: The mismatch Do you work out? Strongly agree Agree Disagree Strongly disagree
The Vague Response When is your birthday? __________ Answer: tomorrow
Sex, drugs and all the good stuff How do we Ask about Sensitive Topics?
Asking sensitive questions Dont ask what you would not answer! Consider social desirability bias Some strategies for asking sensitive questions include: Self-administered questionnaires Ensuring anonymity or confidentiality Careful wording Providing appropriate response ranges
Ways of Asking Threatening Questions Various techniques have been developed to ask embarrassing or sensitive questions in non-threatening ways. The following are examples of several such techniques as applied to the question: "Did you kill your boss?"
Ways to ask: Did you kill your boss? The Casual Approach "Did you happen to have killed your boss? The Numbered Card Please read the number which corresponds with what happened to your boss. 1. Natural death 2. I killed him/her 3. Other (please specify)
Did you kill your boss? The Everybodys Doing It Approach "As you may be aware, many people have been killing their bosses these days. Would you happen to have killed yours?" The Sandwich Method "Have you ever done any of the following?" a. Fire-bombed the Parliament buildings b. Murdered your boss c. Detonated a nuclear device
Did you kill your boss? The Assumption Method "How many of your bosses have you killed in the last ten years of work? The Other People Approach a. "Do you know of other people who have killed their bosses?" b. Have you done that yourself?
Did you kill your boss? The Illustrative Technique Which of the following pictures most accurately depicts what happened to your boss?"
Designing a good survey is hard work Give Yourself Plenty of Time!
Pilot Testing TEST! TEST! TEST! TEST! TEST! Test with colleagues & participants – the more the better. This will save you time in the long run! Discover Poor question wording or ordering or instructions Suggest additional response categories Time issues
Getting your questions answered…. Who are you going to talk to?
3. Determine Your Sample Who will you interview? Who is your clientele? Will this group provide you with the necessary data to meet the goals of the project? Keep the sample large enough for a sufficient amount of data but small enough so that it is manageable.
Sampling Eligibility or inclusion/exclusion criteria want Who do you want to talk to? not want Who do you not want to hear from?
Inclusion & Exclusion Research Question How does socio-economic factors impact on students ability to remain in school? Target Population High School Students Inclusion Criteria years old Students in school & Early school leavers Parental and youth consent Exclusion Criteria Use alcohol, marijuana or other drugs more than once a week
Sampling Probability Sampling (ideal) Simple random sampling Stratified random sampling Systematic random sampling Cluster random sampling Non-random sampling (hard-to-reach, pilot) Whole group Convenience Snowball Quota
Sample Size Best way to determine this is to talk to a statistician! Considerations include: Cost Effect Size Response rate
Getting your questions answered…. How are you going to do it?
4. Choose an Interview Methodology How will you conduct the interview? mail out Questionnaires individual interviews case study focus group on-line surveys language; accessibility options; age; education Consider the make up of your clientele: language; accessibility options; age; education
Types: Face to Face ( door to door ) BENEFITS Interviewer administers survey Ensures consistency & complete response Allows for clarification & probing Good if there are lots of skips High response rate Takes care of literacy issues CONSIDERATIONS Expensive Confidentiality & privacy concerns Interviewer bias Social desirability bias may be stronger
Types: Telephone BENEFITS Interviewer administers survey Ensures consistency & complete response Allows for clarification & probing Good if there are lots of skips Less expensive than face to face CONSIDERATIONS Less response rate than face to face Miss those without a telephone Hard in this climate of telemarketers – must be SHORT!
Types: Self-completed (mail, web) BENEFITS Survey completed by respondent Low-cost Can be good if there is lots of confidential or embarrassing questions Increase anonymity of responses CONSIDERATIONS Requires literacy Variable completeness of answers Bad if there are lots of skips or confusing parts Lower response rate
Factors to Consider when Choosing a Method Cost, budget Human resources Equipment Expertise Who is likely to be reached/not reached What is your target population most likely to respond to? Bias Response rate Participant burden Follow-up necessary? Timelines Sample size
Getting your questions answered…. But nobody wants to talk to me?
Improving Response Rate Keep things short & simple Ensure confidentiality Target the appropriate population Make it easy/convenient Clearly communicate purpose, importance & relevance Give advance notice If you can – provide rewards or incentives
Increasing response rate: Layout matters! Create a booklet Print on white paper, dont look like an ad No questions on 1 st page – just title No questions on last page – room for comments Order fun/easy questions 1 st, harder in the middle and demographics at the end The first question matters: it should be easily answerable! The layout should be neat and clean and easy to follow From Dillman handout
Getting your questions answered…. What do I with all these surveys?
8. Manage Your Data Expect to collect data 2-3 weeks after distributing Stay organized Remember confidentiality Determine ahead of time (before or while survey is being developed) data management program (e.g. NVIVO or NuDist for qualitative; SPSS, SAS, Stata for quantative) Determine ahead of time who will enter data; provide them with adequate training Talk to your statistician!
9. Analyze Data Identify themes & patterns Frequencies (counts of things, how many times things happen; get mean, median, mode) Descriptives (percentages or raw numbers of demographics and other factors that would be nonsensical to take an average such as gender or ethnicity) Outliers (things that fall outside the expected values)
My questions are answered…. Presenting Results Numbers can be deceiving…
10. Report Results Make sure to acknowledge both the strengths & limitations of your data beginning with strengths! ALWAYS CONTEXTUALIZE YOUR DATA How many people did you talk to? How did you decide who to talk to? (random, convenience) Who did you talk to? How easy is it to generalize your results?
Presentation Matters Tables Graphs Pictures Words
best Features of the best Surveys Specific objectives Straightforward questions Sound research design Sound choice of population or sample Reliable & valid instruments Appropriate analysis Accurate reporting of survey results Reasonable resources From The Survey Handbook, Arlene Fink, 1995
Reflection & Next Steps Reflect on and record 1-3 things you learned today. Record 1-3 related goals you have going forward. Record 1-3 related next steps you plan to take upon leaving the workshop.
Objectives Having completed this workshop you should now be able to: Differentiate between what you need to know and what is nice to know Use various strategies for asking survey questions Apply different types of response options Recognize various approaches for administering surveys Describe how to increase response-rates Develop a strategy to manage data Identify various strategies for presenting survey results
Acknowledgements Several of the slides presented here today were developed by San Patten, MSc., Community-Based Research Coordinator for the Alberta Community Council on HIV
CBR 303: A Hands-On Approach to Developing Survey Tools in Community-Based Research