Presentation is loading. Please wait.

Presentation is loading. Please wait.

Module 7 Erika Trapl, PhD. Two-Part Survey Series Module 7 Matching survey items to research questions Question Selection/Development Response categories.

Similar presentations

Presentation on theme: "Module 7 Erika Trapl, PhD. Two-Part Survey Series Module 7 Matching survey items to research questions Question Selection/Development Response categories."— Presentation transcript:

1 Module 7 Erika Trapl, PhD

2 Two-Part Survey Series Module 7 Matching survey items to research questions Question Selection/Development Response categories Survey design and formatting Module 8 Taking your survey “live” Data collection protocols

3 Five Stages of Survey Development and Completion Initial survey design and preliminary planning Pre-testing Final survey design and planning Data collection Data coding, data-file construction, analysis, final report

4 Five Stages of Survey Development and Completion Initial survey design and preliminary planning Pre-testing Final survey design and planning Data collection Data coding, data-file construction, analysis, final report

5 Survey Design and Preliminary Planning What are the goals of the survey? To test a hypothesis (or a hypothetical model of interrelated hypotheses) eg, Females are more likely than males to believe that a woman should be permitted to have an abortion for any reason. To estimate the proportion of people who hold a certain belief or engage in a specific behavior eg, What proportion of our population believe that our criminal justice system is working well?

6 Initial Design of the Questionnaire What types of information do you need to collect from participants? How can you best elicit that information? Open-ended questions eg, “What is the most difficult problem facing families today?” Best for face to face rather than mailed surveys Can provide lists of choices, visual aids, participant can consult records Closed-ended questions eg, “Do you approve or disapprove of abortion?” Respondents choose from a list of provided responses

7 Final Survey Design and Planning ActivityNo. Weeks Review literature and draft questionnaire8 Assemble think alouds with 5 respondents1-2 Revise questionnaire2 Conduct pretest (n=25-30)2 Debrief interviewers and revise questionnaire3 Pretest #2 (n=20-25)2 Debrief interviewers, revise questionnaire, develop training materials 4 Select samples4 Conduct main data collection8 Code data & prepare data files10 Analyze data and write report16 Develop a Study Timeline

8 Drafting the Questionnaire Borrowing questions from other research studies is acceptable Enables comparison of your study results to previous studies Note how researchers in past studies defined concepts Note the questions they used to measure these concepts Free to use the same wording for items unless copyrighted Caveat: don’t assume that the questions are problem- free just because someone else developed and used them

9 Drafting the Questionnaire Must pre-test questionnaires even if borrowed most items from prior studies Expect to do many drafts of the questionnaire to work out all of the problems

10 Guiding Questions …or, What do you want to know? The first step: write a series of guiding questions or objectives that clearly indicate what you are trying to find out through the survey Examples as questions: How do occupational therapists incorporate play into their practice? Do occupational therapists assess play behaviors and what methods do they use? Are there differences between school-based and non-school- based settings that influence the role of play within pediatric occupational therapy?

11 Guiding Questions Hypotheses may be used to examine relationships Examples as hypotheses: Men will rate themselves higher in career success compared with women Men will report greater importance of salary and position compared with women

12 Developing Survey Questions Operationalize the Research Questions into survey questions/items Be concrete Specify exactly what information you want from respondents Consider all possibilities of interpretation and response

13 How is a question answered? For each question, the respondent must: Comprehend the question (interpret the subject and the task) Recall information, form judgment Decide on answer Record answer on a form or speak the answer to an interviewer Anticipate potential problems at each step for each question

14 Developing Survey Questions “How many college mathematics courses have you taken?” Possible interpretations: 1. Courses being taken now but not yet completed 2. College-level math courses taken in high school 3. High school level remedial math courses taken in college 4. Math courses taken by transfer students at their previous college 5. A math course taken but failed last semester 6. A course that primarily involves math but is given by another department or is not listed as a math course 7. An audited math course not taken for credit 8. A math course taken on a pass-fail option 9. College level math courses taken and completed

15 Developing Survey Questions If the choice of what to count is left to respondents, different people will include different things, introducing measurement error into the data For the math question, it may be more useful to break up the question into a number of questions that ask about the different types of courses (ie, regular, audit, pass-fail) But if and how you do this depends on your Research Question(s)!

16 Developing Survey Questions Not every possible question can be included in a survey But you should include all the questions you need to answer your Research Question Each question has a ‘cost’ and must be justified So, how do you decide what’s in and what’s out?

17 Choosing Survey Questions Decision Guide A. Does the question measure some aspect of the Research Questions? B. Does the question provide information needed in conjunction with some other variable? (If no to both A & B, drop the question. If yes to one or both, proceed). C. Will most respondents understand the question in the same way? (If no, revise or drop. If yes, proceed). D. Will most respondents have the information to answer it? (If no, drop. If yes, proceed) E. Will most respondents be willing to answer it? (If no, drop. If yes, proceed) F. Is other information needed to analyze this question? (If no proceed. If yes, proceed if the other information is readily available or can be obtained from the survey) G. Should this question be asked of all respondents or of a subset? (If all, proceed. If a subset, proceed only if the subset is identifiable beforehand or through questions in the questionnaire/interview)

18 Choosing Good Survey Questions The questions should pose tasks that the respondent can do well (ie, recall information, make judgments) In wording the question, do not influence the respondent’s answer in one direction or another Questions should be uncomplicated, explicit, and natural- sounding Avoid ambiguity Avoid double-barreled questions eg, Do you think the police and the courts are doing an excellent, good, fair, or poor job? Avoid ‘agree/disagree’ response format Acquiescence response set

19 Structure of Survey Questions Two main categories of survey questions: open questions & closed questions Responses to open questions are narratives that must be interpreted and coded by the investigators Recommend using closed questions as much as possible

20 Structure of Survey Questions Closed questions have 2 parts Statement of the question (aka ‘item’) Response categories (aka ‘options’) The question should be stated as directly as possible The answer must be given in terms of only one of the answer choices provided (typically) Qualifiers and conditions are commonly used in questions to limit the question to match the research objectives

21 Example Qualifiers How many times have you shopped at the Book Center since January? In the past year, did anything happen to you or your property that you thought was a crime or an attempt at a crime? In the past year, would you say that the violent crime situation in your neighborhood has gotten better, gotten worse, or stayed about the same?

22 Closed Ended Response Formats Responses should be exhaustive; including all possible responses Protections to assure this include ‘NA,’ ‘Don’t know,’ or ‘Other (please specify) _________’ If interested in more than one answer to a question, offer the ‘mark all that apply’ option Each response should be mutually exclusive; each choice should clearly represent a unique answer Order responses according to an inherent hierarchy if one exists (eg, increasing or decreasing intensity)

23 Response options: NOIR NOMINAL: Mutually exclusive, but not ordered, categories (gender) ORDINAL: Order matters but not the difference between values. For example, you might ask patients to express the amount of pain they are feeling on a scale of 1 to 10. A score of 7 means more pain that a score of 5, and that is more than a score of 3. But the difference between the 7 and the 5 may not be the same as that between 5 and 3. The values simply express an order. INTERVAL: Measurement where the difference between two values is meaningful. The difference between a temperature of 100 degrees and 90 degrees is the same difference as between 90 degrees and 80 degrees. RATIO: has all the properties of an interval variable, and also has a clear definition of 0.0. When the variable equals 0.0, there is none of that variable. Variables like height and weight are ratio variables. Response options have implications for data analysis and interpretation.

24 Categorical Response Formats What do you like most about your job? ___ Interaction with clients ___ Interaction with other staff ___ Intellectual challenge ___ Opportunities for professional growth Disadvantages of this type of response set Does not allow respondents to express their personal viewpoints May over look some important responses May bias answers by presenting a particular attitude Categorical - Nominal scale has implications for analysis Almost like four different yes/no items

25 Categorical Response Formats What race or ethnicity do you most identify with? ___ Asian ___ American Indian or Alaskan Native ___ Black or African American ___ White or Caucasian ___ Native Hawaiian or Pacific Islander ___ I identify with more than one ___ I don’t identify with any of these In this example, we want respondents to ‘categorize’ themselves.

26 Continuous Response Formats Characteristics on a continuum, such as attitudes or quality of performance A range of responses allows the respondent to select the choice representing the intensity of their response Example: Very important Important Somewhat important Not important Unsure Frequently used Continuous scale (ordinal, interval or ratio)

27 Common Response Categories OpinionsKnowledgeFrequency of Events Ratings Satisfied / Somewhat satisfied / Somewhat dissatisfied / Very dissatisfied Very Familiar / Somewhat familiar/ Not too familiar/ Not at all familiar Per day/Per week/Per month/Per year/Never Excellent/ Good / Fair/ Poor Important…True/FalseAlways/Frequently / seldom/never High / Medium /Low Oppose/SupportA lot/Some/ A little/ None Always/Sometimes / Never Gotten better / Gotten worse/ Stayed about the same Strongly oppose/ Oppose/ Support/ Strongly support All/Most/Some / A few / None Very fair/ Fair / Unfair / Very unfair Often/ Sometimes/ Rarely / Never

28 Measurement Scales A scale is an ordered system of questions (a set of items) that provides an overall rating representing an attitude, value, or characteristic Measures one dimension or characteristic Each item on the scale carries the same weight or point value Results in a calculated summary score Examples: Stress, Depression, Parental monitoring

29 Types of Scales Likert Semantic differential Visual analogue Guttman

30 Likert Scale Used to assess an attitude or a value Originally a 5-point response category format No consensus on the number of response categories that should be used Should there be a neutral category? Example Strongly agree Agree Neutral Disagree Strongly disagree

31 Likert Scale Development A large pool of items should be developed that reflect an equal number of favorable and unfavorable attitudes Do not include items that are intended to elicit neutral responses Scale should be validated through item analysis Those who score the highest should agree with positively worded statements, those who score the lowest should disagree

32 Semantic Differential Measures attitudes or feelings about a concept on a continuum that extends between the two extremes Typically a 7-point scale Only extremes are labeled Tends to measure evaluation, potency, and/or activity Good ____ ____ ____ ____ ____ ____ ____Bad

33 Visual Analogue Scale Measures intensity of a subjective experience A 100 mm line is typically drawn Word anchors on either end represent extremes of a characteristic Respondent places a mark along the line to indicate their perceived level for that characteristic Scored by measuring in millimeters Often treated as ratio level data No pain _______________________________ Pain as bad as it can be

34 Guttman Scale Addresses problematic issue with Likert scale different individuals with the same summary score on Likert scale may have marked different responses Guttman cumulative scale ensures that there is only one unique combination of responses that can achieve a particular score

35 Guttman Scale Example – Bogardus Social Distance Scale 1. Are you willing to permit immigrants to live in your country? Yes No 2. Are you willing to permit immigrants to live in your community? Yes No 3. Are you willing to permit immigrants to live in your neighborhood? Yes No 4. Are you willing to permit immigrants to live next door to you? Yes No 5. Would you permit your child to marry an immigrant? Yes No

36 Putting it together… You have all of your questions and responses selected—what’s next?

37 Organization of the Questionnaire Group according to categories or topics Questions should proceed from general to specific Initial questions should pique the respondents interest or at least be neutral Sensitive questions should come later Some put demographic questions at the beginning, others at the end because they are less interesting/engaging for the respondent

38 Organization of the Questionnaire Make it easy for the respondent to “navigate” the survey Use headers and simple instructions Page should be uncluttered in appearance Laser quality print Clear alignment of questions/responses Font of 11 or 12 point White or light-colored paper

39 Sources Czaja R, Blair J Designing Surveys: a guide to decisions and procedures. Thousand Oaks, CA: Sage. Portney LG, Watkins MP Foundations of Clinical Research, 2 nd Ed. Upper Saddle River, NJ: Prentice Hall.

Download ppt "Module 7 Erika Trapl, PhD. Two-Part Survey Series Module 7 Matching survey items to research questions Question Selection/Development Response categories."

Similar presentations

Ads by Google