Questionnaire Construction January 28 & 31, 2011

2 Objectives By the end of this meeting, participants should be able to: Distinguish open and closed-ended questions. Distinguish open and closed-ended questions. Evaluate questions for their validity and reliability, as well as how well they follow basic rules of question wording. Evaluate questions for their validity and reliability, as well as how well they follow basic rules of question wording.

3 Florida Datamar Democratic Polling Likely Democratic Voters Clinton40% Obama28% Edwards19% Kucinich2% Richardson2%

4 The Vital Information- FL a) a)Sample size 541 +/- 4.2 percent sampling error b) b)January 5 – 7, 2008 c) c) Methodology: Findings are from a January 5 – 7, 2008 survey using a comprehensive predictive model of “ likely ” voters, based on election cycles and other factors of voters in Florida. The survey was conducted using an automated telephone dialer and the voice of a professional announcer. d) d)The targeted-registration-based-sampling (TRBS)* selection criteria were based on election cycles and other voter factors in Florida. Datamar proprietary algorithms were used to generate random samples from the target group for calling. e) e)Where necessary, responses were weighted according to age, gender, and political party.

5 The Vital Information- FL Q18. If the primary election were held today, for whom would you vote? The candidates are: Joe Biden, U.S. Senator from Delaware; Chris Dodd, U.S. Senator from Connecticut; Bill Richardson, Governor of New Mexico; Mike Gravel, former U.S. Senator from Virginia; Dennis Kucinich, Congressman from Ohio; John Edwards, former U.S. Senator from North Carolina; Barack Obama, U.S. Senator from Illinois; Hillary Clinton, U.S. Senator from New York.

6 Research 2000 FL-D Democratic Primary Voters Clinton 50% Obama28% Edwards13% Others4% Undecided5%

7 Vital Information- Florida Research 2000 The Research 2000 Florida poll was conducted from January 14 th through January 16 th, There were a total of 500 likely Democratic and 500 Republican primary voters as well as 500 statewide primary voters interviewed by telephone. The margin of error is 4.5%. If the Democratic Primary for President were held today, which of the following candidates would you vote for? (Rotated)

8 Types of Questions a)Open-ended questions What do you like about the Democratic Party? What do you like about the Democratic Party? What do you like about the Republican Party? What do you like about the Republican Party? b)Closed-ended questions Multiple choice questions Multiple choice questions Rating scale/ Feeling thermometer Rating scale/ Feeling thermometer c)Disadvantages/ Advantages of Each Approach

9 Rating Scales Common political types a)Rating scales (generally seven or six points from very positive to very negative) b)Thermometers (generally 1 to 100) c)Semantic differential (very good-very bad) d)Seven point scale (two concepts at each end)

10 Good Standard Practice a)Use common questions b)Use short questions with standard speech c)Think of questions as a stimulus that the respondent will react to d)Use a battery of questions for more difficult concepts e)Tailor question wording to audience

11 Things to Consider a)Branching format (particularly for scales) b)Response choice order c)Ambiguous wording d)Biased questions e)Social desirability bias f)Double barreled questions g)Double negatives h)No opinion/ don ’ t know

12 Evaluating Questions a)Reliability b)Validity Face validity Face validity Convergent validity (similar answers) Convergent validity (similar answers) Divergent validity (different answers) Divergent validity (different answers) Criterion validity (direct measure) Criterion validity (direct measure) Content validity (all parts of a concept) Content validity (all parts of a concept) Construct validity (consistent with previous work) Construct validity (consistent with previous work)

13 Constructing a Questionnaire a)Topic Order b)Question order/ Consistency bias c)Response choices/ Response set d)Number of questions half samples half samples rotating questions rotating questions e)Questionnaire layout f) Pretesting the questionnaire cognitive think-aloud cognitive think-aloud

14 Example: Sensitive Questions a) a)Suppose UGA wanted to survey student behavior on campus. b) b)Specifically, the administration wants an accurate picture of what percent of students partake in unscrupulous activities. c) c)How can the university ensure it gets accurate results?

15 Example: Sensitive Questions Innocuous Question Technique a) a)Flip a coin. If you get heads, answer question 1. If you get tails, answer question 2. Do not tell me what you get or what question you answer. 1) I have had unprotected sex during the last month: Yes / No 2) 2) I got tails (on the coin flip): Yes / No

16 Example: Sensitive Questions a) a)Proportion of “ Yes ” responses =(Yes to question 1) + (Yes to question2)/Total # of respondents = b) b)Yes to question 1/Total # of respondents + Yes to question2/ Total # of respondents= c) c)θ P + (1 − θ ) δ, where θ =probability of answering question 1 P= proportion who had unprotected sex in the past month (1 − θ )= probability of responding to question 2 δ = the probability of getting tails (yes: Q.2)

17 Random Half Sample a) a)Divide survey sample in half (randomly). b) b)Half of respondents get column 1 the other half get column 2.

For January 31 a)Read WKB chapter 4. b)Answer the following questions: a)Imagine we conducted a survey of 350 UGA students and found that 93% thought the Bulldogs would win the SEC championship. What is the margin of error (at 95%)? Based on that margin of error, we can be 95% sure that championship predictions lie in what range? b)Imagine we conducted a survey of GA voters and found that Obama had approval of 55%. 750 people were surveyed, what is the margin of error (at 95%)? Based on that margin of error, we can be 95% sure that Obama’s support in GA lies in what range?

19 For February 2 a)Read WKB chapter 5 b)Answer questions 1, 2, & 3 on page 130.