1. Survey Construction A meta-analysis of item wording and response options to reduce Bias in survey research Prepared by: Amanda Mulcahy Loyola University School of Education Department of Research Methodologies

2. Research Focus A meta-analysis of item wording and response option studies Examining threats to reliability and validity

3. Survey Item Wording Use of positive and negative item stems –Acquiescence –Assumptions –Construction

4. Negative Item Types Polar opposites –“I am happy” & “I am sad” Negative polar opposites –“I am not sad” Negated items –“I am not happy”

5. Impact of Negative Items Placement of negative items –Impact on ratings of objects/people –Accuracy Internal Consistency –Factor structures Scoring of negative items –Reverse scoring

6. Impact of Age and Education Age –Younger people have a harder time with negative items –Negation and double negatives require higher levels of verbal reasoning Education –Those with lower education levels less able to distinguish negative item meanings

7. Survey Response Formats Scales –Agree/disagree formats Likert-type True/false –Verbal scales –Numeric scales

8. Response Order and Position “Strongly Agree” “Strongly Disagree” or “Strongly Disagree” “Strongly Agree” Inclusion of midpoints Use of neutral categories Odd or even number of response options

9. Response Style Also known as response bias or an undifferentiated response set Individuals select from a particular response category disproportionately independent of item content Create, mask, or exaggerate relationships

10. Types of Response Styles Acquiescence –Agree or disagree regardless of item content Primacy Effect –Selecting items to the left side or top of a response scale Recency Effect –Selection of responses to the right or bottom of a response list

11. Meta-Analysis “Meta-analysis” (Gene Glass, 1976) Is not a re-analysis or secondary review An analysis of analysis Effect size – mean difference in the outcome variable between treated and untreated subjects divided by the within-group standard deviation

12. Meta-Analysis Research Questions – Item Wording What is the effect of using negative wording on the mean score of an item on an attitude survey? Does exposure to negatively worded items influence survey instrument scores? Do older studies on the topic show different results than more recent studies?

13. Item Wording Procedure 40 Studies were identified, spanning 1927 – 2001 Studies were coded 11 studies were identified for the item wording analysis Microsoft Access and SPSS were used for data analysis Effect sizes were calculated using Glass’s formulas with Hedge’s corrections for small sample size bias

14. Forest plot of the 11 item wording studies

15. Effect Size Mean effect size for the 11 studies: -0.39144 Using negative wording on items causes a drop in the mean scores of those items when compared to their positively worded versions on the attitude instruments

16. Models Examined Homogeneity Tests Fixed Effects Model Random Effects Model Regression Model Exclusion of Outliers

17. Preliminary Conclusions Negative wording causes a drop in the mean scores of those items on attitude surveys There are compounding factors –Year of publication –Control group exposure to negative items

18. Future Analyses Item wording research will continue Response order comparing ascending and descending orders will be examined To develop a best practice for survey construction to improve education research and its findings Goal