Survey Construction A meta-analysis of item wording and response options to reduce Bias in survey research Prepared by: Amanda Mulcahy Loyola University School of Education Department of Research Methodologies

Research Focus A meta-analysis of item wording and response option studies Examining threats to reliability and validity

Survey Item Wording Use of positive and negative item stems –Acquiescence –Assumptions –Construction

Negative Item Types Polar opposites –“I am happy” & “I am sad” Negative polar opposites –“I am not sad” Negated items –“I am not happy”

Impact of Negative Items Placement of negative items –Impact on ratings of objects/people –Accuracy Internal Consistency –Factor structures Scoring of negative items –Reverse scoring

Impact of Age and Education Age –Younger people have a harder time with negative items –Negation and double negatives require higher levels of verbal reasoning Education –Those with lower education levels less able to distinguish negative item meanings

Survey Response Formats Scales –Agree/disagree formats Likert-type True/false –Verbal scales –Numeric scales

Response Order and Position “Strongly Agree” “Strongly Disagree” or “Strongly Disagree” “Strongly Agree” Inclusion of midpoints Use of neutral categories Odd or even number of response options

Response Style Also known as response bias or an undifferentiated response set Individuals select from a particular response category disproportionately independent of item content Create, mask, or exaggerate relationships

Types of Response Styles Acquiescence –Agree or disagree regardless of item content Primacy Effect –Selecting items to the left side or top of a response scale Recency Effect –Selection of responses to the right or bottom of a response list

Meta-Analysis “Meta-analysis” (Gene Glass, 1976) Is not a re-analysis or secondary review An analysis of analysis Effect size – mean difference in the outcome variable between treated and untreated subjects divided by the within-group standard deviation

Meta-Analysis Research Questions – Item Wording What is the effect of using negative wording on the mean score of an item on an attitude survey? Does exposure to negatively worded items influence survey instrument scores? Do older studies on the topic show different results than more recent studies?

Item Wording Procedure 40 Studies were identified, spanning 1927 – 2001 Studies were coded 11 studies were identified for the item wording analysis Microsoft Access and SPSS were used for data analysis Effect sizes were calculated using Glass’s formulas with Hedge’s corrections for small sample size bias

Forest plot of the 11 item wording studies

Effect Size Mean effect size for the 11 studies: Using negative wording on items causes a drop in the mean scores of those items when compared to their positively worded versions on the attitude instruments

Models Examined Homogeneity Tests Fixed Effects Model Random Effects Model Regression Model Exclusion of Outliers

Preliminary Conclusions Negative wording causes a drop in the mean scores of those items on attitude surveys There are compounding factors –Year of publication –Control group exposure to negative items

Future Analyses Item wording research will continue Response order comparing ascending and descending orders will be examined To develop a best practice for survey construction to improve education research and its findings Goal