1. Response option order effects. Scale lengths and horizontal or vertical layout. Johan Martinsson University of Gothenburg www.lore.gu.se

2. Overview 2 related but separate studies The general focus on effects of different ways to arrange response options on a screen (eg. ”primacy effects”, tendency to pick the first response option) Three factors to consider: scale length layout direction (horizontal or vertical) response device (PC, tablet, smartphone)

3. www.mod.gu.se Why do we care? You want a neutral measurement instrument An arbitrary design choice on your part should not influence results An arbitrary choice of device of respondents should not influence results => robust estimates

4. www.mod.gu.se Some background More and more respondents use smartphone to answer surveys (though clearly a minority) Smartphone compatibility  no classic grids  shorter scales  vertical layout Screen sizes differ

5. www.mod.gu.se Some background More automatic adjustment to device by survey softwares (responsive design)  Different respondents get different design  Robustness of measurement instruments more important to examine

6. www.mod.gu.se Examples

7. Reversing response option orders

8. Horizontal or vertical layout

9. Example of automatic mobile adjustment by survey software

10. www.mod.gu.se Study 1 Data from an opt-in sample from a survey connected to a voting advice application for the Swedish election 2014 Field work: July to September 2014 Total n approximately 20,000 Randomized experimental design 2x2x2: ”Right direction” or ”wrong direction” first 7 or 3 scale points Horizontal or vertical layout

11. www.mod.gu.se Study 1 We had a few hypotheses... :  primacy effects exist  stronger effects with vertical layouts than horizontal  stronger effects for smartphone users  (especially with horizontal layouts)

12. www.mod.gu.se Results All respondents 7-point scale with horizontal layout Placement FirstLastdiff Right direction11101 Wrong direction n25612538

13. www.mod.gu.se Results All respondents 7-point scale with horizontal layout Placement FirstLastdiff Right direction11101 Wrong direction1921-2 n25612538

14. www.mod.gu.se all respondents 7-point scale with vertical layout Placement FirstLastdiff Right direction15 Wrong direction26 n24572455

15. www.mod.gu.se all respondents 7-point scale with vertical layout Placement FirstLastdiff Right direction15114 Wrong direction26215 n24572455

16. www.mod.gu.se all respondents 3-point scale with horizontal layout Placement FirstLastdiff Right direction31283 Wrong direction49454 n24212492

17. www.mod.gu.se all respondents 3-point scale with vertical layout Placement FirstLastdiff Right direction32284 Wrong direction48471 n25662550

18. www.mod.gu.se Kind of boring results, no? Luckily, we discovered something more interesting by mistake!

19. www.mod.gu.se 7-point scale with horizontal layout Placement FirstLast Right direction1110 Wrong direction1921 n25612538 7-point scale with vertical layout Placement FirstLast Right direction1511 Wrong direction2621 n24572455

20. www.mod.gu.se 3-point scale with vertical layout Placement FirstLast Right direction3228 Wrong direction4847 n25662550 3-point scale with horizontal layout Placement FirstLast Right direction3128 Wrong direction4945 n24212492

21. www.mod.gu.se Study 2 Data from an opt-in sample from a survey connected to a voting advice application for the Swedish election 2014 Field work: July to September 2014 Total n approximately 34,000 Randomized experimental design 5x2(x2): Horizontal or vertical layout of the questions 10, 7, 5, 3, or 2 scale points (response options)

22. www.mod.gu.se Percent "Wrong direction" among everyone HorizontalVerticalDifference 10-point scale 7-point scale 5-point scale 3-point scale 2-point scale Results

23. www.mod.gu.se Percent "Wrong direction" among everyone HorizontalVerticalDifference 10-point scale1621-5 7-point scale2027-7 5-point scale2431-7 3-point scale 2-point scale Results

24. www.mod.gu.se Percent "Wrong direction" among everyone HorizontalVerticalDifference 10-point scale1621-5 7-point scale2027-7 5-point scale2431-7 3-point scale49 0 2-point scale5859 Results

25. www.mod.gu.se Percent "Wrong direction" among PC users HorizontalVerticalDifference 10-point scale1820-2 7-point scale2127-6 5-point scale2731-4 3-point scale50 0 2-point scale60 0

26. www.mod.gu.se Percent "Wrong direction" among smartphone users HorizontalVerticalDifference 10-point scale1425-11 7-point scale1929-10 5-point scale2133-12 3-point scale 2-point scale

27. www.mod.gu.se Percent "Wrong direction" among smartphone users HorizontalVerticalDifference 10-point scale1425-11 7-point scale1929-10 5-point scale2133-12 3-point scale5051 2-point scale5961-2

28. www.mod.gu.se Difference depending on device AllPCSmartphone 10-point scale 7-point scale 5-point scale 3-point scale 2-point scale

29. www.mod.gu.se Difference depending on device AllPCSmartphone 10-point scale-5-2-11 7-point scale-7-6-10 5-point scale-7-4-12 3-point scale00 2-point scale0-2

30. www.mod.gu.se Conclusions and recommendations We observe weak but fairly robust primacy effects (as usual) These do not seem to depend on device or scale length much (more surprisingly)

31. www.mod.gu.se First, some useless advice... Ban smartphones in your surveys! Only use horizontal scales (not even sure it is better)

32. www.mod.gu.se Some less useless advice Use really short scales Randomize when you can (nominal scales) Split half reversed order? Use long questions wordings and make sure respondents are aware of all available options before allowing them to answer (but increases burden and time)

33. www.mod.gu.se Read more at www.lore.gu.se Laboratory of Opinion Research (LORE) www.lore.gu.se