1. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal1 The Annual Structural Business Survey Developing and Testing an Electronic Form Ger Snijkers Evrim Onat Rachel Visschers Statistics Netherlands Division of Business Statistics

2. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal2 1.Dutch Annual Structural Business Survey Annual survey of economic activity Mandatory 75.000 business each year: Sample of small firms, bigger firms each year Until now: paper questionnaire length may differ, 20 pages is typical Three parts: revenues and costs summary of business accounts: profits and losses industry specific specifications

3. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal3 2.The paper questionnaire Characteristics A4 booklet Right page: items Left page: help texts - long and voluminous Items are grouped in sections Long sections Completion process: complicated and hard Large amount of detailed informationBroad range of business information: several departments, several respondents No match with definitions used, but same label: CBS – business definitions Lay-out: misinterpratations and errors

4. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal4 Vernieuwde huisstijl: 2.Onderzoek naar de PS Q oud

5. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal5 3.The web questionnaire Project goal Develop a web questionnaire same contents mixed-mode design: paper and web support completion process: -motivate respondents to use this mode into the field: March 2006 Start of project: June 2004 In the field: Spring 2006

6. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal6 3.The web questionnaire Developing and testing In five stages: 1.Testing the prototype (31-1-05) pre-tests to test usability: 3 waves 2.Revision of questionnaire (1-9-05) expert reviews 3.Testing of revised questionnaire (1-1-06) additional usability tests 4.Implementation of field pilot (1-3-06) 5.Implementation of survey (1-3-07)

7. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal7 3.1.The web Q: The prototype

8. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal8 3.1.The prototype: 3 test waves Test 1Test 2Test 3 PeriodAug. 2004Oct. 2004Nov./Dec. 2004 FormBlaise ISBlaise - EDR Blaise - CBSquest On/off-lineOn-lineOff-line ModeBrowserCD-romDownloadable via internet Number of interviews 15 CBS employees 37 businesses 6 businesses Tested byCBS employees, external designer 6 business interviewers 2 cogn. lab interviewer + 2 business interviewer ResultsExperiences from CBS employees Experiences from interviewers Experiences with respondents

9. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal9 3.1.The prototype Research issues 1.How does the e-form work in practice? Completing the questionnaire Question-and-answer process 2.What features should be included to make it easy to use? Respondent friendly: Computer-assisted tools User wishes 3.How should the web Q be designed in relation to the paper Q? The same or a different design look-and-feel of paper and e-form

10. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal10 3.1.The prototype Research issues 1.How does the Q work? Laborious and complex process Long, complex questionnaire ( 25 items) Complex completion process: -several sessions, several informants -kick-and-rush behaviour Imagine... a respondent sitting behind his/her computer... Respondent got lost in the questionnaire

11. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal11 2.Features to make it easy to use? What am I supposed to do (next)? Easy to download, install, complete, send data back It is one process: downloading – sending data back, Clear instructions and explanations (but not read) How is the questionnaire built up? Show how the questionnaire is structured: overview Help to find the way in the questionnaire No hidden rules, no unexpected functionalities Where am I? What did I do so far? Provide overview of the completion process Clear navigation, no scrolling Printing function 3.1.The prototype Research issues

12. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal12 2.Features to make it easy to use? Cond Printing Passing on sections of Q to other departments Checking the data before transmitting Getting authorisation to release the data Calculations Well accepted, … even expected Entry-search Google-like search: Where to put these data? Navigation and overview Choose a setup people are familiar with: - setup of tax office, windows explorer 3.1.The prototype Research issues

13. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal13 3.Design of paper and web Q? The computer is different than paper The web Q reacts to the respondent Reading from the screen is different Navigating and getting an overview works differently Kick-and-rush behaviour, even stronger than on paper The use of computer-assisted functionalities; the respondent expects the computer to react 3.1.The prototype Research issues

14. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal14 3.1.The prototype Conclusions of pre-test waves Visual design Clear and logical Simple, transparent, consistent No hidden and unexpected functionalities Support the completion process Other mode, other features, other visual design Different than paper form, same look-and-feel Tailor to kick-and-rush behaviour Small sections, small tasks Short and clear explanations

15. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal15 3.2.The revised questionnaire Based on: Pre-test results Expert reviews Iterative process with - Professional designer - Questionnaire designers - Methodologists A user friendly design was put first, not the IT tool New prototypes designed in Power Point

16. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal16

17. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal17 3.3.The revised questionnaire Conclusions of add. pre-tests 10 concurrent in-depth interviews Usability and user friendliness has been improved respondents enjoyed working with the questionnaire they could handle the task even though … the task had not changed Web questionnaire design is communication design

18. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal18 3.4.Field pilot Set-up: March-July 2006 7200 businesses, 5 industries Advance letter with internet address: www.cbs.nl/productiestatistiek user name and password leaflet to introduce web questionnaire and explain why the survey is conducted paper questionnaire not mentioned

19. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal19 Leaflet

20. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal20 3.4.Field pilot Goals: Implementation of web questionnaire paper and web flows Test the whole process downloading – completing – sending-in data investigate completion process in the field: -debriefing interviews with respondents -audit trails Response rates Data quality data editing, mode effects

21. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal21 3.5.Survey Now running About 75.000 business received this questionnaire

22. Tuesday, June 19, 2007ICES3, June 18-21, 2007, Montreal22 Web questionnaire design is communication design