Eurostat Questionnaire design

Presented by Paweł Lańduch Central Statistical Office, Statistical Office in Poznań, Poland.

Questionnaire design Modules Main theme, Electronic questionnaire design, Editing during data collection, Testing the questionnaire.

Questionnaire design Business surveys – special population –the response process is complex and burdensome, since preparing the required data entails a mixture of organizational and individual tasks. Data are mostly quantitative and their acquisition requires access to business records; –the predominant role of self-administered questionnaires and the role of the respondent. In the case of establishments, the respondent is usually a representative, who acts as a data provider in an organizational environment,

Questionnaire design Business surveys – special population –the mandatory character of reporting as opposed to mostly voluntarily aspect of household surveys, –concepts and definitions are technical, complex, and often based on legal or regulatory considerations. Practices, terminology and standards used by businesses in their daily operations, the accounting standards, for example, are the context that needs to be taken into account. This calls for detailed instructions that accompany questions.

Questionnaire design Business surveys – special population –the distribution is skewed in favor of larger establishments, –timeliness is often given priority over quality. –the longitudinal character of surveys and overall reluctance to changes in measurement instruments used.

Questionnaire design Response process Definition - the result of the interaction between a respondent and a questionnaire (Edwards & Cantor, 1991, Measurement Error in Surveys) Houshold surveys versus business surveys individual organisation using memory using records cognitioncognition + organisation retrieval retrieval + records look up

Concepts and variables (example) Short term statistics –Term – Industrial production sale value in current prices (Sa) –Variables  Revenues from products sale (Sw)  Value of goods production not including in sales (Sz)  Excise imposed on products (Da)  Subsidy for products (recevied for products)(Dp) Sa=Sw+Sz-Da+Dp Questionnaire design

Choosing the mode of data collection -Interviewer administered -Self-administered -Mixing modes -Impact of computer technology – web forms -Actual burden versus perceived burden -Numerical character of data -One of the most burdensome phase – data retrieval Questionnaire design

Building the prototype questionnaire –Content and non-content elements  Introductory letters,  Cognitive aspects  Layout,  navigation  instructions  technological challanges –Heavy reliance on instructions –Guidelines (examples: Economic Directorate Guidelines on Questionnaire Design. Washington, DC: U.S. Census Bureau (2008). Questionnaire design

–Guidelines cont. examples: Forms Design Standards Manual, Australian Bureau of Statistics (2010), –Handbook of Recommended Practices for Questionnaire Development and Testing in the European Statistical System, Eurostat (2006) -Creating standards as a framework for developing, testing and evaluation, also for technical aspects -Example (Web environment – hollistic view, dynamics, one point for collection, one-size-fits-all ?) -Measurement errors-Quality Cost-Response Burden-User satisfaction Questionnaire design

Impact on response process Impact on response burden Impact on quality New features and opportunities but also challages Electronic versus paper Layout – paging, scrolling, should the electronic versions be smiliar in appearance to those on paper ? Editing – moving closer to the respondent, cost reduction, reducing burden, seeking better quality Questionnaire design – Electronic questionnaire

Automatic routing – eliminate routing errors, previous answers determine next questions presented, Calculations – expecting by respondents, are elements of data consistency and validity Progress indicator – possibility of checking the completion status, there are also arguments against placing it Navigation – seeking easy way for moving forward and backward, possibility of interutption and resuming Instructions – close at hands, examples: hyperlinks and pop-ups Security – important for respondents espacially in busines surveys Questionnaire design – Electronic questionnaire cont.

Testing for usability - Gestalt psychology for visual design - Cognitive psychology - designer’s perspective versus user’s perspective - user-centre design - designing for errors - visibility, mapping and feedback - standardizing - heuristic approach (Jacob Nielsen) Questionnaire design – Electronic questionnaire cont.

Example: paper questionaire and its electronic counterpart Questionnaire design – Electronic questionnaire cont.

Microsoft Office as an example of standardizing a toolbar in a suite of programs Questionnaire design – Electronic questionnaire cont.

The effect of automatic routing (the gap in numbering)

-Possibility to move some of editing activities (so far as a post- collection process) on collection stage. -Reduction of costs and data quality improvement -Potentials for lowering burden -Build-in editing rules called edit checks or edits – a respondent is notified about errors while entering responses -Types of implemented edit rules – hard edits and soft edits There is an expectation collected data are of high quality and contain no errors – THAT MAY NOT BE TRUE Questionnaire design – Editing during data collection

Presenting messages to respondents (examples from CSO of Poland) Questionnaire design – Editing during data collection Using icons in messages Marking hard edits failures (errors) Soft edits (warnings)

Text of message as a pop-up box or in a list (examples) Questionnaire design – Editing during data collection Red text: Error: The information you provided does not match our criteria.

Various types of edits for various types of items: -Text – length, format check -Numbers – range, algebraical relations, balance check -a reponse choosing from a predefined list -Required item check -A rule allowing only some sort of characters e.g. numeric -Checking the relationship between two or more items Questionnaire design – Editing during data collection

Considerations when choosing a strategy for edit rules in questionnaires -How many edit rules will be effective and not resulting in item non-response ? -Conservative aproach versus extensive usage of edit rules -Which way of preseting messages is efficent to be recognized by respondents ? -Even if the messages are perceived they should be understood Questionnaire design – Editing during data collection

Considerations when choosing a strategy for edit rules in questionnaires (cont.) -How data with unresolved edits should be treated ? -How to test and evaluate edits in the questionnaire ? -Should be permitted to submit data with unresolved edits ? Some hints from usability testing: –user should be at the center of design, the task should be under the user’s control –Words „error” and „mistake should be avoided Questionnaire design – Editing during data collection

Some hints from usability testing (cont.): –Edit rules should be triggered immidiately as they can, but with a possibilty of a repetead execution, – The message should be provided with attributes for easy localization of the problematic item – Clear signification to the respondent the obligation to resolve all edits or to provide a comment before submission – Deliberation before providing data from earlier periods to avoid acquiescence bias Questionnaire design – Editing during data collection

–European Statistics Code of Practice, Principle 8, Indicator 8.2: In the case of statistical surveys, questionnaires are systematically tested prior to the data collection. The continuous process – A draft questionnaire review pretesting revision – Data collection evaluation revision pretesting –pretesting / field testing Questionnaire design – Testing the questionnaire

-CASM Cognitive aspects of survey methodology - „the subject” thought process of response Think aloudVerbal probing Social surveys Business surveys at business location presurvey visits expansion of cognitive interviewing

Record keeping studies -One of the most burdensome step in the response process – retrieval of the demanded data -Data may not exist in business records -Data may be kept in different places -Cognitive background -Different persons with various degree of access to data Questionnaire design – Testing the questionnaire

Problems with data retrieval estimation rough estimation blunder non-response Usability testing Visual aspect Fonts Graphics Visual clutter User interface Navigation Skipping routes Questionnaire design – Testing the questionnaire

- Usability testing (cont.) Software the process (planning, analysis, implementation, assesement, closure) functional tests (black box) structural tests (white box) - Testing in the field (after data have been collected) Pilot tests – dress rehearsal Debriefing respondents – formal and informal - the source of information about the questionnaire Debriefing survey stuff – formal and informal – as intermediaries between respondents and survey agency have valuable knowledge for the questionnaire evaulation Questionnaire design – Testing the questionnaire

Testing in the field (after data have been collected) Post-collection data evaluation – a tool for assessing data quality, –Measuring non-response –Detecting outliers –Data editing failures –Comparing with data from other sources Questionnaire design – Testing the questionnaire

Qualitative methods Pretesting and cognitive pretesting can lead to changes and improvement Quantitative methods Seeking measures how changes have worked

Setting standards for testing Minimal standards Extended standards Best practices Response process as a tool for the questionnaire evaluation Based on steps of the response process in business surveys Based on a topic list of questions Qualitative method for improvements Questionnaire design – Testing the questionnaire

Thank you very much for your attention