1. S T A T I S T I C SD E N M A R KS T A T I S T I C SD E N M A R K Test of reporting forms Ashu Conrad Statistics Denmark March 21, 2011

2. S T A T I S T I C SD E N M A R KS T A T I S T I C SD E N M A R K Test of reporting forms How to test? What to test? How many tests? Who to test with? How to document testing? How to report results?

3. S T A T I S T I C SD E N M A R KS T A T I S T I C SD E N M A R K Types of testing Pre-test –Expert review –User test with colleagues –User test with actual respondents Pilot test –Test with small sample of enterprises –Response burden measurement

4. S T A T I S T I C SD E N M A R KS T A T I S T I C SD E N M A R K The testing process The test process can be divided into 6 phases: Analysis of results and any corrections to the form are an integrated part of testing Pretest 1 Expert review Pretest 2 User testing (internal /external) Pretest 3 Analysis of results and corrections Pilot test 1 Test with sample of enterprises Pilot test 2 Analysis of data and corrections Implement Implemen- tation

5. S T A T I S T I C SD E N M A R KS T A T I S T I C SD E N M A R K Pre-test 1: Expert review Group of experts review questionnaire to: –identify potential problems –get an understanding of the respondent's task –list potential problems and classify them to specific points where difficulties may occur Can include an analysis of prior data collection Who should conduct an expert review? –Survey design experts conduct the review together with subject matter experts (can also include interviewers)

6. S T A T I S T I C SD E N M A R KS T A T I S T I C SD E N M A R K Pre-test 2: User testing Methods: –Respondent experience - Observation –Response process – “Think aloud” –Local issues – Follow-up questions –Anticipated issues – Prepared questions Concurrent –While the respondent is answering Retrospective –After the respondent has answered

7. S T A T I S T I C SD E N M A R KS T A T I S T I C SD E N M A R K Pilot test - Methods Questionnaire is sent to a small number of respondents who are part of the population Measure how much time it takes to complete each questionnaire Debrief the respondents after they complete the questionnaire Analyze data to assure that the quality is acceptable and no data are missing Revise directions, question wording, or response categories where necessary

8. S T A T I S T I C SD E N M A R KS T A T I S T I C SD E N M A R K How many tests? User tests –Dependent on resources and facilities –5-8 participants is usually enough –Agree on a minimum number Pilot tests –Often a sample of 20-50 responses are collected and the data is analyzed –Give time to make any changes before implementation of the final form

9. S T A T I S T I C SD E N M A R KS T A T I S T I C SD E N M A R K Who to test with? User tests –Select respondents who you expect to have problems –Select respondents with maximum variation –Select accessible respondents Pilot tests –Sample from different sizes of enterprises within the population

10. S T A T I S T I C SD E N M A R KS T A T I S T I C SD E N M A R K How to document tests? Template for testing: –Background information for each participant (age, gender, education, occupation) –List of questions in the questionnaire –“Meta-questions” (questions about the questions) –Observations: issues, reason for problem, severity of problem –Suggestions for improvement

11. S T A T I S T I C SD E N M A R KS T A T I S T I C SD E N M A R K Types of issues Wording, comprehension, terms Access to relevant information Processing of information Response options and format Context of questions Anticipated issues (prepared questions) Suggestions

12. S T A T I S T I C SD E N M A R KS T A T I S T I C SD E N M A R K How to report results? Report on test results –Tested questions and issues –Number of tests –Profile of test participants/criteria for selection –Testing methods –Template from test –Summary of results and suggestions

13. S T A T I S T I C SD E N M A R KS T A T I S T I C SD E N M A R K Thank you for your attention Ashu Conrad - smc@dst.dk