1. Eurostat Overall design

2. Presented by Eva Elvers Statistics Sweden

3. Outline What is ‒ a survey? ‒ statistics? The GSBPM Quality – some perspectives Design of a statistical survey – some aspects

4. What is a survey? From EHQR 2009 1.Sample survey 2.Census 3.Statistical process using administrative source(s) 4.Statistical process involving multiple data sources 5.Price or other economic index process 6.Statistical compilation

5. What is statistics? A statistical table A statistical measure (e.g. sum, mean or median) is used to summarise individual variable values (e.g. turnover) for the statistical units (e.g. enterprise) in a group. The totality of considered statistical units is called the population. There are sub-populations; domains of estimation. There are reference times for variables, units …

6. Typical for statistics (1) Statistical characteristics or parameters as above. Parameters of interest, depending on use and user. Target parameters based on target variables, target units, target population, … Statistics are estimates aiming at the target parameters.

7. Typical for statistics (2) Variables of interest, target variables Observation variables Consider ‒ measurements; ‒ quality; ‒ costs; ‒ sources; ‒ response burden.

8. Typical for statistics (3) Population and subpopulations Several unit types may be needed ‒ enterprise, kind of activity unit, local unit, … ‒ interest, target, observation A frame leading to a frame population ‒ Compare with target population (unit type) ‒ Coverage deficiencies ‒ Time aspects

9. Typical for statistics (4) Consider sources for data collection ‒ direct data collection ‒ accessible from earlier collection ‒ administrative data Express statistical inference ‒ finite population ‒ sample, register

10. The GSBPM – Generic Statistical Business Process Model – the UNECE version in 2009, phases and sub-processes

11. Phases of the statistics production process Phases 1 – 3, preparatory ‒ Specify needs ‒ Design ‒ Build Phases 4 – 8 ‒ Collect, Process, Analyse, Disseminate, Archive Phase 9 ‒ Evaluate

12. Quality Assurance and Quality Control Q is fitness for use, fitness for purpose,... QA: ‒ Approaches and methods to achieve the intended/stated quality. ‒ Providing confidence that the quality requirements will be met. QC: ‒ Verification that the quality achieved was as expected. ‒ Checks,...

13. Typical in a statistical office Business register ‒ Basis, frame construction, auxiliary information Primary statistics ‒ Short-term statistics (monthly, quarterly), STS ‒ Structural statistics (annual), SBS Secondary statistics ‒ National accounts, Balance of payment etc.

14. Output quality components European Statistics CoP 2011, EHQR 2009 Relevance Accuracy and reliability Timeliness and punctuality Coherence and comparability Accessibility and clarity

15. Accuracy and reliability (e.g. SIMS 2013) Sources of error (in-accuracy) ‒ Sampling ‒ Coverage ‒ Measurement ‒ Non-response ‒ Processing ‒ Model assumption Data revision (reliability) ‒ Data revision, average size

16. Coherence and comparability Meaning ‒ Adequacy of being combined, used together Important ‒ Definitions: concepts, units, populations, … ‒ Methods Examples ‒ Across domains, with National Accounts, … ‒ Comparability special case, e.g. EU geographical ‒ Over time

17. Relevance User needs ‒ Content: concepts, … ‒ Quality components User satisfaction Completeness ‒ Data (statistics) and metadata ‒ Regulations ‒ The system of surveys

18. Design aspects Survey situation ‒ Design a new survey ‒ Redesign a survey ‒ Continuous improvements Scope ‒ Methodological, technical, … ‒ Set of surveys, survey, sub-process, tool, system, …

19. Design aim: ”optimisation” There may be a simple statement, like ‒ Minimum cost given quality. ‒ Maximum quality given cost (quality is multi-facetted). Design, two core tasks are to make “optimal” ‒ choices, e.g. of methods; ‒ allocations, e.g. of resources.

20. GSBPM and output quality

21. Which principles do you use to design 1.the frame? 2.the sample? 3.the data collection method? 4.the contact strategy? 5.the editing? 6.the estimation?

22. What do you need to design What is the input? (groups 1, 2, 6) What is the output? (groups 3, 4, 5)

23. Theory or principles for some parts/processes – four examples Sampling and estimation ‒ Mean Squared Error, MSE. Response process ‒ Comprehension, retrieval,... Data collection ‒ Modes, type of data, timeliness etc. Editing ‒ Quality control.

24. Design work in brief Design a forthcoming survey round for statistics (macro data) or micro data through appropriate competences in cooperation/agreement with customers/users/stakeholders so that quality is sufficient for the intended use, the production is within budget and cost-effective (in the long run), and with regard taken to respondents (burden).

25. To design you need to know For appropriate choices and allocations: ‒ The population and how to reach its objects/units. ‒ Variations in the population and sub-groups. ‒ How questions are interpreted. ‒ Possibilities to reply, provide the information requested. ‒...

26. How could you learn? Pilot study ‒ Qualitative, make “discoveries” ‒ Quantitative, statistical inference Embedded experiment ‒ On-going survey ‒ Utilise the sample

27. Responsive/adaptive design Plan with successive decisions ‒ Mostly described for household statistics and telephone interviewing ‒ Information from previous rounds ‒ Successive information this round (new survey) Examples ‒ “Mile-stones” with re-considerations ‒ Reminder strategy, contact mode, …

28. Paradata, metadata-driven Process data, paradata ‒ Collect, suitably chosen set, with aim ‒ Analyse and improve Metadata ‒ Information about the statistics ‒ For the statistics production: parameters, …

29. What is included in an ”optimisation”? Find e.g. the best quality for given cost and subject to a set of constraints, such as ‒ Regulations. ‒ Rules for data collection, response burden. ‒ Resources, financial and personnel. ‒ Quality depends on use(s) – user dialogue! ‒ Quality is multi-facetted!

30. “Conclusions” – aspects on design Teamwork On-going work Constraints assist Metadata, paradata (process data) Architecture: methodology, IT, … Data integration