Some considerations on developing a DWH for SBS estimates Orietta Luzi – Mauro Masselli Istat - Italy march 2013

The rationale of DWH the complete use of all the information (survey and administrative data) we have on the whole or about the entire target population; to build up a platform in which we integrate data and processes (from capturing to integrating data, from checking data to estimating results to disseminating estimates). the advantages in cancellation of sampling errors from one side and process integration and standardization on the other, exceed the disadvantages due to increasing non sampling errors and the partial loss of control on administrative data

goals First step: To establish a common set of estimates (micro/macro) among SBS and NA on observed economy Second step: Integration of other surveys on business (structural – ICT,R&D, externalò trade….. and STS) Implications –Revision of sampling designs of SBS surveys –Revisions of production processes

Business Register BR central role as Selection List and “frame” The target population is identified with all the enterprises listed in Business Register. For each unit BR contains two kind of variables: – classification variables (NACE, legal Status, splits and joins, current status, etc..) –content variables (e.g. the total number of persons employed, subtotals of different kind of workers, labour costs, an estimation of turn over ….). – We assume that the classification variables and the variable “persons employed” and the implicit binary variable “existence of business” are by itself target variables and call them Z; they are kept by BR as they are and do not enter in any procedure of data treatment.

Target variables The target variables can be divided into two groups: A set of “basic variables” X* needed for the estimates required by the SBS - EU Regulation and by NA estimates ; The remaining variables Y* needed only for NA to be estimated conditionally to the first set

Sources the administrative sources: tax file, balance sheets, social security worker’s data, fiscal authority survey SBS surveys at moment, other structural business surveys in the next future

Administrative data How to asses the quality? Some results from essnet admin data Essentially: Definitions how much close are to SBS ones Data analysis »On overlapping data set »To identify biases analysis of distributions models on relationships between data sources

Administrative data Advantages: costs, completness Disadvantages: stability over time – data can be changed for internal decision of the producing administration »Operational definitions »Data indicators from overlapping Agreements with producers data sets Redisign sample surveys

From the collected variables to the target ones For each enterprise, some of the X* variables may exist in one or more of the S sources in different combinations, according to the dimension, the social security rules, the fiscal status etc. only for the sampled respondents units we have a complete set of target variables and these variables are set equal to X*. The variables Ai reported in source “i” may coincide or may approximate the corresponding X*; in the second case it could be possible to “correct” some of them obtaining a set of more precise Xi “estimate” of X*. number of sourcesbusiness ,7% ,6% ,1% ,1% ,2% no source ,3% total ,0%

From A i to X i x ij =a ij in case of “good” fitting x ij = f(a ij …..) otherwise,

The matrix X

BR Z & ID codes group of business Number of Variables X* SBS survey All the X* Source 2 Source3Source4Source5 M1K 1 = K (all) M2K 2 < K M3K 3 < K M4K 4 < K M5K 5 < K …. ………… ………..……….……………. MmKm < K No source 0 The matrix X* by establishing a hierarchy between sources

Macro-operators Establishing target populationList from Business Register and variables Z Establishing target variables X*Reconciliation between NA and SBS operative definitions Establishing A i ……..A S (collected variables) Analysis of data and definitions of the different sources A i with respect to the definitions of X*; the purpose is to evaluate the similarity of definitions in order: (i) to establish a hierarchy between the sources; (ii) to identify the correction to variables A From variables A to variables X;where it is necessary and possible, correction of A; the variables a ij are transformed into x ij by a “function”: x ij =a ij in case of “good” fitting or x ij = F(a ij …..) in case of correction establishing variable X i Outlier detection, selective editing Establishing variable X*Hierarchy between sources/variables

Donor methods Randomly By models Eg the projection estimator By calculating a new variable to be used as a distance between donor and recipient Latent variables model In all the methods we can use ex ante domains or can identify the appropriate variables to build up the donor domains

Establishing coherence: modify data of source i by data of source j Change some var X i Check the impact on the other var X i Modify other var X i asses X i E&I rules Outliers detection and removal

A simplified example Source i Persons employed > Turnover value added labour costs Intermediate costs »Services Value added/persons employed ? BR Persons employed and labour costs

Sources Hierarchy Ex ante - Based on How definitions of source i is close to SBS ones »BR/social security data »SBS sample survey »Balance sheets »Fiscal authority survey »Tax files Prevoius and current data analysis

Correct A data to obtain X data x i,k = f(a i,k,a i,m …) By data analysis on overlapping data sets By definitions Other considerations

How to fill in the matrix X* to obtain the matrix X** except for the group M1, survey respondents, in all the other cases we have a number of X* variable smaller than K (the needed target variables). So for obtaining the estimates we can consider two options: a massive imputation of missing values at micro level an estimation of missing X* at macro level

BR Survey Micro integration Z, X(1), X(2) …X(S) Selection of X*; E&I; coherence among different sources Micro Z X* Massive imputation Micro Z X**Y* SBS estimation Micro data treatment in the single sources admin sources Estimation of variables Y* NA estimates Micro integration Z, A(1) A(2)…A(S) Calculating X(1)…X(S) E&I; coherence among different sources on imputed units Micro NA treatment Massive imputation micro approach

BR Surve y Micro integration Z, X(1), X(2) …X(S) Selection of X*; outliers detection; Micro Z X* Summing up by domains; inconsistencie s clean up Domain D estimates X**Y* SBS estimates Micro data treatment in the single sources admin sources Estimation of variables Y* NA estimates Micro integration Z, A(1) A(2)…A(S) Calculating X(1)…X(S) macro approach

Cross section and longitudinal approach At moment the cross-sectional approach. However the longitudinal approach has the significant features using “variations” is the logic adopted by NA estimating procedures we have “more information” to dealing with. implication all the functions regarding the data control and imputation procedures could be developed considering both cross sectional and longitudinal “rules

Metadata Generally speaking, we can roughly divide them in three broad sets:  Metadata needed to manage the data  the information related to process and procedures,  the wider documentation related to the different topics in developing the DWH.  Sustainability different tools for managing