Towards a Process Oriented View on Statistical Data Quality Michaela Denk, Wilfried Grossmann

Contents  Approaches Towards Data Quality  Example Data Integration  A Generic Statistical Workflow Model  Quality Assessment  Conclusions 2Grossmann, Denk

Approaches Towards Data Quality  The usual approach towards data quality is the Reporting View  Define a number of so called quality dimensions and evaluate the final product according to criteria for these dimensions Some frequently used dimensions: Accuracy, Relevance, Accessibility, Timeliness, Coherence, Comparability,... 3Grossmann, Denk

Approaches Towards Data Quality  These dimensions are many times broken down in sub-dimensions Example Accuracy: Sampling Effects, Representativity, Over-Coverage, Under-Coverage, Missing Values, Imputation Error,....  Such an approach is fine as long as production of data follows a predefined scheme, which has limited degrees of freedom 4Grossmann, Denk

Approaches Towards Data Quality  If we have a number of different opportunities for data production such an approach is probably not the best one  Compare the ideas of Total Quality Management (TQM) in industrial production: Systematic treatment of the influence of different production steps on quality of the final product  We need a Processing View on data quality: How is data quality influenced by production? 5Grossmann, Denk

Approaches Towards Data Quality  How can we arrive at a Processing View on data quality?  We need a statistical workflow model  We have to organize the processing information necessary for quality assessment in appropriate way C ompare (old) ideas of B. Sundgren about capture of metadata 6Grossmann, Denk

Approaches Towards Data Quality  We have to know functions for assessing quality Output_Quality = F(Input_Quality, Processing_Quality)  Such functions have to be applied according to The object we are interested in, e.g. a variable or a population or a classification The quality aspect we are interested in 7Grossmann, Denk

Example Data Integration  Data integration occurs many times in statistical data production, in particular in case of data production from administrative sources  It uses a number of operations usually understood as data pre-processing  Basic goal: Combine information from two or more already existing data sets 8Grossmann, Denk

Example Data Integration  Example for a Data Integration Dataflow Input → Integration → Post-alignment 9Grossmann, Denk

Example Data Integration  Top level task description  Match the datasets according matching key  Align V1 (gender)  Align V2 (status) 10Grossmann, Denk

Example Data Integration  Details, Decisions to be made  Are datasets appropriate? Quality of matching keys Quality of data sources  Method for identification of matches?  Method for handling ambiguities in V1 (Gender)?  Method for imputation of V2 (Status)?  How is quality measured At level of a summary measure? At level of a specific variable? At level of individual records? 11Grossmann, Denk

Example Data Integration  There are no generally accepted standard tools and methods for answering such questions  Probably we have to compare a number of alternative approaches  Apply the generic format for different datasets  Try different statistical methods and models  Use different methods for quality assessment Traditional formulas Simulation based evaluation Assessment by using strategic surveys 12Grossmann, Denk

Example Data Integration  Conclusion  Different statistical methods may be an essential part of data production and quality assessment  There is no longer such a clear distinction between “objective” data collection and statistical analysis  Statistics generates added value beyond (administrative) accounting and IT 13Grossmann, Denk

A Generic Statistical Workflow Model  Statistical Workflow: A mixture from  Business Workflow (Process oriented)  Scientific Workflow (Data oriented)  Quality evaluation is the main control element of the process  We have to consider the workflow at two levels  Meta-level (Control of the process)  Data-level (Production of data) 14Grossmann, Denk

A Generic Statistical Workflow Model  Building blocks of the workflow model  Transformations (Basic data operations)  Process components (Tasks) defined by: Task definition Pre-Alignment Feasibility Check Main Transformation Post-Alignment Quality Evaluation  Workflow (Sequence of Process components) 15Grossmann, Denk

A Generic Statistical Workflow Model  Example for Data Integration Component Workflow 16Grossmann, Denk

A Generic Statistical Workflow Model  In order to understand how statistics influences the boxes and data quality let us zoom into the box for post-alignment 17Grossmann, Denk

Quality Assessment  For quality assessment we need a detailed description of the changes in meta-information during the dataflow 18Grossmann, Denk

Quality Assessment  Example for meta- information flow in data integration  Details for register based census in the presentation of Fiedler/Lenk in Session 26 (Thursday) 19Grossmann, Denk

Quality Assessment  Example: Assessment of accuracy of variables V1 (Gender) and V2 (Status) in the example 20Grossmann, Denk

Quality Assessment  V1 (Gender)  Input Coincidence of matching keys in both datasets Matching of the variable Gender in both datasets Beliefs about quality of the variable in both sources  Accuracy Assessment It seems that models developed in decision analysis (calculus from belief networks) are appropriate Alternatively we can use a strategic sample to check whether our prior beliefs are correct and our decision rule is confirmed by statistical arguments 21Grossmann, Denk

Quality Assessment  V2 (Status):  Input Coincidence of matching keys in both datasets Reliability of the model used for imputation Measurement technique for quality of imputation  Accuracy Assessment In this case we can apply traditional statistical techniques like false classification rate, ROC-curve, simulation 22Grossmann, Denk

Conclusions  We have presented a model, which allows tighter coupling of quality assessment to the data production process  Such a model seems useful if data production has more degrees of freedom  What data should be used?  What techniques should be used  The approach allows identification of the different factors influencing quality 23Grossmann, Denk

Conclusions  It allows formulation of precise questions about possible alternatives and defines new issues for research in statistical data quality  Hopefully it helps to understand better the added value generated by statistics 24Grossmann, Denk

Thank you for attention 25Grossmann, Denk