“Solving Data Inconsistencies and Data Integration with a Data Quality Manager” Presented by Maria del Pilar Angeles, Lachlan M.MacKinnon School of Mathematical and Computer Sciences, Heriot-Watt University, Edinburgh,EH14 4AS Doctoral Consortium

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 2 Agenda Introduction Proposal Data Quality Manager Components – Reference Model – Measure Model – Assessment Model – Quality Metadata Information Integration Process – Classification of DataSources – Selection of Best Datasources – Query Planning – Data Fusion – Ranking of Query results Questions

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 3 Naming Data Representation Domain Data scaling definition Data Precision Generalization Abstract Aggregation Data value attribute Schematic Attribute entity discrepancy Data value entity Known inconsistency Data Value Temporal inconsistency Acceptable inconsistency Default value Database id Entity Naming definition Union compatibility StructuralSchema isomorphism Conflicts Missing data item Attribute integrity constraints (Sheth92) Introduction Approached by Ontology Metadata Transformation rules Mapping

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 4 Introduction Emp_noNamesalary Alastair Freich Fernando Lujan NULL SSNfullnamesal A. Freich Fiona Shaning employeSFEsalary Al. FreichNULL Lauren MacMillan DS 1DS 2DS 3 Employee_numberFull_name_employeeSalary Alastair F A. Freich Al. FreichNULL Fernando LujanNULL Lauren MacMillan Fiona Shaning15000

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 5 Proposal We propose the development of a Data Quality Manager (DQM) to establish communication between the process of integration of information, the user and the application, to deal with semantic heterogeneity.

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 6 Proposal Local Schema 1 Local User 1Local User 2Local User N Wrapper Global User 1Global User 2Global User 3 Export Schema 1 Export Schema N Export Schema 2 Mediator Data Quality Manager Applications Global Schema Data Source 1 Data Source 2 Data Source N Wrapper … Local Schema 2 Local Schema N Global User M … Information Integration Process Selection of data sources Query Planning Detection and Fusion of data inconsistencies Query Integration Ranking query results Data Quality Manager DQ Metadata DQ Criteria Model DQ Assessment DQ Measure

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 7 Definition of Quality Criteria Reference Model DQM Components

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 8 Definition of Quality Criteria Definition of Metrics Measurement Model Reference Model DQM Components

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 9 Definition of Quality Criteria Definition of Metrics Definition of Assessment methods Assessment Model Measurement Model Reference Model DQM Components

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 10 Definition of Quality Criteria Definition of Metrics Definition of Assessment methods Definition of Quality Metadata (QMD) Quality Metadata Assessment Model Measurement Model Reference Model DQM Components

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 11 Completeness Accuracy Currency Survey, Queries, benchmarks # incomplete # total # errors # total Age + delivery time – input time Based on DQM components, classify the data sources QMD QMD Population DQM: Data Quality Manager QMD: Quality Meta Data

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 12 Data Quality Manager Selection of Best Data Sources Information Integration Process

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 13 Data Quality Manager Query Planning Selection of Best Data Sources Information Integration Process

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 14 Data Quality Manager Fusion of Data Inconsistencies Query Planning Selection of Best Data Sources Information Integration Process

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 15 Data Quality Manager Query Integration Fusion of Data Inconsistencies Query Planning Selection of Best Data Sources Information Integration Process

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 16 Data Quality Manager Ranking of Query results Query Integration Fusion of Data Inconsistencies Query Planning Selection of Best Data Sources Information Integration Process

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 17 User Query 1. The Quality user priorities are given by the user. Mapping Local/Global Schemas Selection of best Data Sources QMD Quality User Priorities Data sources Involved in the Query Ranking of best Data Sources 2. The ranking of best data sources involved in the query is given before execution

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 18 User Query Top ranking Query Plan Query Partition QMD Quality User Priorities QueryA QueryB QueryC Plan 1 Plan 2 Plan 3. Plan N Query Planning

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 19 ResultX Data Inconsistencies Detection Data fusion QMD Quality user priorities ResultY ResultZ Inconsistent Query Result Execute Query Plan Data Fusion Consistent Query Result As in the DQM is stored where data comes from, it is possible to make decisions at data fusion time.

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 20 QMD Quality user priorities Data Fusion ResultJ ResultK ResultL Query Integration Query Result Ranking Ranking Query Result Consistent Query Result

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 21 Conclusion Using Data Quality Manager we can.. Approach data value level inconsistencies during Information Integration Process, using data quality properties. User may demand different quality priorities at query time. Manage user quality priorities AND data quality properties to give the expected quality query result by the user. What we need to do now…. Identify tools for measurement, assessment and develop a QMD. Store quality of data sources involved in the heterogeneous system. Identify techniques for Ranking of data sources and plans involved in the query Inconsistency detection Fusion data using data source and data level properties Ranking of query results.

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 22 Questions ?

Solving Data Inconsistencies and Data Integration with a Data Quality Manager Angeles Maria del Pilar, MacKinnon Lachlan M. 23 Thanks !!