1. © 2011 Pearson Education, Inc. Publishing as Prentice Hall 1 Chapter 10: Data Quality and Integration Modern Database Management 10 th Edition Jeffrey A. Hoffer, V. Ramesh, Heikki Topi

2. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 2 Objectives Define terms Define terms Describe importance and goals of data governance Describe importance and goals of data governance Describe importance and measures of data quality Describe importance and measures of data quality Define characteristics of quality data Define characteristics of quality data Describe reasons for poor data quality in organizations Describe reasons for poor data quality in organizations Describe a program for improving data quality Describe a program for improving data quality Describe three types of data integration approaches Describe three types of data integration approaches Describe the purpose and role of master data management Describe the purpose and role of master data management Describe four steps and activities of ETL for data integration for a data warehouse Describe four steps and activities of ETL for data integration for a data warehouse Explain various forms of data transformation for data warehouses Explain various forms of data transformation for data warehouses

3. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall Data Governance Data governance Data governance High-level organizational groups and processes overseeing data stewardship across the organization High-level organizational groups and processes overseeing data stewardship across the organization Data steward Data steward A person responsible for ensuring that organizational applications properly support the organization’s data quality goals A person responsible for ensuring that organizational applications properly support the organization’s data quality goals 3

4. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall Requirements for Data Governance Sponsorship from both senior management and business units Sponsorship from both senior management and business units A data steward manager to coordinate data stewards A data steward manager to coordinate data stewards Data stewards for different business units, subjects, and/or source systems Data stewards for different business units, subjects, and/or source systems A governance committee to provide data management guidelines and standards A governance committee to provide data management guidelines and standards 4

5. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 5 Importance of Data Quality Minimize IT project risk Minimize IT project risk Make timely business decisions Make timely business decisions Ensure regulatory compliance Ensure regulatory compliance Expand customer base Expand customer base

6. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall Characteristics of Quality Data Uniqueness Uniqueness Accuracy Accuracy Consistency Consistency Completeness Completeness Timeliness Timeliness Currency Currency Conformance Conformance Referential integrity Referential integrity 6

7. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 7 Causes of poor data quality External data sources External data sources Lack of control over data quality Lack of control over data quality Redundant data storage and inconsistent metadata Redundant data storage and inconsistent metadata Proliferation of databases with uncontrolled redundancy and metadata Proliferation of databases with uncontrolled redundancy and metadata Data entry Data entry Poor data capture controls Poor data capture controls Lack of organizational commitment Lack of organizational commitment Not recognizing poor data quality as an organizational issue Not recognizing poor data quality as an organizational issue

8. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 8 Data quality improvement Get business buy-in Get business buy-in Perform data quality audit Perform data quality audit Establish data stewardship program Establish data stewardship program Improve data capture processes Improve data capture processes Apply modern data management principles and technology Apply modern data management principles and technology Apply total quality management (TQM) practices Apply total quality management (TQM) practices

9. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall Business Buy-in Executive sponsorship Executive sponsorship Building a business case Building a business case Prove a return on investment (ROI) Prove a return on investment (ROI) Avoidance of cost Avoidance of cost Avoidance of opportunity loss Avoidance of opportunity loss 9

10. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall Data Quality Audit Statistically profile all data files Statistically profile all data files Document the set of values for all fields Document the set of values for all fields Analyze data patterns (distribution, outliers, frequencies) Analyze data patterns (distribution, outliers, frequencies) Verify whether controls and business rules are enforced Verify whether controls and business rules are enforced Use specialized data profiling tools Use specialized data profiling tools 10

11. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall Data Stewardship Program Roles: Roles: Oversight of data stewardship program Oversight of data stewardship program Manage data subject area Manage data subject area Oversee data definitions Oversee data definitions Oversee production of data Oversee production of data Oversee use of data Oversee use of data Report to: business unit vs. IT organization? Report to: business unit vs. IT organization? 11

12. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall Improving Data Capture Processes Automate data entry as much as possible Automate data entry as much as possible Manual data entry should be selected from preset options Manual data entry should be selected from preset options Use trained operators when possible Use trained operators when possible Follow good user interface design principles Follow good user interface design principles Immediate data validation for entered data Immediate data validation for entered data 12

13. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall TQM Principles and Practices TQM – Total Quality Management TQM – Total Quality Management TQM Principles: TQM Principles: Defect prevention Defect prevention Continuous improvement Continuous improvement Use of enterprise data standards Use of enterprise data standards Balanced focus Balanced focus Customer Customer Product/Service Product/Service 13

14. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall Master Data Management (MDM) The disciplines, technologies, and methods to ensure the currency, meaning, and quality of reference data within and across various subject areas The disciplines, technologies, and methods to ensure the currency, meaning, and quality of reference data within and across various subject areas Three main architectures Three main architectures Identity registry Identity registry Integration hub Integration hub Persistent Persistent 14

15. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall Data Integration Data integration creates a unified view of business data Data integration creates a unified view of business data Other possibilities: Other possibilities: Application integration Application integration Business process integration Business process integration User interaction integration User interaction integration Any approach requires changed data capture (CDC) Any approach requires changed data capture (CDC) Indicates which data have changed since previous data integration activity Indicates which data have changed since previous data integration activity 15

16. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall Techniques for Data Integration Consolidation (ETL) Consolidation (ETL) Consolidating all data into a centralized database (like a data warehouse) Consolidating all data into a centralized database (like a data warehouse) Data federation (EII) Data federation (EII) Provides a virtual view of data without actually creating one centralized database Provides a virtual view of data without actually creating one centralized database Data propagation (EAI and ERD) Data propagation (EAI and ERD) Duplicate data across databases, with near real-time delay Duplicate data across databases, with near real-time delay 16

17. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 17

18. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 18 The Reconciled Data Layer Typical operational data is: Typical operational data is: Transient–not historical Transient–not historical Not normalized (perhaps due to denormalization for performance) Not normalized (perhaps due to denormalization for performance) Restricted in scope–not comprehensive Restricted in scope–not comprehensive Sometimes poor quality–inconsistencies and errors Sometimes poor quality–inconsistencies and errors After ETL, data should be: After ETL, data should be: Detailed–not summarized yet Detailed–not summarized yet Historical–periodic Historical–periodic Normalized–3 rd normal form or higher Normalized–3 rd normal form or higher Comprehensive–enterprise-wide perspective Comprehensive–enterprise-wide perspective Timely–data should be current enough to assist decision- making Timely–data should be current enough to assist decision- making Quality controlled–accurate with full integrity Quality controlled–accurate with full integrity

19. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 19 The ETL Process Capture/Extract Capture/Extract Scrub or data cleansing Scrub or data cleansing Transform Transform Load and Index Load and Index ETL = Extract, transform, and load

20. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 20 Static extract Static extract = capturing a snapshot of the source data at a point in time Incremental extract Incremental extract = capturing changes that have occurred since the last static extract Capture/Extract…obtaining a snapshot of a chosen subset of the source data for loading into the data warehouse Figure 10-1 Steps in data reconciliation

21. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 21 Scrub/Cleanse…uses pattern recognition and AI techniques to upgrade data quality Fixing errors: Fixing errors: misspellings, erroneous dates, incorrect field usage, mismatched addresses, missing data, duplicate data, inconsistencies Also: Also: decoding, reformatting, time stamping, conversion, key generation, merging, error detection/logging, locating missing data Figure 10-1 Steps in data reconciliation (cont.)

22. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 22 Transform = convert data from format of operational system to format of data warehouse Record-level: Selection–data partitioning Joining–data combining Aggregation–data summarization Field-level: single-field–from one field to one field multi-field–from many fields to one, or one field to many Figure 10-1 Steps in data reconciliation (cont.)

23. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 23 Load/Index= place transformed data into the warehouse and create indexes Refresh mode: Refresh mode: bulk rewriting of target data at periodic intervals Update mode: Update mode: only changes in source data are written to data warehouse Figure 10-1 Steps in data reconciliation (cont.)

24. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 24 Figure 10-2 Single-field transformation In general–some transformation function translates data from old form to new form a) Basic Representation

25. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 25 Figure 10-2 Single-field transformation (cont.) Algorithmic transformation uses a formula or logical expression b) Algorithmic

26. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 26 Figure 10-2 Single-field transformation (cont.) Table lookup–another approach, uses a separate table keyed by source record code c) Table lookup

27. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 27 Figure 10-3 Multi-field transformation a) Many sources to one target

28. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 28 Figure 10-3 Multi-field transformation (cont.) b) One source to many targets

29. Chapter 10 © 2011 Pearson Education, Inc. Publishing as Prentice Hall 29 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher. Printed in the United States of America. Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall