1. Chapter 10: data Quality and Integration
Modern Database Management

2. Objectives Define terms
Describe importance and goals of data governance
Describe importance and measures of data quality
Define characteristics of quality data
Describe reasons for poor data quality in organizations
Describe a program for improving data quality
Describe three types of data integration approaches
Describe the purpose and role of master data management
Describe four steps and activities of ETL for data integration for a data warehouse
Explain various forms of data transformation for data warehouses

3. Data Governance Data governance Data steward
High-level organizational groups and processes overseeing data stewardship across the organization
Data steward
A person responsible for ensuring that organizational applications properly support the organization’s data quality goals

4. Requirements for Data Governance
Sponsorship from both senior management and business units
A data steward manager to support, train, and coordinate data stewards
Data stewards for different business units, subjects, and/or source systems
A governance committee to provide data management guidelines and standards

5. Importance of Data Quality
If the data are bad, the business fails. Period.
GIGO – garbage in, garbage out
Sarbanes-Oxley (SOX) compliance by law sets data and metadata quality standards
Purposes of data quality
Minimize IT project risk
Make timely business decisions
Ensure regulatory compliance
Expand customer base

6. Characteristics of Quality Data
Uniqueness
Accuracy
Consistency
Completeness
Timeliness
Currency
Conformance
Referential integrity 6

7. Causes of poor data quality
External data sources
Lack of control over data quality
Redundant data storage and inconsistent metadata
Proliferation of databases with uncontrolled redundancy and metadata
Data entry
Poor data capture controls
Lack of organizational commitment
Not recognizing poor data quality as an organizational issue

8. Steps in Data quality improvement
Get business buy-in
Perform data quality audit
Establish data stewardship program
Improve data capture processes
Apply modern data management principles and technology
Apply total quality management (TQM) practices

9. Business Buy-in Executive sponsorship Building a business case
Prove a return on investment (ROI)
Avoidance of cost
Avoidance of opportunity loss

10. Data Quality Audit Statistically profile all data files
Document the set of values for all fields
Analyze data patterns (distribution, outliers, frequencies)
Verify whether controls and business rules are enforced
Use specialized data profiling tools

11. Data Stewardship Program
Roles:
Oversight of data stewardship program
Manage data subject area
Oversee data definitions
Oversee production of data
Oversee use of data
Report to: business unit vs. IT organization?

12. Improving Data Capture Processes
Automate data entry as much as possible
Manual data entry should be selected from preset options
Use trained operators when possible
Follow good user interface design principles
Immediate data validation for entered data

13. Apply modern data management principles and technology
Software tools for analyzing and correcting data quality problems:
Pattern matching
Fuzzy logic
Expert systems
Sound data modeling and database design

14. TQM Principles and Practices
TQM – Total Quality Management
TQM Principles:
Defect prevention
Continuous improvement
Use of enterprise data standards
Strong foundation of measurement
Balanced focus
Customer
Product/Service

15. Master Data Management (MDM)
Disciplines, technologies, and methods to ensure the currency, meaning, and quality of reference data within and across various subject areas
Three main architectures
Identity registry – master data remains in source systems; registry provides applications with location
Integration hub – data changes broadcast through central service to subscribing databases
Persistent – central “golden record” maintained; all applications have access. Requires applications to push data. Prone to data duplication.

16. Data Integration
Data integration creates a unified view of business data
Other possibilities:
Application integration
Business process integration
User interaction integration
Any approach requires changed data capture (CDC)
Indicates which data have changed since previous data integration activity

17. Techniques for Data Integration
Consolidation (ETL)
Consolidating all data into a centralized database (like a data warehouse)
Data federation (EII)
Provides a virtual view of data without actually creating one centralized database
Data propagation (EAI and ERD)
Duplicate data across databases, with near real-time delay

18. 18

19. The Reconciled Data Layer
Typical operational data is:
Transient–not historical
Not normalized (perhaps due to denormalization for performance)
Restricted in scope–not comprehensive
Sometimes poor quality–inconsistencies and errors

20. The Reconciled Data Layer
After ETL, data should be:
Detailed–not summarized yet
Historical–periodic
Normalized–3rd normal form or higher
Comprehensive–enterprise-wide perspective
Timely–data should be current enough to assist decision-making
Quality controlled–accurate with full integrity

21. The ETL Process Capture/Extract Scrub or data cleansing Transform
Load and Index
ETL = Extract, transform, and load
During initial load of Enterprise Data Warehouse (EDW)
During subsequent periodic updates to EDW

22. Figure 10-1 Steps in data reconciliation
Capture/Extract…obtaining a snapshot of a chosen subset of the source data for loading into the data warehouse
Figure 10-1 Steps in data reconciliation
Incremental extract = capturing changes that have occurred since the last static extract
Static extract = capturing a snapshot of the source data at a point in time 22

23. Figure 10-1 Steps in data reconciliation
Scrub/Cleanse…uses pattern recognition and AI techniques to upgrade data quality
Figure 10-1 Steps in data reconciliation
(cont.)
Fixing errors: misspellings, erroneous dates, incorrect field usage, mismatched addresses, missing data, duplicate data, inconsistencies
Also: decoding, reformatting, time stamping, conversion, key generation, merging, error detection/logging, locating missing data 23

24. Figure 10-1 Steps in data reconciliation
Transform … convert data from format of operational system to format of data warehouse
Figure 10-1 Steps in data reconciliation
(cont.)
Record-level:
Selection–data partitioning
Joining–data combining
Aggregation–data summarization
Field-level:
single-field–from one field to one field
multi-field–from many fields to one, or one field to many 24

25. Figure 10-1 Steps in data reconciliation
Load/Index…place transformed data into the warehouse and create indexes
Figure 10-1 Steps in data reconciliation
(cont.)
Refresh mode: bulk rewriting of target data at periodic intervals
Update mode: only changes in source data are written to data warehouse 25

26. Record Level Transformation Functions
Selection – the process of partitioning data according to predefined criteria
Joining – the process of combining data from various sources into a single table or view
Normalization – the process of decomposing relations with anomalies to produce smaller, well-structured relations
Aggregation – the process of transforming data from detailed to summary level

27. Figure 10-2 Single-field transformation
a) Basic Representation
In general, some transformation function translates data from old form to new form

28. Figure 10-2 Single-field transformation (cont.)
b) Algorithmic
Algorithmic transformation uses a formula or logical expression

29. Figure 10-2 Single-field transformation (cont.)
c) Table lookup
Table lookup uses a separate table keyed by source record code

30. Figure 10-3 Multi-field transformation
a) Many sources to one target

31. Figure 10-3 Multi-field transformation (cont.)
b) One source to many targets