1. 1 Introduction to Data Quality Management (DQM)

2. 2 What is Quality? Informally Some things are better than others i.e. they are of higher quality. How much “better” is better? Is the right item the best item to purchase? How about after the purchase? What is quality of service? The bank example

3. 3 What is Quality? Formally “Quality is conformance to requirements” P. Crosby, “Quality is Free” 1979 “Degree of excellence” Webster’s Third New International Dictionary

4. 4 What is Quality? Examples from Auto Industry Quality means meeting customer’s needs, not necessarily exceeding them. Quality means improving things customers care about, because that makes their lives easier and more comfortable. Why example from auto-industry?

5. 5 What is Data Quality?  Albert Einstein Height = 5’8” Weight = 160 lbs Gender = Male Age = 35 yrs Emp_ID = 440 All data is an abstraction of something real What is Data?

6. 6 What is Data Quality? Intrinsic Data Quality Electronic reproduction of reality. Realistic Data Quality Degree of utility or value of data to business.

7. 7 Data Quality & Organizations Intelligent Learning Organization: High-quality data is an open, shared resource with value- adding processes. The dysfunctional learning organization: Low-quality data is a proprietary resource with cost-adding processes.

8. 8 Law #1 - “Data that is not used cannot be correct!” Law #2 - “Data quality is a function of its use, not its collection!” Law #3 - “Data will be no better than its most stringent use!” Law #4 - “Data quality problems increase with the age of the system!” Law #5 – “The less likely something is to occur, the more traumatic it will be when it happens!” Orr’s Laws of Data Quality

9. 9 Total Quality Control (TQM) Philosophy of involving all for systematic and continuous improvement. It is customer oriented. Why? TQM incorporates the concept of product quality, process control, quality assurance, and quality improvement. Quality assurance is NOT Quality improvement

10. 10 Co$t of fixing data quality Lowest Quality Highest quality Cost of achieving quality Defect minimization is economical. Defect elimination is very very expensive. Exponential rise in cost

11. 11 Co$t of Data Quality Defects  Controllable Costs  Recurring costs for analyzing, correcting, and preventing data errors  Resultant Costs  Internal and external failure costs of business opportunities missed.  Equipment & Training Costs

12. 12 Where data quality is critical?  Almost everywhere, some examples:  Marketing communications.  Customer matching.  Retail house-holding.  Combining MIS systems after acquisition.

13. 13 Characteristics or Dimensions of Data Quality Data Quality Characteristic Definition Accuracy Qualitatively assessing lack of error, high accuracy corresponding to small error. Completeness The degree to which values are present in the attributes that require them.

14. 14 Completeness Vs Accuracy 95% accurate and 100% complete OR 100% accurate and 95% complete Which is better? Depends on data quality (i) tolerances, the (ii) corresponding application and the (iii) cost of achieving that data quality vs. the (iv) business value.

15. 15 Characteristics or Dimensions of Data Quality Data Quality Characteristic Definition Consistency A measure of the degree to which a set of data satisfies a set of constraints. Timeliness A measure of how current or up to date the data is. Uniqueness The state of being only one of its kind or being without an equal or parallel. Interpretability The extent to which data is in appropriate languages, symbols, and units, and the definitions are clear. Accessibility The extent to which data is available, or easily and quickly retrievable Objectivity The extent to which data is unbiased, unprejudiced, and impartial

16. 16 Total DQM

17. 17 Data Quality Management Process Establish TDQM Environment Scope Data Quality Projects & Develop Implementation Plans Implement Data Quality Projects (Define, Measure, Analyze, Improve) Evaluate Data Quality Management Methods

18. 18 Data Quality Management Process 1.Establish Data Quality Management Environment IS project managers Development professionals. Functional users of legacy information systems with domain knowledge IS developers know solutions but don’t know how and where to modify

19. 19 Data Quality Management Process 2. Scope Data Quality Projects & Develop Implementation Plans Task Summary: Project goals, scope, and potential benefits Task Description: Describe data quality analysis tasks Project Approach: Summarize tasks and tools used to provide a baseline of existing data quality Schedule: Identify task start, completion dates, and project milestones Resources: Include costs connected with tools acquisition, labor hours (by labor category), training, travel, and other direct and indirect costs

20. 20 Data Quality Management Process 3. Implement Data Quality Projects (Define, Measure, Analyze, Improve) Define: Identify functional user DQ requirements and establish DQ metrics Measure: conformance to current business rules and develop exception reports Analyze: Verify, validate, and assess poor DQ causes. Define improvement opportunities Improve: Select/prioritize DQ improvement opportunities i.e. data entry procedures, updating data validation rules, and/or company data standards.

21. 21 Data Quality Management Process 4. Evaluate Data Quality Management Methods modifying or rejuvenating existing methods of DQ management determining if DQ projects have helped to achieve demonstrable goals and benefits. Evaluating and assessing DQ work as, it is not a program, but a new way of doing business.

22. 22 The House of Quality Matrix

23. 23 How to improve Data Quality? The four categories of Data Quality Improvement  Process  System  Policy & Procedure  Data Design

24. 24 Quality Management Maturity Grid CMM Level-1 Uncertainty CMM Level-2 Awakening CMM Level-3 Enlightenment CMM Level-4 Wisdom CMM Level-5 Certainity

25. 25 Misconceptions on Data Quality  You Can Fix Data  Problem NOT in data, but how it was used.  It is NOT a one time process.  Buying a cleansing tool is NOT the solution.  Some live with the problem, cant afford the tool.  Data Quality is an IT Problem  It is the company problem.  Define the metrics of quality.  Business has to strike a balance between quality and ROI.  Joint business and IT effort.

26. 26 Misconceptions on Data Quality  (All) Problem is in the Data Sources or Data Entry  NOT the only problem.  Systems could be responsible, but actually it is the metrics.  Two divisions using different codes for same entity.  Need to track, trace, check data from creation to usage.  The Data Warehouse will provide a single source of truth  In ideal world it is indeed true.  In real world maybe multiple data warehouses, data marts, external source i.e. silos of data resulting in multiple sources of “truth”.  Even with single source of truth, if transformations and interpretations are different, an issue.