1. Slide 1 Chapter 17: Data Integration Practices and Relational DBMS Extensions Database Design, Application Development, and Administration, 5 th Edition Copyright © 2011 by Michael V. Mannino. All rights reserved. Chapter 17 Data Integration Practices and Relational DBMS Extensions for Data Warehouses

2. Slide 2 Chapter 17: Data Integration Practices and Relational DBMS Extensions Outline  Data integration concepts  SQL extensions for multidimensional data  Summary data storage and optimization

3. Slide 3 Chapter 17: Data Integration Practices and Relational DBMS Extensions Data Integration Concepts  Data sources  Workflow representation  Data cleaning techniques  Data integration architectures and tools  Managing the refresh process

4. Slide 4 Chapter 17: Data Integration Practices and Relational DBMS Extensions Data Sources  Cooperative:  Notification using triggers  Requires source system changes  Logged  Readily available  Extraneous data in logs  Queryable  Queries using timestamps  Requires timestamps in source data  Snapshot  Periodic dumps of source data  Significant processing for difference operations

5. Slide 5 Chapter 17: Data Integration Practices and Relational DBMS Extensions Maintenance Workflow

6. Slide 6 Chapter 17: Data Integration Practices and Relational DBMS Extensions Initial Data Warehouse Load  Major development activity  Different characteristics than refresh  Discover many data quality problems  Difficult to estimate time requirements  Part of data warehouse extensions

7. Slide 7 Chapter 17: Data Integration Practices and Relational DBMS Extensions Data Quality Measures  Completeness  Lack of ambiguity  Timeliness  Correctness  Consistency

8. Slide 8 Chapter 17: Data Integration Practices and Relational DBMS Extensions Data Quality Problems  Multiple identifiers  Multiple field names  Different units  Missing values  Orphaned transactions  Multipurpose fields  Conflicting data  Different update times

9. Slide 9 Chapter 17: Data Integration Practices and Relational DBMS Extensions Data Cleaning Tasks  Parsing  Correcting  Standardizing  Matching  Consolidating

10. Slide 10 Chapter 17: Data Integration Practices and Relational DBMS Extensions Parsing Locates and identifies individual data elements in the source files Context free and context sensitive parsing Requires pattern specification Regular expressions (regex) for pattern specification Isolates parsed data elements in the target record

11. Slide 11 Chapter 17: Data Integration Practices and Relational DBMS Extensions Regular Expression Basics  Search expression: combination of meta characters, literals, and escape characters that define a search pattern  Meta character: special meaning within a search expression  Literal: character to match exactly  Escape sequence: removes special meaning of meta character

12. Slide 12 Chapter 17: Data Integration Practices and Relational DBMS Extensions Simple Examples  Target strings  String1 Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)  String2 Mozilla/4.75 [en](X11;U;Linux2.2.16-22 i586)  Regular expressions  “m” match in String1 but not String2  “a/4” match in both String1 and String2  “5 [“ no match in String1 but match in String2  “le” match in String1 but no match in String2

13. Slide 13 Chapter 17: Data Integration Practices and Relational DBMS Extensions Enumeration, Range and Position Meta Characters  [] match any enclosed character once  [Ff] matches F or f  [0123456789] matches any digit  [0-9] matches any digit  [^0-9] matches anything other than a digit  Position characters  ^ string beginning: ^win matches window but not erwin  $ string ending: win$ matches erwin but not window . any character in specified position: win. matches window but not erwin

14. Slide 14 Chapter 17: Data Integration Practices and Relational DBMS Extensions Iteration Meta Characters  ?: matches preceding character 0 or 1 times  *: matches preceding character 0 or more times  +: matches preceding character 1 or more times  {n} matches preceding character exactly n times  {n,m} matches preceding character a minimum of n and maximum of m times

15. Slide 15 Chapter 17: Data Integration Practices and Relational DBMS Extensions Iteration Meta Character Examples  colou?r matches both color and colour  tre* matches tree, tread, and trough  tre+ matches tree and tread but not trough  [0-9]{3}-[0-9]{4} matches 123-4567 but not 1234-567  ba{2,3}b matches baab and baaab but not bab or baaaab

16. Slide 16 Chapter 17: Data Integration Practices and Relational DBMS Extensions Groups in Regular Expressions  Pair of parentheses used to group subpatterns  Usage  Reuse patterns in an expression  Parsing subpatterns  Example  Input: abbc  Pattern: a(b*)c  Group 0: abbc, Group 1: bb

17. Slide 17 Chapter 17: Data Integration Practices and Relational DBMS Extensions Parsing Example

18. Slide 18 Chapter 17: Data Integration Practices and Relational DBMS Extensions Correcting Values  Missing values -Default value -Typical value: median or average -Relationship to other attributes  Conflicting values -More recent value -More credible source

19. Slide 19 Chapter 17: Data Integration Practices and Relational DBMS Extensions Correcting Example

20. Slide 20 Chapter 17: Data Integration Practices and Relational DBMS Extensions Standardization  Applies conversion routines to transform data into its preferred (and consistent) format  Uses both standard and custom business rules  Common standardizations:  Unit of measure transformations  Standard abbreviations (state names, titles, street types)

21. Slide 21 Chapter 17: Data Integration Practices and Relational DBMS Extensions Standardization Example

22. Slide 22 Chapter 17: Data Integration Practices and Relational DBMS Extensions Matching  Identify duplicates  Difficult matching process: no common identifier  Data mining problem  Known as the record linkage problem or entity identification problem  Many approaches

23. Slide 23 Chapter 17: Data Integration Practices and Relational DBMS Extensions Entity Identification Applications  Marketing  Law enforcement  Fraud detection  Government

24. Slide 24 Chapter 17: Data Integration Practices and Relational DBMS Extensions Entity Identification Outcomes Predicted \ ActualMatchNon Match MatchTrue matchFalse match Possible MatchInvestigation Non MatchFalse non matchTrue non match

25. Slide 25 Chapter 17: Data Integration Practices and Relational DBMS Extensions Distance Measures for Comparison  Edit distance: number of deletions, insertions, or substitutions required to transform a source string into a target string.  N-gram distance: breaks text into subsequences of length N  Phonetic distance: codes words into standard consonant sounds

26. Slide 26 Chapter 17: Data Integration Practices and Relational DBMS Extensions Matching Example

27. Slide 27 Chapter 17: Data Integration Practices and Relational DBMS Extensions Consolidation  Analyzing and identifying relationships between matched records  Consolidating/merging them into one representation.

28. Slide 28 Chapter 17: Data Integration Practices and Relational DBMS Extensions Consolidation Example

29. Slide 29 Chapter 17: Data Integration Practices and Relational DBMS Extensions Household Consolidation WilliamJonesJanetJonesKarenJonesWilliam Jones Jr.

30. Slide 30 Chapter 17: Data Integration Practices and Relational DBMS Extensions Legacy Systems View (3 Clients) Account No. 83451234 Policy No. ME309451-2 Transaction B498/97

31. Slide 31 Chapter 17: Data Integration Practices and Relational DBMS Extensions The Reality – ONE Client Account No. 83451234 Policy No. ME309451-2 Transaction B498/97

32. Slide 32 Chapter 17: Data Integration Practices and Relational DBMS Extensions Data Integration Tools  Specification based for workflow and transformations  Minimize custom coding  Open source, third party, and DBMS based tools  Evolution from disjoint, specialized tools to integrated development environments  Sometimes called ETL tools

33. Slide 33 Chapter 17: Data Integration Practices and Relational DBMS Extensions Data Integration Tool Features  Workflow specification  Transformation specification  Job management  Data profiling  Change data capture  Integrated development environment  Repository  Connectivity for databases and files

34. Slide 34 Chapter 17: Data Integration Practices and Relational DBMS Extensions Data Integration Marketplace  Balanced with proprietary and open- source products from DBMS vendors and third party vendors  Base products and subscription services for open source products  Developing marketplace with acquisitions and new product development

35. Slide 35 Chapter 17: Data Integration Practices and Relational DBMS Extensions Data Integration Architectures

36. Slide 36 Chapter 17: Data Integration Practices and Relational DBMS Extensions Talend Open Data Solutions  Base product: Talend Open Studio  Extended product and services: Talend Integration Suite and MPx/RTx options  Repository service: Talend On Demand  Data quality/profiling product: Talend Data Quality

37. Slide 37 Chapter 17: Data Integration Practices and Relational DBMS Extensions Talend Open Studio  Business modeling  Graphical job design using components  Meta data respository  Database connectivity  Job execution

38. Slide 38 Chapter 17: Data Integration Practices and Relational DBMS Extensions Talend Job Design  Graphical notation for transformations  Palette of transformation components  Data quality components  Database and file components  ELT components: transformations in target database  Processing components  XML components

39. Slide 39 Chapter 17: Data Integration Practices and Relational DBMS Extensions Excel Job Design Example

40. Slide 40 Chapter 17: Data Integration Practices and Relational DBMS Extensions Talend Component Details

41. Slide 41 Chapter 17: Data Integration Practices and Relational DBMS Extensions Excel Job Execution Example

42. Slide 42 Chapter 17: Data Integration Practices and Relational DBMS Extensions Oracle Integration Tools  Oracle Warehouse Builder  Oracle Data Integrator  INSERT statement extensions  Change data capture  Data pump

43. Slide 43 Chapter 17: Data Integration Practices and Relational DBMS Extensions Oracle Warehouse Builder  Extraction, Transformation, and Load (ETL)  Data modeling  Data profiling and data quality  Metadata management  Business-level integration of ERP application data  Integration with Oracle business intelligence tools for reporting purposes  Advanced data lineage and impact analysis

44. Slide 44 Chapter 17: Data Integration Practices and Relational DBMS Extensions INSERT Statement Extensions  MERGE: the ability to update or insert a row conditionally into a table.  Conditions are specified in the ON clause  Combines insert and update statements  Multiple table INSERT  External data sources have to be segregated based on logical attributes for insertion into different target objects  Data can end up in several or exactly one target, depending on the business transformation rules

45. Slide 45 Chapter 17: Data Integration Practices and Relational DBMS Extensions Change Data Capture  Capture and publish committed change data  Synchronous using triggers  Asynchronous using log files  Publisher: a DBA who creates and maintains schema objects  Determines source databases and tables  Allows subscribers to have controlled access to change data  Subscriber: consumers of the published change data  Create subscriptions using subscriber views  Notify when ready to receive a set of change data

46. Slide 46 Chapter 17: Data Integration Practices and Relational DBMS Extensions Data Pump  Enables very high-speed movement of data and metadata from one database to another  Import: a utility for loading an export dump file set into a target system  Export: utility for unloading data and metadata into a set of operating system files called a dump file set

47. Slide 47 Chapter 17: Data Integration Practices and Relational DBMS Extensions Refresh Processing

48. Slide 48 Chapter 17: Data Integration Practices and Relational DBMS Extensions Determining the Refresh Frequency  Maximize net refresh benefit  Value of data timeliness  Cost of refresh  Satisfy data warehouse and source system constraints

49. Slide 49 Chapter 17: Data Integration Practices and Relational DBMS Extensions Refresh Constraints  Source access: restrictions on time and frequency  Integration: restrictions that require concurrent reconciliation  Completeness/consistency: loading in the same refresh period  Availability: load scheduling restrictions due to storage capacity, online availability, and server usage

50. Slide 50 Chapter 17: Data Integration Practices and Relational DBMS Extensions SQL Extensions  Review of GROUP BY clause  Subtotal operators  Analytic functions

51. Slide 51 Chapter 17: Data Integration Practices and Relational DBMS Extensions GROUP BY Review  Provides summary data not row data  One row per combination of grouping columns  Aggregate function: statistical function that returns one value for a set of values (min, max, average, sum, count, …)  Row pattern: grouping columns, aggregate functions  Syntax rule: all non aggregate columns in SELECT must be in GROUP BY  HAVING clause for conditions involving aggregate functions

52. Slide 52 Chapter 17: Data Integration Practices and Relational DBMS Extensions GROUP BY Example SELECT StoreZip, TimeMonth, SUM(SalesDollar) AS SumSales FROM Sales, Store, TimeDim WHERE Sales.StoreId = Store.StoreId AND Sales.TimeNo = TimeDim.TimeNo AND (StoreNation = 'USA' OR StoreNation = 'Canada') AND TimeYear = 2012 GROUP BY StoreZip, TimeMonth

53. Slide 53 Chapter 17: Data Integration Practices and Relational DBMS Extensions Motivation for Subtotal Extensions  Lack of subtotals in GROUP BY result  Show subtotals in a data cube  Provide control over subtotals in GROUP BY result  Provide a bridge between relational database representation and data cubes

54. Slide 54 Chapter 17: Data Integration Practices and Relational DBMS Extensions CUBE/GROUP BY Comparison State Month DecJanFebTotal CA100-75175 CO150100200450 CN5075-125 Total300175275750 SELECT State, Month, SUM(Sales) GROUP BY State, Month StateMonthSUM(Sales) CADec100 CAFeb75 CODec150 COJan100 COFeb200 CNDec50 CNJan75

55. Slide 55 Chapter 17: Data Integration Practices and Relational DBMS Extensions GROUP BY Extensions  ROLLUP operator  CUBE operator  GROUPING SETS operator  Other extensions  Ranking  Ratios  Moving summary values

56. Slide 56 Chapter 17: Data Integration Practices and Relational DBMS Extensions CUBE Operator  Complete set of subtotals  Appropriate for independent dimensions  Not order dependent: specify columns in any order

57. Slide 57 Chapter 17: Data Integration Practices and Relational DBMS Extensions CUBE Example SELECT StoreZip, TimeMonth, SUM(SalesDollar) AS SumSales FROM Sales, Store, Time WHERE Sales.StoreId = Store.StoreId AND Sales.TimeNo = Time.TimeNo AND (StoreNation = 'USA' OR StoreNation = 'Canada') AND TimeYear = 2012 GROUP BY CUBE (StoreZip, TimeMonth)

58. Slide 58 Chapter 17: Data Integration Practices and Relational DBMS Extensions CUBE/GROUP BY Comparison SELECT State, Month, SUM(Sales) GROUP BY State, Month StateMonthSUM(Sales) CADec100 CAFeb75 CODec150 COJan100 COFeb200 CNDec50 CNJan75 SELECT State, Month, SUM(Sales) GROUP BY CUBE(State, Month) StateMonthSUM(Sales) CADec100 CAFeb75 CODec150 COJan100 COFeb200 CNDec50 CNJan75 CA-175 CO-450 CN-125 - Dec 300 -Jan175 -Feb275 --750

59. Slide 59 Chapter 17: Data Integration Practices and Relational DBMS Extensions CUBE Operator Details  Two grouping columns  Maximum of M × N rows  Maximum subtotal rows: M + N + 1 (not in GROUP BY but in data cube)  3 additional SELECT statements with GROUP BY clauses: one SELECT statement per set of subtotals  Three grouping columns  Maximum of M × N × P rows  Maximum subtotal rows: M + N + P + M*N + M*P + N*P + 1  Number of additional SELECT blocks: 7

60. Slide 60 Chapter 17: Data Integration Practices and Relational DBMS Extensions SELECT Statements without CUBE SELECT StoreZip, TimeMonth, SUM(SalesDollar) AS SumSales … GROUP BY StoreZip, TimeMonth UNION SELECT StoreZip, 0, SUM(SalesDollar) AS SumSales … GROUP BY StoreZip UNION SELECT '', TimeMonth, SUM(SalesDollar) AS SumSales … GROUP BY TimeMonth UNION SELECT '', 0, SUM(SalesDollar) AS SumSales

61. Slide 61 Chapter 17: Data Integration Practices and Relational DBMS Extensions ROLLUP Operator  Appropriate for hierarchical dimensions  Partial set of subtotals  Order dependent: broad (general) to narrow (specific) order

62. Slide 62 Chapter 17: Data Integration Practices and Relational DBMS Extensions ROLLUP/GROUP BY Comparison SELECT State, Month, SUM(Sales) GROUP BY State, Month StateMonthSUM(Sales) CADec100 CAFeb75 CODec150 COJan100 COFeb200 CNDec50 CNJan75 SELECT State, Month, SUM(Sales) GROUP BY ROLLUP(State, Month) StateMonthSUM(Sales) CADec100 CAFeb75 CODec150 COJan100 COFeb200 CNDec50 CNJan75 CA-175 CO-450 CN-125 --750

63. Slide 63 Chapter 17: Data Integration Practices and Relational DBMS Extensions ROLLUP Example SELECT StoreZip, TimeYear, TimeMonth, SUM(SalesDollar) AS SumSales FROM Sales, Store, TimeDim WHERE Sales.StoreId = Store.StoreId AND Sales.TimeNo = TimeDim.TimeNo AND (StoreNation = 'USA' OR StoreNation = 'Canada') AND TimeYear BETWEEN 2012 AND 2013 GROUP BY ROLLUP (StoreZip, TimeYear, TimeMonth);

64. Slide 64 Chapter 17: Data Integration Practices and Relational DBMS Extensions ROLLUP Details  Two grouping columns  N values in outer most column  N + 1 subtotal rows  Three grouping columns  ROLLUP (Col1, Col2, Col3) where Col1 has M distinct values, Col2 has N distinct values, and Col3 has P distinct values  Maximum subtotal rows: M * N + M + 1  N additional SELECT clauses for N rollup columns

65. Slide 65 Chapter 17: Data Integration Practices and Relational DBMS Extensions SELECT Statements without ROLLUP SELECT StoreZip, TimeYear, TimeMonth, SUM(SalesDollar) AS SumSales … GROUP BY StoreZip, TimeYear, TimeMonth UNION SELECT StoreZip, TimeYear, 0, SUM(SalesDollar) AS SumSales … GROUP BY StoreZip, TimeYear UNION SELECT StoreZip, 0, 0, SUM(SalesDollar) AS SumSales … GROUP BY StoreZip UNION SELECT ‘’, 0, 0, SUM(SalesDollar) AS SumSale

66. Slide 66 Chapter 17: Data Integration Practices and Relational DBMS Extensions GROUPING SETS Operator  Precise control of subtotals  Explicit specification of columns in which totals are produced  Normal grouping columns must be specified if desired  Can produce subtotals only  Specification is similar to explicit specification of SELECT statements

67. Slide 67 Chapter 17: Data Integration Practices and Relational DBMS Extensions GROUPING SETS Example SELECT StoreZip, TimeMonth, SUM(SalesDollar) AS SumSales FROM Sales, Store, Time WHERE Sales.StoreId = Store.StoreId AND Sales.TimeNo = Time.TimeNo AND (StoreNation = 'USA' OR StoreNation = 'Canada') AND TimeYear = 2012 GROUP BY GROUPING SETS((StoreZip, TimeMonth), StoreZip, TimeMonth, ());

68. Slide 68 Chapter 17: Data Integration Practices and Relational DBMS Extensions ROLLUP/GROUPING SETS Comparison SELECT TimeYear, TimeMonth, SUM(Sales) … GROUP BY ROLLUP(TimeYear, TimeMonth) SELECT TimeMonth, TimeYear, Month, SUM(Sales) … GROUP BY GROUPING SETS ((TimeYear, TimeMonth), TimeYear, ()) SELECT TimeYear, TimeMonth, TimeDay, SUM(Sales) … GROUP BY ROLLUP(TimeYear, TimeMonth, TimeDay) SELECT TimeYear, TimeMonth, TimeDay, SUM(Sales) … GROUP BY GROUPING SETS ((TimeYear, TimeMonth, TimeDay), (TimeYear, TimeMonth), TimeYear, ())

69. Slide 69 Chapter 17: Data Integration Practices and Relational DBMS Extensions CUBE/GROUPING SETS Comparison SELECT State, Month, SUM(Sales) … GROUP BY CUBE(State, Month) SELECT State, Month, SUM(Sales) … GROUP BY GROUPING SETS ((State, Month), State, Month,()) -- (State, Month): normal GROUP BY result SELECT State, Month, Product, SUM(Sales) … GROUP BY CUBE(State, Month, Product) SELECT State, Month, Product, SUM(Sales) … GROUP BY GROUPING SETS ((State,Month,Product), (State,Month), (State,Product), (Month,Product), State, Month, Product,()) -- (State,Month,Product): normal GROUP BY result

70. Slide 70 Chapter 17: Data Integration Practices and Relational DBMS Extensions Variations of the Grouping Operators  Partial cube  Partial rollup  Composite columns  CUBE and ROLLUP inside a GROUPIING SETS operation

71. Slide 71 Chapter 17: Data Integration Practices and Relational DBMS Extensions Variation Examples  Partial CUBE  GROUP BY Month, CUBE(Product, State)  Generates totals on,,,  Partial ROLLUP  GROUP BY State, ROLLUP(Year, Month)  Generates totals on,,  Composite columns  GROUP BY ROLLUP(Nation, Region, (State, City))  Generates totals on,,, and <>.  Composite column (State, City) is treated as a single column.

72. Slide 72 Chapter 17: Data Integration Practices and Relational DBMS Extensions Materialized Views  Stored view  Periodically refreshed with source data  Usually contain summary data  Fast query response for summary data  Appropriate in query dominant environments

73. Slide 73 Chapter 17: Data Integration Practices and Relational DBMS Extensions SQL Analysis Functions  Ranking  Ratio  Moving totals and averages using the WINDOW clause  Oracle ranking functions  RANK  DENSE_RANK  CUME_DIST  PERCENT_RANK  NTILE

74. Slide 74 Chapter 17: Data Integration Practices and Relational DBMS Extensions Materialized View Example CREATE MATERIALIZED VIEW MV1 BUILD IMMEDIATE REFRESH COMPLETE ON DEMAND ENABLE QUERY REWRITE AS SELECT StoreState, TimeYear, SUM(SalesDollar) AS SUMDollar1 FROM Sales, Store, Time WHERE Sales.StoreId = Store.StoreId AND Sales.TimeNo = Time.TimeNo AND TimeYear > 2010 GROUP BY StoreState, TimeYear;

75. Slide 75 Chapter 17: Data Integration Practices and Relational DBMS Extensions Query Rewriting  Substitution process  Materialized view replaces references to fact and dimension tables in a query  Query optimizer must evaluate whether the substitution will improve performance over the original query  More complex than query modification process for traditional views

76. Slide 76 Chapter 17: Data Integration Practices and Relational DBMS Extensions Query Rewriting Process

77. Slide 77 Chapter 17: Data Integration Practices and Relational DBMS Extensions Query Modification Process

78. Slide 78 Chapter 17: Data Integration Practices and Relational DBMS Extensions Query Rewriting Principles  Row conditions: query conditions at least as restrictive as MV conditions  Grouping detail: query grouping columns at least as general as MV grouping columns  Grouping dependencies: query columns must match or be derivable by functional dependencies  Aggregate functions: query aggregate functions must match or be derivable from MV aggregate functions

79. Slide 79 Chapter 17: Data Integration Practices and Relational DBMS Extensions Query Rewriting Example -- Data warehouse query SELECT StoreState, TimeYear, SUM(SalesDollar) FROM Sales, Store, Time WHERE Sales.StoreId = Store.StoreId AND Sales.TimeNo = Time.TimeNo AND StoreNation IN ('USA','Canada') AND TimeYear = 2012 GROUP BY StoreState, TimeYear; -- Query Rewrite -- Replace Sales and Time tables with MV1 SELECT DISTINCT MV1.StoreState, TimeYear, SumDollar1 FROM MV1, Store WHERE MV1.StoreState = Store.StoreState AND TimeYear = 2012 AND StoreNation IN ('USA','Canada');

80. Slide 80 Chapter 17: Data Integration Practices and Relational DBMS Extensions Storage and Optimization Technologies  MOLAP: direct storage and manipulation of data cubes  ROLAP: relational extensions to support multidimensional data  HOLAP: combine MOLAP and ROLAP storage engines

81. Slide 81 Chapter 17: Data Integration Practices and Relational DBMS Extensions ROLAP Techniques  Bitmap join indexes  Star join optimization  Query rewriting  Summary storage advisors  Parallel query execution

82. Slide 82 Chapter 17: Data Integration Practices and Relational DBMS Extensions Summary  Maintaining a data warehouse is an important, operational problem.  Increasing usage of data integration tools  SQL extensions for subtotals and analysis functions  Substantial product extensions for efficient management of summary data and optimization techniques