1. CPSC-608 Database Systems Fall 2011 Instructor: Jianer Chen Office: HRBB 315C Phone: 845-4259 Email: chen@cse.tamu.edu Notes #15

2. 2 Brief Overview on Data/information integration (data warehouse) Data mining

3. 3 Data Warehouse (Overview) A data warehouse is the main repository of an organization's historical data, its corporate memory. It contains the raw material for management's decision support system. The critical factor leading to the use of a data warehouse is that a data analyst can perform complex queries and analysis, such as data mining, on the information without slowing down the operational systems. [Wikipedia]

4. What is a Warehouse? Collection of (possibly diverse) data – subject oriented – aimed at executive, decision maker, analysts – often a copy of operational data – with value-added data (e.g., summaries, history) – integrated schema – time-varying – non-volatile 4

5. What is a Warehouse? Collection of tools/services – gathering data – cleansing, integrating,... – querying, reporting, aggregation, analysis – data mining – monitoring, administration 5

6. Why a Warehouse? Ship and integrate data from different sources to the analyst Three Approaches: – Database federations (legacy) – Query-driven (lazy) – Warehouse (eager) 6 6

7. Database Federations 7 An application program for each connection, Simple, good if DB communications are limited Needs to write many application programs

8. Warehouse Architecture 8 Each source has a wrapper/extractor that consists of a collection of predefined queries on the source, and communication mechanisms

9. Query-Driven Approach 9 Each source has a wrapper, which classifies queries into templates, and translates them into queries for the source. The wrapper can be generated from templates using modern compiler techniques.

10. Advantages of Query-Driven No need to copy data – less storage – no need to purchase data More up-to-date data Query needs can be unknown Only query interface needed at sources May be less draining on sources 10

11. Advantages of Warehousing High query performance Queries not visible outside warehouse Local processing at sources unaffected Can operate when sources unavailable Can query data not stored in a DBMS Extra information at warehouse – Modify, summarize (store aggregates) – Add historical information 11

12. OLTP vs. OLAP OLTP: On Line Transaction Processing – Describes processing at operational sites (sources) OLAP: On Line Analytical Processing – Describes processing at warehouse 12

13. OLTP vs. OLAP Mostly updates Many small transactions Megabyte-terabyte of data Raw data Up-to-date data Consistency, recoverability critical Clerical users Mostly reads Queries long, typically complex aggregations Gigabyte-terabyte of data Summarized, consolidated data Decision-makers, analysts as users 13 OLTP OLAP

14. Implementing a Warehouse Monitoring: Sending data from sources Integrating: Data loading, cleaning,... Processing: Query processing, indexing,... Managing: Metadata, Design,... 14

15. Monitoring Issues Frequency – periodic: daily, weekly, … – triggered: on “big” change, lots of changes,... Data transformation/normalization – convert data to uniform format – remove & add fields (e.g., add date to get history) Standards Gateways (Intranet/internet, firewalls, VPN, etc.) 15

16. Integration Data Cleaning Data Loading Derived Data 16

17. Processing Index Structures What to Materialize? Algorithms 17 Client Warehouse Source Query & Analysis Integration Metadata

18. Managing Metadata Warehouse Design Tools 18 Client Warehouse Source Query & Analysis Integration Metadata

19. Warehouse Design What data is needed? Where does it come from? How to clean data? How to represent in warehouse (schema)? What to summarize? What to materialize? What to index? 19

20. Conclusions Massive amounts of data and complexity of queries will push limits of current warehouses Need better systems: – easier to use – provide quality information – scalability CS 245Notes12 20

21. Data Mining (Overview) What is data mining? A process of examining data and finding simple rules or models that summarize the data. Mining Techniques: Decision Trees Clustering Association Rules 21

22. Decision Trees 22 Example: Conducted survey to see what customers were interested in new model car Want to select customers for advertising campaign training set training set

23. One Possibility 23 car=taurus city=sfage<45 likely unlikely YY Y N N N

24. Another Possibility 24 age<30 city=sfcar=van likely unlikely YY Y N N N

25. Issues Decision tree should not be “too deep” – would not have statistically significant amounts of data for lower decisions Need to select tree that most reliably predicts outcomes – automatic decision tree construction from training data (“unsupervised learning”) – exploit training data statistics to detect most ”discriminative” attribute/value conditions at each level 25

26. Clustering 26 age income education

27. Another Example: Text Each document is a vector Clusters contain “similar” documents Useful for understanding, searching documents 27 international news sports business

28. Issues Given desired number of clusters? Finding “best” clusters Are clusters semantically meaningful? Using clusters for disk storage 28

29. Association Rule Mining 29 transaction id customer id products bought sales records: Trend 1) Products p5, p8 often bought together Trend 2) Customer 12 likes product p9 market-basket data market-basket data

30. Association Rule Rule: {p 5, p 8 }, {cust 12, p 9 }, … Support: number of “baskets” where these products appear High-support set: support  threshold s Problem: find all high support sets 30

31. Association Rules How do we perform rule mining efficiently? Observation: – If set X has support t, then each X subset must have at least support t For 2-sets: – if we need support s for {i, j} – then each i, j must appear in at least s baskets A-Priori Algorithm 31

32. 32 CSCE-608 Course Summary Overview of DB and DBMS systems; The memory architecture; Indexing and hashing; Query processing; Crash recovery; Concurrency control; Transaction processing; Data integrity and data mining;

33. 33 CSCE-608 Course Summary Overview of DB and DBMS systems; The memory architecture; Indexing and hashing; Query processing; Crash recovery; Concurrency control; Transaction processing; Data integrity and data mining;

34. 34 Indexing and Hashing B+ trees structure operations: search, insert, delete Hashing hash table and hash function operations: search, insert, delete extensible hashing linear hashing

35. 35 Query Processing Query compiler, parse tree Logic query plan, physical query plan Disk I/O efficient algorithms Cost estimation of query plans

36. 36 Crash Recovery Undo logging Redo logging Undo/redo logging Recovery algorithms Checkpoints

37. 37 Concurrent Control Serialization Locking systems Timestamp Validation

38. 38 Transaction processing Recoverability Handling deadlocks