1. M. Sulaiman Khan (mskhan@liv.ac.uk)‏ Dept. of Computer Science University of Liverpool 2009 This is the full course notes, but not quite complete. You should come to the lectures anyway. Really. COMP527: Data Mining Data Warehousing February 04, 2009 Slide 1 COMP527: Data Mining

2. Introduction to the Course Introduction to Data Mining Introduction to Text Mining General Data Mining Issues Data Warehousing Classification: Challenges, Basics Classification: Rules Classification: Trees Classification: Trees 2 Classification: Bayes Classification: Neural Networks Classification: SVM Classification: Evaluation Classification: Evaluation 2 Regression, Prediction COMP527: Data Mining Data Warehousing February 04, 2009 Slide 2 COMP527: Data Mining Input Preprocessing Attribute Selection Association Rule Mining ARM: A Priori and Data Structures ARM: Improvements ARM: Advanced Techniques Clustering: Challenges, Basics Clustering: Improvements Clustering: Advanced Algorithms Hybrid Approaches Graph Mining, Web Mining Text Mining: Challenges, Basics Text Mining: Text-as-Data Text Mining: Text-as-Language Revision for Exam

3. Data Warehouses Data Cubes Warehouse Schemas OLAP Materialisation Today's Topics Data Warehousing February 04, 2009 Slide 3 COMP527: Data Mining

4. Most common definition: “A data warehouse is a subject-oriented, integrated, time-variant and nonvolatile collection of data in support of management's decision-making process.” - W. H. Inmon  Corporate focused, assumes a lot of data, and typically sales related  Data for “Decision Support System” or “Management Support System”  1996 survey: Return on Investment of 400+% Data Warehousing: Process of constructing (and using) a data warehouse What is a Data Warehouse? Data Warehousing February 04, 2009 Slide 4 COMP527: Data Mining

5.  Subject-oriented: Focused on important subjects, not transactions Concise view with only useful data for decision making  Integrated: Constructed from multiple, heterogeneous data sources. Normally distributed relational databases, not necessarily same schema. Cleaning, pre-processing techniques applied for missing data, noisy data, inconsistent data (sounds familiar, I hope)‏ Data Warehouse Data Warehousing February 04, 2009 Slide 5 COMP527: Data Mining

6.  Time-variant: Has different values for the same fields over time. Operational database only has current value. Data Warehouse offers historical values.  Nonvolatile: Physically separate store Updates not online, but in offline batch mode only Read only access required, so no concurrency issues Data Warehouse Data Warehousing February 04, 2009 Slide 6 COMP527: Data Mining

7. Data Warehouses are distinct from: Distributed DB: Integrated via wrappers/mediators. Far too slow, semantic integration much more complicated. Integration done before loading, not at run time. Operational DB: Only records current value, lots of extra non useful information such as HR. Different schemas/models, access patterns, users, functions, even though the data is derived from an operational db. Data Warehouse Data Warehousing February 04, 2009 Slide 7 COMP527: Data Mining

8. OLAP: Online Analytical Processing (Data Warehouse)‏ OLTP: Online Transaction Processing (Traditional DBMS)‏ OLAP data typically: historical, consolidated, and multi- dimensional (eg: product, time, location). Involves lots of full database scans, across terabytes or more of data. Typically aggregation and summarisation functions. Distinctly different uses to OLTP on the operational database. OLAP vs OLTP Data Warehousing February 04, 2009 Slide 8 COMP527: Data Mining

9. Data is normally Multi-Dimensional, and can be thought of as a cube. Often: 3 dimensions of time, location and product. No need to have just 3 dimensions -- could have one for cars with make, colour, price, location, and time for example. Image courtesy of IBM OLAP Miner documentation Data Cubes Data Warehousing February 04, 2009 Slide 9 COMP527: Data Mining

10.  Can construct many 'cuboids' from the full cube by excluding dimensions.  In an N dimensional data cube, the cuboid with N dimensions is the 'base cuboid'. A 0 dimensional cuboid (other than non existent!) is called the 'apex cuboid'.  Can think of this as a lattice of cuboids... (Following lattice courtesy of Han & Kamber) Data Cubes Data Warehousing February 04, 2009 Slide 10 COMP527: Data Mining

11. Lattice of Cuboids Data Warehousing February 04, 2009 Slide 11 COMP527: Data Mining all timeitemlocationsupplier time,itemtime,location time,supplier item,location item,supplier location,supplier time,item,location time,item,supplier time,location,supplier item,location,supplier time, item, location, supplier 0-D(apex) cuboid 1-D cuboids 2-D cuboids 3-D cuboids 4-D(base) cuboid

12. Each dimension can also be thought of in terms of different units.  Time: decade, year, quarter, month, day, hour (and week, which isn't strictly hierarchical with the others!)‏  Location: continent, country, state, city, store  Product: electronics, computer, laptop, dell, inspiron This is called a “Star-Net” model in data warehousing, and allows for various operations on the dimensions and the resulting cuboids. Multi-dimensional Units Data Warehousing January 18, 2008 Slide 12 COMP527: Data Mining

13. Star-Net Model Data Warehousing February 04, 2009 Slide 13 COMP527: Data Mining Shipping Method AIR-EXPRESS TRUCK ORDER Customer Orders CONTRACTS Customer Product PRODUCT GROUP PRODUCT LINE PRODUCT ITEM SALES PERSON DISTRICT DIVISION OrganizationPromotion DISTRICT REGION COUNTRY Geography DAILY QTRLY ANNUALY Time

14.  Roll Up: Summarise data by climbing up hierarchy. Eg. From monthly to quarterly, from Liverpool to England  Drill Down: Opposite of Roll Up Eg. From computer to laptop, from £100-199 to £100-999  Slice: Remove a dimension by setting a value for it Eg. location/product where time is Q1,2007  Dice: Restrict cube by setting values for multiple dimensions Eg. Q1,Q2 / North American cities / 3 products sub cube  Pivot: Rotate the cube (mostly for visualisation) Data Cube Operations Data Warehousing January 18, 2008 Slide 14 COMP527: Data Mining

15.  Star Schema: Single fact table in the middle, with connected set of dimension tables (Hence a star)‏  Snowflake Schema: Some of the dimension tables further refined into smaller dimension tables (Hence looks like a snow flake)‏  Fact Constellation: Multiple fact tables can share dimension tables (Hence looks like a collection of star schemas. Also called Galaxy Schema)‏ Data Cube Schemas Data Warehousing February 04, 2009 Slide 15 COMP527: Data Mining

16. Star Schema Data Warehousing January 18, 2008 Slide 16 COMP527: Data Mining Sales Fact Table time_key item_key location_key units_sold Time Dimension time_key day day_of_week month quarter year Item Dimension item_key name brand type supplier_type Loc.n Dimension location_key street city state country continent Measure (value)‏

17. Snowflake Schema Data Warehousing February 04, 2009 Slide 17 COMP527: Data Mining Sales Fact Table time_key item_key location_key units_sold Time Dimension time_key day day_of_week month quarter year Item Dimension item_key name brand type supplier_key Loc Dimension location_key street city_key Measure (value)‏ City Dimension city_key city state country

18. Fact Constellation Data Warehousing February 04, 2009 Slide 18 COMP527: Data Mining Sales Fact Table time_key item_key location_key units_sold Time Dimension time_key day day_of_week month quarter year Item Dimension item_key name brand type supplier_key Loc Dimension location_key street city_key Measure (value)‏ City Dimension city_key city state country Shipping Table time_key item_key from_key units_shipped

19. ROLAP: Relational OLAP Uses relational DBMS to store and manage the warehouse data Optimised for non traditional access patterns Lots of research into RDBMS to make use of! MOLAP: Multidimensional OLAP Sparse array based storage engine Fast access to precomputed data HOLAP: Hybrid OLAP Mixture of both MOLAP and ROLAP OLAP Server Types Data Warehousing February 04, 2009 Slide 19 COMP527: Data Mining

20. Data Warehouse Architecture Data Warehousing February 04, 2009 Slide 20 COMP527: Data Mining Data Warehouse Extract Transform Load Refresh OLAP Engine Analysis Query Reports Data mining Monitor & Integrator Metadata Data Sources Front-End Tools Serve Data Marts Operational DBs Other sources Data Storage OLAP Server (also courtesy of Han & Kamber)‏

21. In order to compute OLAP queries efficiently, need to materialise some of the cuboids from the data. None: Very slow, as need to compute entire cube at run time Full: Very fast, but requires a LOT of storage space and time to compute all possible cuboids Partial: But which ones to materialise? Called an 'iceberg cube', as only partially materialised and the rest is "below water". Many cells in a cuboid will be empty, only materialise sections that contain more values than a minimum threshold. Materialisation Data Warehousing February 04, 2009 Slide 21 COMP527: Data Mining

22. Han, Chapters 3,4 Dunham Sections 2.1, 2.6, 2.7 Berry and Linoff, Chapter 15 Inmon, Building the Data Warehouse Inmon, Managing the Data Warehouse http://en.wikipedia.org/wiki/Data_warehouse and subsequent links http://en.wikipedia.org/wiki/Data_warehouse Further Reading Data Warehousing February 04, 2009 Slide 22 COMP527: Data Mining