1. Data Warehouse Fundamentals Rabie A. Ramadan, PhD 2

2. 2 What did you do in Your Assignment ? For an airlines company, how can strategic information increase the number of frequent flyers? Discuss giving specific details. You are a Senior Analyst in the IT department of a company manufacturing automobile parts. The marketing heads are complaining about the poor response by IT in providing strategic information. Draft a proposal to them explaining the reasons for the problems and why a data warehouse would be the only viable solution.

3. 3 What did you do in the Project ? Egypt Election System Governorates’ database system Multiple databases on Multiple Servers Summarization System Meta data Data Warehouse Server Web page with query based system

4. 4 http://www.inf.unibz.it/dis/teaching/DWDM/index.html

5. 5 Definitions & Motivations Why Data Mining? Explosive Growth of Data: from terabytes to petabytes Data Collections and Data Availability Crawlers, database systems, Web, etc. Sources Business: Web, e-commerce, transactions, etc. Science: Remote sensing, bioinformatics, etc. Society and everyone: news, YouTube, etc.

6. 6 Why Data Mining? Problem: We are drowning in data, but starving for knowledge! Solution: Use Data Mining tools for Automated Analysis of massive data sets

7. 7 What is Data Mining? Data mining (knowledge discovery from data) Extraction of interesting (non-trivial, implicit, previously unknown and potentially useful) patterns or knowledge from huge amount of data

8. 8 What is Data Mining? Alternative names Knowledge discovery (mining) in databases (KDD), knowledge extraction, data/pattern analysis, data archeology, Data dredging, information harvesting, business intelligence, etc.

9. 9 Knowledge Discovery (KDD) Process

10. 10 Knowledge Discovery (KDD) Process

11. 11 Typical Architecture of a Data Mining System

12. 12 Confluence of Multiple Disciplines

13. 13 Why Confluence of Multiple Disciplines? Tremendous amount of data Scalable algorithms to handle terabytes of data (e.g., Flickr had 5 billion images in September, 2010 [http://blog.flickr.net/en/2010/09/19/5000000000/]) High dimensionality of data Data can have tens of thousands of features (e,g., DNA microarray)

14. 14 Why Confluence of Multiple Disciplines?

15. 15 Different Views of Data Mining Data View Kinds of data to be mined Knowledge view Kinds of knowledge to be discovered Method view Kinds of techniques utilized Application view Kinds of applications

16. 16 Data to Mined In principle, data mining should be applicable to any data repository We will have examples about: Relational databases Data warehouses Transactional databases Advanced database systems

17. 17 Relational Databases

18. 18 Data Warehouses

19. 19 Transactional Databases

20. 20 Advanced Database Systems(1)

21. 21 Advanced Database Systems(2)

22. 22 Knowledge to be Discovered

23. 23 Characterization and Discrimination

24. 24 Characterization and Discrimination (1)

25. 25 Class Activity Differentiate between Data Mining and Data warehousing? Data warehousing is merely extracting data from different sources, cleaning the data and storing it in the warehouse. Where as data mining aims to examine or explore the data using queries What are the Different problems that “Data mining” can solve? Data mining can be used in a variety of fields/industries like marketing, advertising of goods, products, services, AI, government intelligence. How does the data mining and data warehousing work together? Data warehousing can be used for analyzing the business needs by storing data in a meaningful form. Using Data mining, one can forecast the business needs. Data warehouse can act as a source of this forecasting.

26. 26 Frequent Patterns, Associations, Correlations

27. 27 Classification and Prediction

28. 28 Cluster Analysis

29. 29 Outlier Analysis

30. 30 Evolution Analysis

31. 31 Techniques Utilized

32. 32 Applications Adapted

33. 33 Major Challenges in Data Mining

34. 34 Summary