1. Data Mining Lecture 2

2. Course Syllabus Course topics: Introduction (Week1-Week2) –What is Data Mining? –Data Collection and Data Management Fundamentals –The Essentials of Learning –The Emerging Needs for Different Data Analysis Perspectives Data Management and Data Collection Techniques for Data Mining Applications (Week3-Week4) –Data Warehouses: Gathering Raw Data from Relational Databases and transforming into Information. –Information Extraction and Data Processing Techniques –Data Marts: The need for building highly specialized data storages for data mining applications

3. Data: – raw – atomic –(mostly!) operational Information: –processed –re-organized –grouped Knowledge –patterns, models, findings ‘behind’ Information Wisdom –perfect orchestration of Knowledge Data Data (Operation) Information (Analytic) Knowledge Wisdom “Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?” T. S. Eliot Week 2- Data vs. Knowledge

4. Week 2- Evolution of Database and Information Systems 1960s: (focus on efficient data collection) Data collection, database creation, IMS and network DBMS 1970s: (focus on structured data collection) Relational data model, relational DBMS implementation 1980s: (focus on information extraction) RDBMS, advanced data models (extended- relational, OO, deductive, etc.) and application-oriented DBMS (spatial, scientific, engineering, etc.) 1990s – 2000s: (focus on knowledge extraction and modeling) Data Mining, Data Warehousing, Multi Dimensional Databases

5. Week 2- Data Collection and Data Management Fundamentals – What is Data Warehouse “A data warehouse is a subject-oriented, integrated, time-variant, and nonvolatile collection of data in support of management’s decision making process” William H. Inmon Subject-oriented: A data warehouse is organized around major subjects, such as customer,supplier, product, and sales.Rather than concentrating on the day-to-day operations and transaction processing of an organization, a data warehouse focuses on the modeling and analysis of data for decision makers

6. Week 2- Data Collection and Data Management Fundamentals – What is Data Warehouse “A data warehouse is a subject-oriented, integrated, time-variant, and nonvolatile collection of data in support of management’s decision making process” William H. Inmon Integrated: A data warehouse is usually constructed by integrating multiple Heterogeneous sources, such as relational databases, flat files, and on-line transaction records. Data cleaning and data integration techniques are applied to ensure consistency in naming conventions, encoding structures, attribute measures, and so on.

7. Week 2- Data Collection and Data Management Fundamentals – What is Data Warehouse “A data warehouse is a subject-oriented, integrated, time-variant, and nonvolatile collection of data in support of management’s decision making process” William H. Inmon Time-variant: Data are stored to provide information from a historical perspective (e.g., the past 5–10 years). Every key structure in the data warehouse contains, either implicitly or explicitly, an element of time. Nonvolatile: A data warehouse is always a physically separate store of data transformed from the application data found in the operational environment. Due to this separation, a data warehouse does not require transaction processing, recovery, and concurrency control mechanisms. It usually requires only two operations in data accessing: initial loading of data and access of data.

8. data cleaning data integration data consolidation Week 2- Data Collection and Data Management Fundamentals – What is Data Warehouse

9. object oriented methodology comes in entities (cubes) attributes (dimensions) Week 2- Data Collection and Data Management Fundamentals – What is OLAP

10. taken from the Text Book

11. Multi Dimensional Database Modeling –star schema –snowflake schema –fact constellation schema fact vs dimension Week 2- Data Collection and Data Management Fundamentals – What is OLAP

12. taken from the Text Book

13. Week 2- Data Collection and Data Management Fundamentals – What is OLAP taken from the Text Book

14. Week 2- Data Collection and Data Management Fundamentals – What is OLAP taken from the Text Book

15. Week 2- Data Collection and Data Management Fundamentals – OLAP Operations taken from the Text Book roll-up drill-down slice dice pivot (rotation)

16. Week 2- Data Collection and Data Management Fundamentals – OLAP Operations

17. Week 2- Data Collection and Data Management Fundamentals – What is Data Mart ? data warehouse information about subjects that span the entire organization, its scope is enterprise-wide. which modeling schema ? the fact constellation schema is commonly used, since it can model multiple, interrelated subjects. data mart a department subset of the data warehouse that focuses on selected subjects, its scope is departmentwide. which modeling schema ? the star or snowflake schema are commonly used, since both are geared toward modeling single subjects

18. Week2-OLAP vs Data Mining On-Line Analytical Processing provides the ability to pose statistical and summary queries interactively (traditional On-Line Transaction Processing (OLTP) databases may take minutes or even hours to answer these queries) Advantages relative to data mining Can obtain a wider variety of results Generally faster to obtain results Disadvantages relative to data mining User must “ask the right question” Generally used to determine high-level statistical summaries, rather than specific relationships among instances

19. Week2-Reporting vs Data Mining Reporting Last months sales for each service type Sales per service grouped by customer sex or age bracket List of customers who lapsed their policy Data Mining What characteristics do customers that lapse their policy have in common and how do they differ from customers who renew their policy? Which motor insurance policy holders would be potential customers for my House Content Insurance policy?

20. Week2- Data to Knowledge Pyramid Increasing potential to support business decisions End User Business Analyst Data Analyst DBA Making Decisions Data Presentation Visualization Techniques Data Mining Information Discovery Data Exploration OLAP, MDA Statistical Analysis, Querying and Reporting Data Warehouses / Data Marts Data Sources Paper, Files, Information Providers, Database Systems, OLTP

21. Week 2- Data Mining Perspective to Knowledge Discovery adapted from: U. Fayyad, et al. (1995), “From Knowledge Discovery to Data Mining: An Overview,” Advances in Knowledge Discovery and Data Mining, U. Fayyad et al. (Eds.), AAAI/MIT Press Data Target Data Selection Knowledge Preprocessed Data Patterns Data Mining Interpretation/ Evaluation Preprocessing

22. Week2- Data Mining Process Flow Background Knowledge Goals for LearningKnowledge BaseDatabase(s) Plan for Learning Discover Knowledge Determine Knowledge Relevancy Evolve Knowledge/ Data Generate and Test Hypotheses Visualization and Human Computer Interaction Discovery Algorithms “In order to discover anything, you must be looking for something” Laws of Serendipity

23. Week2-Simplified view of Data Mining Process Flow Data Warehouse Data cleaning & data integration Filtering Databases Database or data warehouse server Data mining engine Pattern evaluation Graphical user interface Knowledge-base

24. Week 2- Extended Perspective on Data Mining Process Flow Data Warehouse Meta Data MDDB OLAM Engine OLAP Engine User GUI API Data Cube API Database API Data cleaning Data integration Layer3 OLAP/OLAM Layer2 MDDB Layer1 Data Repository Layer4 User Interface Filtering&IntegrationFiltering Databases Mining queryMining result

25. Week 2- Essentials of Learning Learning ? can we formalize it? is it just a chemical activation? is it memorization? is it continous node connecting/disconnecting on dynamically changing brain network topology?

26. Week 2- Essentials of Learning The Artifical Intelligence View: central to human knowledge and intelligence, essential for building intelligent machines. years of effort in AI has shown that trying to build intelligent computers by programming all the rules cannot be done; automatic learning is crucial. For example, we humans are not born with the ability to understand language — we learn it — and it makes sense to try to have computers learn language instead of trying to program it all it

27. Week 2- Essentials of Learning The Software Engineering View: Machine Learning allows us to program computers by example, which can be easier than writing code the traditional way. The Stats View: Machine Learning is the marriage of computer science and statistics computational techniques are applied to statistical problems. Machine Learning has been applied to a vast number of problems in many contexts, beyond the typical statistics problems. Machine Learning is often designed with different considerations than statistics (e.g., speed is often more important than accuracy).

28. Week 2-End Please check the web site for Learning Theory and its Esssentials: http://www.infed.org/biblio/b-learn.htm read –Course Text Book Chapter 3