1. Data Mining Concepts 1.1 COT5230 Data Mining Week 1 Data Mining Concepts M O N A S H A U S T R A L I A ’ S I N T E R N A T I O N A L U N I V E R S I T Y

2. Data Mining Concepts 1.2 A Definition of Data Mining Use of analytical tools to discover knowledge in a collection of data The knowledge takes the form of patterns, relationships and facts which would not otherwise be immediately apparent These analytical tools may be drawn from a number of disciplines, which include: »machine learning »pattern recognition »machine discovery »statistics »artificial intelligence »human-computer interaction »information visualization

3. Data Mining Concepts 1.3 Data Mining Why has the area appeared? –Large volumes of data stored by organizations in a competitive environment combined with advances in technologies which can be applied to the data Background and evolution –The failure of traditional approaches The need for Data Mining –Niche marketing, customer retention, the internet The means to implement Data Mining –The data warehouse, the available computing power, effective modeling approaches

4. Data Mining Concepts 1.4 A Case Study - Data Preparation (Cabena et al. page 106) Health Insurance Commission Australia –550Gb online; 1300Gb in 5 year history DB –Aim to prevent fraud and inappropriate practice –Considered 6.8 million visits requesting up to 20 pathology tests and 17,000 doctors –Descriptive variables were added to the GP records –Records were pivoted to create separate records for each pathology test –Records were then aggregated by provider number (GP) –An association discovery operation was carried out

5. Data Mining Concepts 1.5 An Association Rule The Rule –When a customer buys a shirt, in 70% of cases, he or she will also buy a tie –The Confidence Factor is 70% The Support Factor –This occurs in 13.5% of all purchases –The Support Factor is 13.5%

6. Data Mining Concepts 1.6 A Case Study - Modeling and Analysis –Rules with a confidence factor greater than 50% were considered –The software Intelligent Miner (IBM) was used –The level of support was gradually reduced »i.e. the number of records to which the rule applied was reduced –Rules considered to be noise were excluded. –Domain knowledge indicated that some tests should be excluded and more useful rules were revealed –GP profiling was carried out –The new segments were related back to existing classifications of GPs –Some rules corresponded to expensive tests that could be substituted

7. Data Mining Concepts 1.7 Episodes DatabaseGP Database Rules 1% support If test A then test B will occur in 62% of cases Segment 1 Segment 2 97 GPs 206 GPs Score = 1.8 Score = 2.7 Data Preparation Merge Association DiscoveryDatabase Segmentation

8. Data Mining Concepts 1.8 Data Mining for Business Decision Support (From Berry & Linoff 1997) Identify the business problem Use data mining techniques to transform the data into actionable information Act on information Measure the results

9. Data Mining Concepts 1.9 The Process of Knowledge Discovery Pre-processing –data selection –cleaning –coding Data Mining –select a model –apply the model Analysis of results and assimilation –Take action and measure the results

10. Data Mining Concepts 1.10 The Process of Knowledge Discovery DataCleaning & Enrichment CodingData mining Reporting selection -domain consistency - clustering - segmentation -de-duplication - prediction -disambiguation Information Requirement Action Feedback Operational dataExternal data The Knowledge Discovery in Databases (KDD) process (Adriens/Zantinge)

11. Data Mining Concepts 1.11 Data Selection Identify the relevant data, both internal and external to the organization Select the subset of the data appropriate for the particular data mining application Store the data in a database separate from the operational systems

12. Data Mining Concepts 1.12 Data Preprocessing Cleaning –Domain consistency: replace certain values with null –De-duplication: customers are often added to the DB on each purchase transaction –Disambiguation: highlighting ambiguities for a decision by the user »e.g. if names differed slightly but addresses were the same Enrichment –Additional fields are added to records from external sources which may be vital in establishing relationships. Coding »e.g. take addresses and replace them with regional codes »e.g. transform birth dates into age ranges –It is often necessary to convert continuous data into range data for categorization purposes.

13. Data Mining Concepts 1.13 Data Mining Preliminary Analysis –Much interesting information can be found by querying the data set –May be supported by a visualization of the data set. Choose a one or more modeling approaches There are two styles of data mining –Hypothesis testing –Knowledge discovery The styles and approaches are not mutually exclusive

14. Data Mining Concepts 1.14 Data Mining Tasks Various taxonomies exist. Berry & Linoff define 6 tasks: »Classification »Estimation »Prediction »Affinity Grouping »Clustering »Description The tasks are also referred to as operations. Cabena et al define 4 operations: »Predictive Modeling »Database Segmentation »Link Analysis »Deviation Detection

15. Data Mining Concepts 1.15 Classification Classification involves considering the features of some object then assigning it it to some pre- defined class, for example: –Spotting fraudulent insurance claims –Which phone numbers are fax numbers –Which customers are high-value

16. Data Mining Concepts 1.16 Estimation Estimation deals with numerically valued outcomes rather than discrete categories as occurs in classification. –Estimating the number of children in a family –Estimating family income

17. Data Mining Concepts 1.17 Prediction Essentially the same as classification and estimation but involves future behaviour Historical data is used to build a model explaining behaviour (outputs) for known inputs The model developed is then applied to current inputs to predict future outputs –Predict which customers will respond to a promotion –Classifying loan applications

18. Data Mining Concepts 1.18 Affinity Grouping Affinity grouping is also referred to as Market Basket Analysis A common example is which items are bought together at the supermarket. Once this is known, decisions can be made on, for example: –how to arrange items on the shelves –which items should be promoted together

19. Data Mining Concepts 1.19 Clustering Clustering is also sometimes referred to as segmentation (though this has other meanings in other fields) In clustering there are no pre-defined classes. Self-similarity is used to group records. The user must attach meaning to the clusters formed Clustering often precedes some other data mining task, for example: –once customers are separated into clusters, a promotion might be carried out based on market basket analysis of the resulting cluster

20. Data Mining Concepts 1.20 Description A good description of data can provide understanding of behaviour The description of the behaviour can suggest an explanation for it as well Statistical measures can be useful in describing data, as can techniques that generate rules

21. Data Mining Concepts 1.21 Deviation Detection Records whose attributes deviate from the norm by significant amounts are also called outliers Application areas include: –fraud detection –quality control –tracing defects. Visualization techniques and statistical techniques are useful in finding outliers A cluster which contains only a few records may in fact represent outliers

22. Data Mining Concepts 1.22 Data Mining Techniques –Query tools –Decision Trees –Memory-Based Reasoning –Artificial Neural Networks –Genetic Algorithms –Association and sequence detection –Statistical Techniques –Visualization –Others (Logistic regression,Generalized Additive Models (GAM), Multivariate Adaptive Regression Splines (MARS), K Means Clustering,...)

23. Data Mining Concepts 1.23 Data Mining and the Data Warehouse Organizations realized that they had large amounts of data stored (especially of transactions) but it was not easily accessible The data warehouse provides a convenient data source for data mining. Some data cleaning has usually occurred. It exists independently of the operational systems –Data is retrieved rather than updated –Indexed for efficient retrieval –Data will often cover 5 to 10 years A data warehouse is not a pre-requisite for data mining

24. Data Mining Concepts 1.24 Data Mining and OLAP Online Analytic Processing (OLAP) Tools that allow a powerful and efficient representation of the data Makes use of a representation known as a cube A cube can be sliced and diced OLAP provide reporting with aggregation and summary information but does not reveal patterns, which is the purpose of data mining