Presentation is loading. Please wait.

Presentation is loading. Please wait.

Data Mining with Analysis Services

Similar presentations

Presentation on theme: "Data Mining with Analysis Services"— Presentation transcript:

1 Data Mining with Analysis Services
Carlos Bossy Principal Consultant MCTS, MCITP BI Aabcom Solutions

2 My Background Experience Current Projects
25 years Software Development 15 years software companies as Programmer, Architect, Manager, Director, VP, CTO 5+ years Business Intelligence Consultant Current Projects Data Warehouse deployment for State of WA Child Welfare Data Integration/Warehouse architecture for Solar Energy company Data Warehouse model for State of OR Data Mining model for Houston sports league

3 Make Data Mining an integral component of your Data Architecture.
Overview Make Data Mining an integral component of your Data Architecture. Applications Reports Data Integration

4 Session Overview What is Data Mining
Data Mining Algorithms in SQL Server Creating a Data Mining Model Using the Model in an Application Using the Model in SSIS Awareness of Data Mining Architecture and Process

5 Application Architecture
Service Business Data Relational Database

6 Application Architecture with BI
Service Business Data Data Mining Relational Database Data Integration Data Warehouse Cube Text Warehouse Performance Management Reports Analysis Data Mining Text Mining Ad-hoc

7 Why Mine Data? If you want to… Data Mining is for you!
Predict the future Get rich buying stocks Win the lottery Win your Fantasy Football League Data Mining is for you!

8 Really - Why Mine Data? Maximize value of Data Competitive Edge
Uncover Hidden Relationships Find something unusual or unexpected Improve upon domain expert’s knowledge Manage large data sets Create Predictive Analytics platform Maximize value of Data Competitive Edge

9 Data Mining Defined Data Mining is the process of sorting through large amounts of data and picking out relevant information (Wikipedia). Data Mining is the extraction of hidden predictive information from large databases. The Process of Knowledge Discovery Sophisticated Statistical Model Discovery of Patterns and Relationships It is not Dredging, Snooping or an Invasion of Privacy

10 Today vs. Yesterday Legacy Future Manageable Volumes of Data
The power of SQL Domain Experts could grasp and analyze a complete Database Limited CPU Horsepower Finite Explosion of Data doubles every 3 years (Moore’s Law) Data Volumes can’t be comprehended by humans Uncover complex and difficult to find patterns for competitive edge Improve professional judgment of Domain Expert (small but valuable) Knowledge Discovery Converting Data to Information N  Infinity

11 Outcomes Predict number of runs Colorado Rockies would score in their next game using Neural Network. Input Previous 12 games runs scored Home/Away Predict Number runs next game Model Home Runs = (G7*.142)+(G1*.118)+… Away Runs = (G7*.129)+(G1*.091)+…

12 Non-linear Results Fitting a line to a set of data points to measure the effect of a single independent variable. Statistical Methods y = mx + b Data Mining

13 Outcomes – Decision Tree
Annual Income = 108, *(Savings Balance-27, )-2, *(Avg Check Size )-0.247*(Credit Card Limit-8, ) *(Age )-1, *(Over Drafts-1.000)

14 Outcomes – Time Series Total Gen-1 < and Flow Date < 3/2/2007 4:58:07 AM and Flow Date < 8/22/2006 5:54:22 PM and Flow Date < 8/2/2006 1:35:37 PM Total Gen = * Total Gen(-1) Total Gen-1 < and Flow Date >= 3/2/2007 4:58:07 AM and Flow Date < 5/16/2007 7:46:52 PM and Flow Date >= 3/15/ :56:15 PM Total Gen = * Total Gen(-2) * Total Gen(-1)

15 Applications Credit Risk Analysis Churn Analysis Customer Retention
Targeted Marketing Market Basket Analysis Sales Forecasting Stock Predictions Medical Diagnosis Bioscience Research Surveys Insurance Rate Quotes Credit Card Fraud Web Site Events Loan Applications Hiring and Recruiting Cross-Marketing Social Science Economics

16 Data Mining Sources Data Mining DW Table SSAS Cube OLTP Database Col 1
SQL Server Oracle MySQL Etc. OLTP Database Data Mining Table Col 1 Col 2 Col 3

17 Training the Model Cleanse Massage Select Training Set
Apply DM Algorithm Train Test

18 Using the Model Customer CustomerID CustName Mining Model
CheckingBalance SavingsBalance NumTransactions DMX Prediction Join SQL Join MonthlyBankAccount BankAccountID CheckingBalance SavingsBalance NumTransactions CustomerID Result Prediction Income

19 Prediction Join DMX SELECT Predict([Movie Purchases],3) as Movies From
[Customer Movie Association] NATURAL PREDICTION JOIN (SELECT 44 AS [Age], 'Female' AS [Gender], 'Rent' AS [Home Ownership], 'Divorced' AS [Marital Status], (SELECT 'Mrs. Doubtfire' AS [Movie Title] UNION SELECT 'My Big Fat Greek Wedding' AS [Movie Title] UNION SELECT 'Patriot Games' AS [Movie Title]) AS [Movie Purchases]) AS t

20 Market Basket Analysis
Approaches Clustering Classification Regression Market Basket Analysis

21 Mining Algorithms Time Series Naïve Bayes Association Clustering
Decision Trees Logistic Regression Sequence Clustering Neural Networks

22 Data Mining Algorithms
Analytical problem Examples Algorithms Classification: Assign cases to predefined classes Credit risk analysis Churn analysis Customer retention Decision Trees Naive Bayes Neural Nets Segmentation: Taxonomy for grouping similar cases Customer profile analysis Mailing campaign Clustering Sequence Clustering Association: Advanced counting for correlations Market basket analysis Advanced data exploration Association Time Series Forecasting: Predict the future Forecast sales Predict stock prices Time Series Prediction: Predict a value for a new case based on values for similar cases Quote insurance rates Predict customer income All Deviation analysis: Discover how a case or segment differs from others Credit card fraud detection Network infusion analysis * Andy Cheung, Microsoft

23 Time Series Analyzes how a variable changes over time.
Uses Autoregression + Decision Tree to build model Each time series is a single case No prediction join with test or actual cases Prediction is always the same for given time slots

24 Node Regression Formula
Time Series Tree All Stock(t-5) <= 22.83 Stock(t-5) > 22.83 Stock(t-1) <= 30.15 Stock(t-1) > 30.15 Node Regression Formula Stock = *Stock(t-1) + .21*Stock(t-2)

25 Naïve Bayes Probabilistic classifier based on Bayes’ theorem with strong (naive) independence assumptions. Simple Classification Algorithm Good starting point for better understanding of your data Uses only discrete data

26 Premium Svc = Yes Ring Tones = Yes International = No
Naïve Bayes Example Cell Phone Service Gender Premium Service Custom Ring Tones International Calls Female 53% 19% 56% 27% Male 47% 41% 14% 38% Premium Svc = Yes Ring Tones = Yes International = No Likelihood of Female = .53 * .19 * .56 * .73 = .0412 Likelihood of Male = .47 * .41 * .14 * .62 = .0167 P(Female) = .0412/( ) = 71.2% P(Male) = .0167/( ) = 28.8%

27 Association Rules Detect relationships or associations between specific values of categorical variables in large data sets. Uses only Discrete Data Rule - Attribute value conditions that occur frequently together in a given dataset {Male, IT, Star Wars}  {Star Trek} Itemset - A set of attribute values. Support - Total number of transactions. Confidence - Probability that {X}  {Y} Importance (Lift) –Interestingness. Measure of whether Correlation is positive, negative or none.

28 Logistic Regression Predict the probability of a discrete outcome from a set of variables that are continuous, discrete or both. Non-linear regression model that produces results between 0 and 1. Popular in health science for disease prediction. Marketing uses for dichotomous predictions (buy or not buy, renew or cancel). Same as Neural Network without the hidden layer. Probability = 1/1 + e-z where z = c + yx1 + yx2 + … If x1 = Weight and x2 = Age then Risk of heart attack = 1/1 + e-z where z = x1 - .7x2

29 Decision Tree Graphical representation displaying options, risks and the decision-making sequence. Most popular data mining model. East to visualize because of it’s graphical representation. Branches represent choices with associated risks, costs, results, or probabilities. Each test examines the value of a single column in the data and uses it to determine the next test to apply. The results of all tests determine which label to predict. Similar to human thought process when making a decision. Finds non-linear relationships. Supports classification, regression and association within the model.

30 Neural Network Classifies large and complex data sets by grouping cases together in a way loosely based on the brain. Most sophisticated algorithm but difficult to interpret. Works well with non-linear data and finds smooth non-linear relationships. Modeled as a group of interconnected nodes. No agreed upon definition. Microsoft algorithm is one of many techniques. Can build multiple models based on discrete inputs. H I O H Back to Input layer after weights adjusted for error I H O I H

31 Clustering Places data elements into related groups without advance knowledge of the group definitions. Good starting point for better understanding of your data. Finds the hidden variable that accurately classifies data. Data grouped into clusters have a high similarity based on the attribute values.

32 Sequence Clustering Discovers the properties of sequences by grouping them into clusters and assigning them to one of the clusters. Hybrid of sequence and clustering techniques. Typically used with web and event logs as data sources.

33 Demo

34 Conclusion Slow Adoption Where do you start? Science + Art
Not quite A.I. … yet! More Info and References TDWI – The Data Warehousing Institute ACM, IEEE Books: Data Mining with SQL Server 2005/8 (Wiley) Mining the Talk (IBM Press) Data Mining know it all (Morgan/Kaufman)

Download ppt "Data Mining with Analysis Services"

Similar presentations

Ads by Google