1. Data Warehousing and Data Mining
Prepared By: Amit Patel

2. Decision Tree A DECISION TREE is a flowchart-like tree structure,
Where each internal node (nonleaf node) denotes a test on an attribute,
Each branch represents an outcome of the test,
And each leaf node (or terminal node) holds a class label.
The topmost node in a tree is the root node.

3. Decision Tree

4. Decision Tree
Algorithm: Generate_decision_tree. Generate a decision tree from the training tuples of data partition D.
Input:
Data partition, D, which is a set of training tuples and their associated class labels;
Attribute_list, the set of candidate attributes;
Attribute_selection_method, a procedure to determine the splitting criterion that “best” partitions the data tuples into individual classes. This criterion consists of a splitting attribute and, possibly, either a split point or splitting subset.
Output: A decision tree.

5. Decision Tree

6. Let A be the splitting attribute
Let A be the splitting attribute. (a) If A is discrete-valued, then one branch is grown for each known value of A. (b) If A is continuous-valued, then two branches are grown, corresponding to A < split point and A > split point. (c) If A is discrete-valued and a binary tree must be produced, then the test is of the form A 2 SA, where SA is the splitting subset for A.

7. Let A be the splitting attribute
Let A be the splitting attribute. (a) If A is discrete-valued, then one branch is
grown for each known value of A. (b) If A is continuous-valued, then two branches are
grown, corresponding to A split point and A > split point. (c) If A is discrete-valued
and a binary tree must be produced, then the test is of the form A 2 SA, where SA is the
splitting subset for A.
where

8. Naïve Bayesian Classification P(Ci|X)=[P(X|Ci).P(Ci)]/P(X)
Procedure:
Assume that there are a set of m samples S={s1, s2,…,sm} (the training data set).
Where every sample si is represented as an n-dimensional vector (x1,x2,…,xn)
Values xi correspond to attributes A1, A2,…, An respectively.
Also, there are k classes C1, C2,…, Ck, and every sample belongs to one of these classes.
These probabilities are computed using Bayes’ Theorem:
P(Ci|X)=[P(X|Ci).P(Ci)]/P(X)
As P(X) is constant for all classes, only the product P(X|Ci).P(Ci) needs to be maximized.

9. Naïve Bayesian Classification

10. Naïve Bayesian Classification

11. Naïve Bayesian Classification
The data tuples are described by the attributes age, income, student, and credit rating. The class label attribute, buys_computer, has two distinct values namely, {yes, no}.
Let Class:
C1: buys_computer= “yes”
C2: buys_computer= “no”
The data sample we want to classify is:
X=(age=youth, income=medium, student=yes, credit rating=fair)

12. Naïve Bayesian Classification
We need to maximize P(X|Ci).P(Ci), for i = 1, 2. P(Ci), the prior probability of each class, can be computed based on the training tuples:
P(buys computer = yes)
P(buys computer = no)
To compute P(X|Ci), for i = 1, 2, we compute the following conditional probabilities:
= 9/14 = 0.643
= 5/14 = 0.357

13. Naïve Bayesian Classification
P(age = youth | buys_computer = yes)
P(age = youth | buys_computer = no)
P(income = medium | buys_computer = yes)
P(income = medium | buys_computer = no)
P(student = yes | buys_computer = yes)
P(student = yes | buys_computer = no)
P(credit_rating = fair | buys_computer = yes)
P(credit_rating = fair | buys_computer = no)
X=(age=youth, income=medium, student=yes, credit_rating=fair)
= 2/9 = 0.222
= 3/5 = 0.600
= 4/9 = 0.444
= 2/5 = 0.400
= 6/9 = 0.667
= 1/5 = 0.200
= 6/9 = 0.667
= 2/5 = 0.400

14. Naïve Bayesian Classification
Using the above probabilities, we obtain
P(X|buys_computer = yes) =
P(age = youth | buys_computer = yes) *
P(income = medium | buys_computer = yes) *
P(student = yes | buys_computer = yes) *
P(credit_rating = fair | buys_computer = yes)
= * * * = 0.044

15. Naïve Bayesian Classification
Similarly,
P(X|buys_computer = no) = * * * = 0.019
To find the class, Ci, that maximizes P(X|Ci).P(Ci), we compute
P(X|buys_computer = yes) * P(buys_computer = yes)
= 0:0440 * = 0.028
P(X|buys_computer = no) * P(buys_computer = no)
= * = 0.007
Therefore, the naïve Bayesian classifier predicts buys_computer = yes for tuple X.

16. Rule-Based Classification
Using IF-THEN Rules for Classification
Rules are a good way of representing information or bits of knowledge. A rule-based classifier uses a set of IF-THEN rules for classification. An IF-THEN rule is an expression of the form
IF condition THEN conclusion.
An example is rule R1,
R1: IF age = youth AND student = yes THEN
buys computer = yes.

17. Rule-Based Classification
The “IF”-part (or left-hand side)of a rule is known as the rule antecedent or precondition.
The “THEN”-part (or right-hand side) is the rule consequent.
In the rule antecedent, the condition consists of one or more attribute tests (such as age = youth, and student = yes) that are logically ANDed.
The rule’s consequent contains a class prediction (in this case, we are predicting whether a customer will buy a computer).
R1 can also be written as
R1: (age = youth)^(student = yes)  (buys computer = yes).

18. Rule-Based Classification
If the condition (that is, all of the attribute tests) in a rule antecedent holds true for a given tuple, we say that the rule antecedent is satisfied (or simply, that the rule is satisfied) and that the rule covers the tuple.
A rule R can be assessed by its coverage and accuracy.
Given a tuple, X, from a class labeled data set, D,
let ncovers be the number of tuples covered by R;
ncorrect be the number of tuples correctly classified by R; and
|D| be the number of tuples in D.

19. Rule-Based Classification
We can define the coverage and accuracy of R as:
Find the Coverage and Accuracy for the Rule R1: IF age = youth AND student = yes THEN buys_computer = yes ???

20. Rule-Based Classification

21. Rule-Based Classification
These are class-labeled tuples from the AllElectronics customer database.
Our task is to predict whether a customer will buy a computer.
Consider rule R1 above, which covers 2 of the 14 tuples. It can correctly classify both tuples.
Therefore,
coverage(R1)
accuracy (R1)
= 2/14 = 14.28%
= 2/2 = 100%

22. Rule-Based Classification
R2: IF age = youth AND student = no THEN buys_computer = no
coverage(R2) =
accuracy (R2) =
R3: IF age = senior AND credit_rating = excellent THEN buys_computer = yes
coverage(R3) =
accuracy (R3) =
R4: IF age = middle_aged THEN buys_computer = yes
coverage(R4) =
accuracy (R4) =
3/14 = 21.43%
3/3 = 100%
2/14 = 14.28%
0/2 = 0%
4/14 = 28.57%
4/4 = 100%

23. Basic Sequential Covering algorithm
IF-THEN rules can be extracted directly from the training data (i.e., without having to generate a decision tree first) using a sequential covering algorithm.
Input:
D, a data set class-labeled tuples;
Att_vals, the set of all attributes and their possible values.

24. Rule-Based Classification

25. CART Classification and Regression Trees
University of California- a study into patients after admission for a heart attack.
19 variables collected during the first 24 hours for patients (for those who survived the 24 hours)
Question: Can the high risk (will not survive 30 days) patients be identified?

26. CART
Is the minimum systolic blood pressure over the 1st 24 hours>91?
Is age>62.5? H
Is sinus present? L H L

27. CART Plan for construction of a CART Selection of the Splits
Decisions when to decide that a node is a terminal node (i.e. not to split it any further)
Assigning a class to each terminal node

28. CART Where splits need to be evaluated…??? Sorted by Age
Sorted by Blood Pressure X X

29. CART Features of CART Binary Splits Splits based only on one variable

30. CART Advantages of CART Can cope with any data structure or type
Classification has a simple form
Uses conditional information effectively
Invariant under transformations of the variables
Is robust with respect to outliers
Gives an estimate of the misclassification rate

31. CART Disadvantages of CART CART does not use combinations of variables
Tree can be deceptive – if variable not included it could be as it was “masked” by another
Tree structures may be unstable – a change in the sample may give different trees
Tree is optimal at each split – it may not be globally optimal.

32. Neural Networks
Neural networks are useful for data mining and decision- support applications.
People are good at generalizing from experience.
Computers excel at following explicit instructions over and over.
Neural networks bridge this gap by modeling, on a computer, the neural behavior of human brains.

33. Neural Networks
Neural Networks map a set of input-nodes to a set of output-nodes
Number of inputs/outputs is variable
The Network itself is composed of an arbitrary number of nodes with an arbitrary topology

34. Neural Networks A neuron: many-inputs / one-output unit
Output can be excited or not excited
Incoming signals from other neurons determine if the neuron shall excite ("fire")
Output subject to attenuation in the synapses, which are junction parts of the neuron

35. Neural Networks Neural Networks
A multilayer feed-forward neural network.

36. Prediction (Numerical) prediction is similar to classification
construct a model
use model to predict continuous or ordered value for a given input
Prediction is different from classification
Classification refers to predict categorical class label
Prediction models continuous-valued functions

37. Prediction Major method for prediction: regression
Model the relationship between one or more independent or predictor variables and a dependent or response variable
Regression analysis
Linear and multiple regression
Non-linear regression
Other regression methods: generalized linear model, Poisson regression, log-linear models, regression trees

38. Linear Regression
Linear regression: involves a response variable y and a single predictor variable x
y = w0 + w1 x
where w0 (y-intercept) and w1 (slope) are regression coefficients
Method of least squares: estimates the best-fitting straight line

39. Linear Regression
Multiple linear regression: involves more than one predictor variable
Training data is of the form (X1, y1), (X2, y2),…, (X|D|, y|D|)
Ex. For 2-D data, we may have: y = w0 + w1 x1+ w2 x2
Many nonlinear functions can be transformed into the above

40. Nonlinear Regression
Some nonlinear models can be modeled by a polynomial function
A polynomial regression model can be transformed into linear regression model. For example,
y = w0 + w1 x + w2 x2 + w3 x3
convertible to linear with new variables: x2 = x2, x3= x3
Other functions, such as power function, can also be transformed to linear model
Some models are intractable nonlinear (e.g., sum of exponential terms)
possible to obtain least square estimates through extensive calculation on more complex formulae

41. Information Gain
ID3 uses information gain as its attribute selection measure.
Let node N represent or hold the tuples of partition D. The attribute with the highest information gain is chosen as the splitting attribute for node N.
This attribute minimizes the information needed to classify the tuples in the resulting partitions and reflects the least randomness or “impurity” in these partitions.

42. Information Gain (cont…)
InfoA(D) is the expected information required to classify a tuple from D based on the partitioning by A.
Information gain is defined as the difference between the original information requirement (i.e., based on just the proportion of classes) and the new requirement (i.e., obtained after partitioning on A).

43. Information Gain (cont…)
Examples
Over all Information for a database

44. Information Gain (cont…)
Hence, the gain in information from such a partitioning would be

45. Gain Ratio
Consider an attribute that acts as a unique identifier, such as product_ID. A split on product_ID would result in a large number of partitions (as many as there are values), each one containing just one tuple.
Because each partition is pure, the information required to classify data set D based on this partitioning would be Infoproduct_ID(D) = 0.
Therefore, the information gained by partitioning on this attribute is maximal. Clearly, such a partitioning is useless for classification.

46. Gain Ratio (cont…)
This value represents the potential information generated by splitting the training data set, D, into v partitions, corresponding to the v outcomes of a test on attribute A. Note that, for each outcome, it considers the number of tuples having that outcome with respect to the total number of tuples in D. It differs from information gain, which measures the information with respect to classification that is acquired based on the same partitioning. The gain ratio is defined as

47. Cluster Analysis Cluster: a collection of data objects
Similar to one another within the same cluster
Dissimilar to the objects in other clusters
Cluster analysis
Finding similarities between data according to the characteristics found in the data and grouping similar data objects into clusters
Unsupervised learning: no predefined classes
Typical applications
As a stand-alone tool to get insight into data distribution
As a preprocessing step for other algorithms

48. Clustering: Rich Application
Pattern Recognition
Spatial Data Analysis
Create thematic maps in GIS by clustering feature spaces
Detect spatial clusters or for other spatial mining tasks
Image Processing
Economic Science (especially market research)
WWW
Document classification
Cluster Weblog data to discover groups of similar access patterns

49. Requirements of Clustering in DM
Scalability
Ability to deal with different types of attributes
Ability to handle dynamic data
Discovery of clusters with arbitrary shape
Minimal requirements for domain knowledge to determine input parameters
Able to deal with noise and outliers
Insensitive to order of input records
High dimensionality
Incorporation of user-specified constraints
Interpretability and usability

50. End of the Unit-5