Data Mining Classification: Basic Concepts,

Data Mining Classification: Basic Concepts,
Lecture Notes for Chapter 4 - 5 Introduction to Data Mining by Tan, Steinbach, Kumar

Classification: Definition
Given a collection of records (training set ) Each record contains a set of attributes, one of the attributes is the class. Find a model for class attribute as a function of the values of other attributes.

Classification: Definition
Goal: previously unseen records should be assigned a class as accurately as possible. A test set is used to determine the accuracy of the model. Usually, the given data set is divided into training and test sets, with training set used to build the model and test set used to validate it.

Illustrating Classification Task
Learning Algorithm Learn Model Model Apply Model

Examples of Classification Task
Predicting tumor cells as benign or malignant Classifying credit card transactions as legitimate or fraudulent Classifying spam mail Classifying user in a social network Categorizing news stories as finance, weather, entertainment, sports, etc

Classification Techniques
Decision Tree Naïve Bayes Instance Based Learning Rule-based Methods Neural Networks Bayesian Belief Networks Support Vector Machines

Classification: Measure the quality
Usually the Accuracy measure is used: Number of Correctly Classified record Accuracy = Total Number of record in the test set

Decision Tree Uses a tree structure to model the training set
Classifies a new record following the path in the tree Inner nodes represent attributes and leaves nodes represent the class

Example of a Decision Tree
categorical continuous class Splitting Attributes Refund Yes No NO MarSt Single, Divorced Married TaxInc NO < 80K > 80K NO YES Training Data Model: Decision Tree

Another Example of Decision Tree
categorical categorical continuous class MarSt Single, Divorced Married NO Refund No Yes NO TaxInc < 80K > 80K NO YES There could be more than one tree that fits the same data!

Decision Tree Classification Task
Learning Algorithm Learn Model Model Decision Tree Apply Model

Apply Model to Test Data
Start from the root of tree. Refund MarSt TaxInc YES NO Yes No Married Single, Divorced < 80K > 80K

Refund MarSt TaxInc YES NO Yes No Married Single, Divorced < 80K > 80K

Refund Yes No NO MarSt Single, Divorced Married TaxInc NO < 80K > 80K NO YES

Refund Yes No NO MarSt Married Assign Cheat to “No” Single, Divorced TaxInc NO < 80K > 80K NO YES

Decision Tree Classification Task
Learning Tree Algorithm Learn Model Model Decision Tree Apply Model

Decision Tree Induction
Many Algorithms: Hunt’s Algorithm (one of the earliest) CART ID3, C4.5 (J48 on WEKA) SLIQ,SPRINT

Tree Induction Greedy strategy. Issues
Split the records based on an attribute test that optimizes certain criterion. Issues Determine how to split the records How to specify the attribute test condition? How to determine the best split? Determine when to stop splitting

How to Specify Test Condition?
Depends on attribute types Nominal Ordinal Continuous Depends on number of ways to split 2-way split Multi-way split

Splitting Based on Nominal Attributes
Multi-way split: Use as many partitions as distinct values. Binary split: Divides values into two subsets Need to find optimal partitioning. Marital Status Single Married Divorced Marital Status {Married, Single} Divorced Marital Status {Divorced, Single} Married OR

Splitting Based on Ordinal Attributes
We can imagine an attribute SIZE defined over the ordered set {Small, Medium, Large} Multi-way split: Use as many partitions as distinct values. Size Small Medium Large

Splitting Based on Ordinal Attributes
Binary split: Divides values into two subsets. Need to find optimal partitioning. OR Size {Medium, Large} {Small} Size {Small, Medium} {Large} What about this split? Size {Small, Large} {Medium}

Splitting Based on Continuous Attributes
Different ways of handling Discretization to form an ordinal categorical attribute Static – discretize once at the beginning Dynamic – ranges can be found by equal interval bucketing, equal frequency bucketing (percentiles), or clustering. Binary Decision: (A < v) or (A  v) consider all possible splits and finds the best cut can be more compute intensive

Splitting Based on Continuous Attributes

Example Married Yes No 4 3 Single, Divorced MarSt Married
4 Single, Divorced 3

Example Married YES NO 3 1 2 MarSt Single, Divorced NO Refund No Yes
Refund = Yes 2

Example Married YES NO 1 3 MarSt Single, Divorced NO Refund No Yes NO
TaxInc >= 80K < 80K {NO:2, YES:0} YES NO Single, Divorced Refund = NO TaxInc = < 80k 1 >= 80k 3

Example Married MarSt Single, Divorced NO Refund No Yes NO TaxInc
>= 80K < 80K {NO:2, YES:0} NO YES {NO:1, YES:0} {NO:0, YES:3}

How to determine the Best Split
Greedy approach: Nodes with homogeneous class distribution are preferred Need a measure of node impurity: Non-homogeneous, High degree of impurity Homogeneous, Low degree of impurity

Measures of Node Impurity
Given a node t Gini Index Entropy Misclassification error

Stopping Criteria for Tree Induction
Stop expanding a node when all the records belong to the same class Stop expanding a node when all the records have similar attribute values

Decision Tree Based Classification
Advantages: Inexpensive to construct Extremely fast at classifying unknown records Easy to interpret for small-sized trees Accuracy is comparable to other classification techniques for many simple data sets

Naive Bayes Uses probability theory to model the training set
Assumes independence between attributes Produces a model for each class

Bayes Theorem Conditional Probability: Bayes theorem:

Example of Bayes Theorem
Given: A doctor knows that meningitis causes headache 50% of the time Prior probability of any patient having meningitis is 1/50,000 Prior probability of any patient having headache is 1/20 If a patient has headache, what’s the probability he/she has meningitis ? (M = meningitis, S = headache)

Bayesian Classifiers Consider each attribute and class label as random variables Given a record with attributes (A1, A2,…,An) Goal is to predict class C Specifically, we want to find the value of C that maximizes P(C| A1, A2,…,An ) Can we estimate P(C| A1, A2,…,An ) directly from data?

Bayesian Classifiers Approach:
compute the posterior probability P(C | A1, A2, …, An) for all values of C using the Bayes theorem Choose value of C that maximizes P(C | A1, A2, …, An) Equivalent to choosing value of C that maximizes How to estimate P(A1, A2, …, An | C )?

Naïve Bayes Classifier
Assume independence among attributes Ai when class is given: Can estimate P(Ai| Cj) for all Ai and Cj. New point is classified to Cj if is maximal.

How to Estimate Probabilities from Data?
Class: P(C) = Nc/N e.g., P(No) = 7/10, P(Yes) = 3/10 For discrete attributes: P(Ai | Ck) = |Aik|/ Nc where |Aik| is number of instances having attribute Ai and belongs to class Ck Examples: P(Status=Married|No) = 4/7 P(Refund=Yes|Yes)=0

For continuous attributes: Discretize the range into bins one ordinal attribute per bin violates independence assumption Two-way split: (A < v) or (A > v) choose only one of the two splits as new attribute Probability density estimation: Assume attribute follows a normal distribution Use data to estimate parameters of distribution (e.g., mean and standard deviation) Once probability distribution is known, can use it to estimate the conditional probability P(Ai|c) k

Naïve Bayes (Summary) Robust to isolated noise points
Model each class separately Robust to irrelevant attributes Use the whole set of attribute to perform classification Independence assumption may not hold for some attributes

Ensemble Learning Roberto Esposito and Dino Ienco dino
Ensemble Learning Roberto Esposito and Dino Ienco Acknowledgements Most of the material is based on Nicholaus Kushmerick’s slides. You can find his original slideshow at: Several pictures are taken from the slides by Thomas Dietterich. You can find his original slideshow (see slides about Bias/Variance theory) at:

Agenda 1. What is ensemble learning 2. (Ensemble) Classification:
2.1. Bagging 2.2. Boosting 2.3 Why do Ensemble Classifiers Work? 3. (Ensemble) Clustering: Cluster-based Similarity Partitioning Algorithm (CSPA) HyperGraph-Partitioning Algorithm (HGPA) Some hints on how to build base clusters

part 1. What is ensemble learning?
Ensemble learning refers to a collection of methods that learn a target function by training a number of individual learners and combining their predictions [Freund & Schapire, 1995]

Ensemble learning different training sets and/or learning algorithms

How to make an effective ensemble?
Two basic decisions when designing ensembles: How to generate the base model? h1, h2, … How to integrate/combine them? F(h1(x), h2(x), …)

Ensemble Classification
Ensemble of Classifier: How to generate a classifier in an ensemble schema (which is the training set)? How to combine the vote of each classifier? Why do ensemble Ensemble Classifiers do?

Question 2: How to integrate them
Usually take a weighted vote: ensemble(x) = f( i wi hi(x) ) wi is the “weight” of hypothesis hi wi > wj means “hi is more reliable than hj” typically wi>0 (though could have wi<0 meaning “hi is more often wrong than right”) (Fancier schemes are possible but uncommon)

Question 1: How to generate base classifiers
Lots of approaches… A. Bagging B. Boosting …

BAGGing = Bootstrap AGGregation
(Breiman, 1996) for i = 1, 2, …, K: Ti  randomly select M training instances with replacement hi  learn(Ti) [ID3, NB, kNN, neural net, …] Now combine the hi together with uniform voting (wi=1/K for all i)

decision tree learning algorithm; along the lines of ID3

shades of blue/red indicate strength of vote for particular classification

Boosting Bagging was one simple way to generate ensemble members with trivial (uniform) vote weighting Boosting is another…. “Boost” as in “give a hand up to” suppose A can learn a hypothesis that is better than rolling a dice – but perhaps only a tiny bit better Theorem: Boosting A yields an ensemble with arbitrarily low error on the training data! ensemble error rate 50% error rate of flipping a coin 49% error rate of A by itself … size of ensemble time

Boosting Idea: assign a weight to every training set instance
initially, all instances have the same weight as boosting proceedgs, adjusts weights based on how well we have predicted data points so far - data points correctly predicted low weight - data points mispredicted  high weight Results: as learning proceeds, the learner is forced to focus on portions of data space not previously well predicted

Time=0 blue/red = class size of dot = weight hypothesis =
horizontal or vertical line

Time=1 The WL error is 30% The ensemble error is 30% (note T=1)

Time=3

Time=7

Time=21 Notice the slope of the weak learner error: AdaBoost creates problems of increasing difficulty.

Time=51 Look, the training error is zero. One could think that we cannot improve the test error any more.

Time=57 But… the test error still decreases!

Time=110

AdaBoost (Freund and Schapire)
binary class y  {0,1} = 1/N normalize wt to get a probability distribution pt I pti = 1 [0,1] penalize mistakes on high-weight instances more if ht correctly classify xi multiply weight by t < 1 otherwise multiple weight by 1 weighted vote, with wt = log(1/ t)

Learning from weighted instances?
One piece of the puzzle missing… So far, learning algorithms have just taken as input a set of equally important learning instances. [0,1] Reweighting What if we also get a weight vector saying how important each instance is? Turns out.. it’s very easy to modify most learning algorithms to deal with weighted instances: ID3: Easy to modify entropy, information-gain equations to take into consideration the weights associated to the examples, rather than to take into account only the count (which simply assumes all weights=1) Naïve Bayes: ditto k-NN: multiple vote from an instance by its weight

Resampling As an alternative to modify learning algorithms to support weighted datasets, we can build a new dataset which is not weighted but it shows the same properties of the weighted one. Let L’ be the empty set Let (w1,..., wn) be the weights of examples in L sorted in some fixed order (we assume wi corresponds to example xi) Draw n[0..1] according to U(0,1) set L’L’{xk} where k is such that if enough examples have been drawn return L’ else go to 3

How many examples are “enough”? The higher the number, the better L’ approximate a dataset following the distribution induced by W. As a rule of thumb: |L’|=|L| usually works reasonably well. Why don’t we always use resampling instead of reweighting? Resampling can be always applied, unfortunately it requires more resources and produces less accurate results. One should use this technique only when it is too costly (or unfeasible) to use reweighting.

Why do ensemble classifiers work?
1 3 2 [T. G. Dietterich. Ensemble methods in machine learning. Lecture Notes in Computer Science, 1857:1–15, 2000.]

1. Statistical Given a finite amount of data, many hypothesis are typically equally good. How can the learning algorithm select among them? Optimal Bayes classifier recipe: take a weighted majority vote of all hypotheses weighted by their posterior probability. That is, put most weight on hypotheses consistent with the data. Hence, ensemble learning may be viewed as an approximation of the Optimal Bayes rule (which is provably the best possible classifier).

2. Representational The desired target function may not be implementable with individual classifiers, but may be approximated by ensemble averaging Suppose you want to build a decision boundary with decision trees The decision boundaries of decision trees are hyperplanes parallel to the coordinate axes. By averaging a large number of such “staircases”, the diagonal decision boundary can be approximated with arbitrarily good accuracy

3. Computational All learning algorithms do some sort of search through some space of hypotheses to find one that is “good enough” for the given training data Since interesting hypothesis spaces are huge/infinite, heuristic search is essential (eg ID3 does greedy search in space of possible decision trees) So the learner might get stuck in a local minimum One strategy for avoiding local minima: repeat the search many times with random restarts  bagging

Reading Dietterich: Ensemble methods in machine learning (2000).
Schapire: A brief introduction to boosting (1999). [Sec 1-2, 5-6] Dietterich & Bakiri: Solving multiclass learning problems via error-correcting output codes (1995). [Skim]

Summary… Ensembles Classifiers: basic motivation – creating a committee of experts is more effective than trying to derive a single super-genius Key issues: Generation of base models Integration of base models Popular ensemble techniques manipulate training data: bagging and boosting (ensemble of “experts”, each specializing on different portions of the instance space) Current Research: Ensemble pruning (reduce the number of classifiers, select only non redundant and informative ones) versus

Ensemble Clustering Ensemble of Clustering (Partition):
How to formulate the problem? How to define how far (or similar) are two different clustering solutions? How to combine different partitions? (CSPA, How to generate different paritions?

Example We have a dataset of 7 examples and we run 3 clustering algorithm and we obtain 3 different clustering solution: C1 = (1,1,1,2,2,3,3) C2 = (1,1,2,2,3,3,3) C3 = (2,2,2,3,3,1,1) Each clustering solution is represented as a vector with as many componenet as the number of original example (7) and associate to each component we have the label of corresponding clustering. How much information is shared among the different partitions? How we combine them?

How to formulate the problem
GOAL: Seek a final clustering that shares the most information with the original clusterings. Find a partition that share as much information as possible with all individual clustering results If we have N clustering results we want obtain a solution such that:

How to formulate the problem
Where Ci is a clustering solution and Copt is the optimal solution so the problem to the clustering ensemble problem and phi is a measure able to evaluate the similarity between 2 clustering results.

How to define similarity between partitions
How we can define the function We use the Normalized Mutual Information (original paper: Cluster Ensembles – A Knowledge Reuse Framework for Combining Multiple Partitions, Alexander Strehl and Joydeep Ghosh) Normalized version of Mutual Information Mutual Information usually involved to evalute correlation between two random variables

How to combine different partitions
How to combine different partitions? Cluster-based Similarity Partitioning Algorithm (CSPA) Simple approach: Given an ensemble of partitions: - if M is the number of objects, build a M x M matrix - each cell of the matrix contains how many times 2 objects co-occur together in a cluster - the matrix can be seen as object-object similarity matrix - re-cluster the matrix with a clustering algorithm (in the original paper they apply METIS, graph-based clustering approach, to re-cluster the similarity matrix

Example (I) Given the different partition of the dataset:
C1 = (1,1,1,2,2,3,3) C2 = (1,1,2,2,3,3,3) C3 = (2,2,2,3,3,1,1) We obtain this co-occurrence matrix: 3 2 1

Example (II) We obtain this co-occurrence matrix: 3 2 1
1 We obtain this graph of co-occurrence: X2 2 X4 X6 1 X3 1 3 2 3 2 X1 2 1 X5 X7

How to combine different partitions
How to combine different partitions? HyperGraph-Partitioning Algorithm (HGPA) Hypergraph-based approach: Given an ensemble of partitions: - each partition is seen as an edge in the hypergraph - each object is a node - all nodes and hyperdeges have the same weight - try to eliminate hyperedges to obtain K unconnected partitions with approximately the same size - Apply standard techniques to partition hypergraph

Hypergraph An hypergraph is defined as: a set of nodes
a set of hyperedges over nodes Each edge is not a relation between 2 nodes, but it is a relation among more than 1 nodes Hyperedge: is defined as set of nodes in some relations each other

Example Given the different partition of the dataset:
C1 = (1,1,1,2,2,3,3) C2 = (1,1,2,2,3,3,3) C3 = (2,2,2,3,3,1,1) The dataset is composed by 7 examples We obtain this hypergraph: X2 X3 X4 X6 X1 X5 X7

How to generate different paritions (I)
Ways to generate different partitions: 1) Using the same clustering algorithm: a) fix the dataset and change the parameter values in order to obtain different results. b) Produce as many random projection of the original dataset as the number of partitions needed and then use the clustering algorithm to obtain a partition over each projected dataset.

How to generate different paritions (II)
Ways to generate different partitions: 2) Using different clustering algorithms: a) fix the dataset and run as many clustering algorithms as you need. If the number of algorithms available is smaller than the number of partition needed, re-run the same algorithms with different parameters. b) Produce some random projections of the dataset and then apply the different clustering algorithms.

Reading Strehl and Ghosh: Cluster Ensembles – A Knowledge Reuse Framework for Combining Multiple Partitions (2002). Al-Razgan and Domeniconi: Weighted Clustering Ensembles (2006). Fern and Lin: Cluster Ensemble Selection (2008).

Summary… Ensembles Clustering: basic motivation – No single clustering algorithm can adequately handle all types of cluster shapes and structures Key issues: How to choose an appropriate algorithm? How to interpret different partitions produced by different clustering algorithms? More difficult than Ensemble Classifiers Popular ensemble techniques Combine partitions obtained in different way using similarity between partitions or co-occurrence Current Research: Cluster Selection (reduce the number of clusters, select only non redundant and informative ones) versus

Data Mining Classification: Basic Concepts,

Similar presentations

Presentation on theme: "Data Mining Classification: Basic Concepts,"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Data Mining Classification: Basic Concepts,

Similar presentations

Presentation on theme: "Data Mining Classification: Basic Concepts,"— Presentation transcript:

Similar presentations

About project

Feedback