1. Cost-Sensitive Learning
Prepared with Lei Tang
9/18/2018
CSE 572: Data Mining by H. Liu

2. Cost-Sensitive Learning
Motivation
Data with different misclassification costs.
Objective: Minimize the total misclassification cost.
Application
Medical diagnosis
Fraud Detection
Spam filtering
Intrusion detection ……
9/18/2018
CSE 572: Data Mining by H. Liu

3. CSE 572: Data Mining by H. Liu
Toy Example
A Decision Tree(T1) based on Accuracy:
Body Heat
Tumor
Ill
abnormal
yes no
normal
Tumor
yes no
Ill
Not Ill
Two prediction errors
9/18/2018
CSE 572: Data Mining by H. Liu

4. CSE 572: Data Mining by H. Liu
Toy Example(cont’d)
Misclassification costs for false positive and
false negative are 1 and 100 respectively.
Then misclassification cost for T1:
1 *1+100*1=101
T2 is another tree with
higher error-rate, but lower
misclassification cost.
Errors : 3
Cost: 1*3=3
Tumor
yes no
ill
Heat
RDBMS - relational database management systems
RDBMS offer simple operators for the deduction of information, such as join
abnormal
normal
ill
Not ill
T2 based on cost
9/18/2018
CSE 572: Data Mining by H. Liu

5. CSE 572: Data Mining by H. Liu
Cost matrix
Cost matrix of the toy example:
(Similar to confusion matrix)
Quiz: Dose the absolute value matter?
How to get cost matrix?
--User defined or
--based on the class distribution
Actual Pred
ill
Not ill
Ill
100 1
Induction is different from deduction and DBMS does not not support induction;
The result of induction is higher-level information or knowledge: general statements about data
There are many approaches. Refer to the lecture notes for CS3244 available at the Co-Op.
We focus on three approaches here, other examples:
Other approaches
Instance-based learning
other neural networks
Concept learning (Version space, Focus, Aq11, …)
Genetic algorithms
Reinforcement learning
9/18/2018
CSE 572: Data Mining by H. Liu

6. CSE 572: Data Mining by H. Liu
Different approaches
There exist many techniques
Stratification(sampling based on cost)
Algorithm specific methods
Build or prune decision tree based on cost
cost-sensitive boosting, AdaCost …
Meta-cost
9/18/2018
CSE 572: Data Mining by H. Liu

7. CSE 572: Data Mining by H. Liu
Meta-cost
Sampling multiple times to build different models
For each example, calculate the probability of prediction for each class
Re-label the training data based on the probability and cost matrix
Build a normal error-based classifier
Issues
How to build such a tree from the data?
What are the criteria for performance measurement
correctness
conciseness
What are the key components?
test
stopping criterion
9/18/2018
CSE 572: Data Mining by H. Liu

8. CSE 572: Data Mining by H. Liu
Further issues
Multiple classes?
Individual example cost?
Different types of cost? Test cost?
(Tasks for the group on cost-sensitive learning!)
Issues
How to build such a tree from the data?
What are the criteria for performance measurement
correctness
conciseness
What are the key components?
test
stopping criterion
9/18/2018
CSE 572: Data Mining by H. Liu

9. CSE 572: Data Mining by H. Liu
Ensemble Learning
Prepared with Surendra
9/18/2018
CSE 572: Data Mining by H. Liu

10. CSE 572: Data Mining by H. Liu
Types of Ensemble
Homogeneous Ensembles
Uses the same learning algorithm e.g.. Bagging, Boosting
Heterogeneous Ensembles
Uses different learning algorithms e.g.., combination of Decision Tree, Nearest Neighbour, K-Star, etc.
9/18/2018
CSE 572: Data Mining by H. Liu

11. Phases of Building an Ensemble
Model Generation
Generate diverse set of classifiers
Resampling, using different learning algorithm, various other strategies
Model Combination
Decide upon a strategy of combining the predictions of the classifiers making the ensemble
9/18/2018
CSE 572: Data Mining by H. Liu

12. Meta–Classification Framework
Classifiers at two levels
Base level or low level classifiers generated during model generation phase.
Meta level classifier created during model combination phase.
9/18/2018
CSE 572: Data Mining by H. Liu

13. Categorization of Model Combination Strategies
Voting
As the name implies do some sort of voting of base classifiers
Stacking
Find a pattern between the predictions of base classifiers and the actual class label
Grading
Grade the base-classifiers and decide the subset which should be used
9/18/2018
CSE 572: Data Mining by H. Liu

14. CSE 572: Data Mining by H. Liu
Voting Techniques
Majority Voting
Sum the prediction probabilities of different classes given by the base classifier and predict in favor of the majority class
Weighted Voting
Assign weights to classifiers and do a weighted sum of prediction probabilities
Weight calculated using the error rate
Threshold Voting
Use majority voting or weighted voting only when the error rate is above a certain threshold
9/18/2018
CSE 572: Data Mining by H. Liu

15. CSE 572: Data Mining by H. Liu
Stacking Techniques
Stacking
Use complete class distribution from each classifier
Build a stacking classifier for each class
9/18/2018
CSE 572: Data Mining by H. Liu

16. CSE 572: Data Mining by H. Liu
StackingC
Use class distribution only for the concerned class
9/18/2018
CSE 572: Data Mining by H. Liu

17. CSE 572: Data Mining by H. Liu
Grading Techniques
Grading/Referee Method
For each base classifier there is a grader classifier which determines whether the base classifier will be correct or not for the given test instance. G1 C1 G2 C2 G3 C3
9/18/2018
CSE 572: Data Mining by H. Liu