1. A task of induction to find patterns
Classification
A task of induction to find patterns
6/25/2019
CSE 591: Data Mining by H. Liu

2. CSE 591: Data Mining by H. Liu
Outline
Data and its format
Problem of Classification
Learning a classifier
Different approaches
Key issues
6/25/2019
CSE 591: Data Mining by H. Liu

3. CSE 591: Data Mining by H. Liu
Data and its format
Data
attribute-value pairs
with/without class
Data type
continuous/discrete
nominal
Data format
flat
6/25/2019
CSE 591: Data Mining by H. Liu

4. CSE 591: Data Mining by H. Liu
Sample data
6/25/2019
CSE 591: Data Mining by H. Liu

5. Induction from databases
Inferring knowledge from data
The task of deduction
infer information that is a logical consequence of querying a database
Who conducted this class before?
Which courses are attended by Mary?
Deductive databases: extending the RDBMS
RDBMS - relational database management systems
RDBMS offer simple operators for the deduction of information, such as join
6/25/2019
CSE 591: Data Mining by H. Liu

6. CSE 591: Data Mining by H. Liu
Classification
It is one type of induction
data with class labels
Examples -
If weather is rainy then no golf If
Induction is different from deduction and DBMS does not not support induction;
The result of induction is higher-level information or knowledge: general statements about data
There are many approaches. Refer to the lecture notes for CS3244 available at the Co-Op.
We focus on three approaches here, other examples:
Other approaches
Instance-based learning
other neural networks
Concept learning (Version space, Focus, Aq11, …)
Genetic algorithms
Reinforcement learning
6/25/2019
CSE 591: Data Mining by H. Liu

7. CSE 591: Data Mining by H. Liu
Different approaches
There exist many techniques
Decision trees
Neural networks
K-nearest neighbors
Naïve Bayesian classifiers
Support Vector Machines
Ensemble methods
Co-training
and many more ...
6/25/2019
CSE 591: Data Mining by H. Liu

8. CSE 591: Data Mining by H. Liu
A decision tree
Outlook
Humidity
Wind
sunny
overcast
rain
YES
high
normal
strong
weak NO
Issues
How to build such a tree from the data?
What are the criteria for performance measurement
correctness
conciseness
What are the key components?
test
stopping criterion
6/25/2019
CSE 591: Data Mining by H. Liu

9. Inducing a decision tree
There are many possible trees
let’s try it on the golfing data
How to find the most compact one
that is consistent with the data?
Why the most compact?
Occam’s razor principle
Issue of efficiency w.r.t. optimality
6/25/2019
CSE 591: Data Mining by H. Liu

10. CSE 591: Data Mining by H. Liu
Information gain
and
Entropy -
Information gain - the difference between the node before and after splitting
6/25/2019
CSE 591: Data Mining by H. Liu

11. Building a compact tree
The key to building a decision tree - which attribute to choose in order to branch.
The heuristic is to choose the attribute with the maximum IG.
Another explanation is to reduce uncertainty as much as possible.
6/25/2019
CSE 591: Data Mining by H. Liu

12. CSE 591: Data Mining by H. Liu
Learn a decision tree
Outlook
sunny
overcast
rain
Humidity
Wind
YES
high
normal
strong
weak NO
YES NO
YES
6/25/2019
CSE 591: Data Mining by H. Liu

13. CSE 591: Data Mining by H. Liu
K-Nearest Neighbor
One of the most intuitive classification algorithm
An unseen instance’s class is determined by its nearest neighbor
The problem is it is sensitive to noise
Instead of using one neighbor, we can use k neighbors
6/25/2019
CSE 591: Data Mining by H. Liu

14. CSE 591: Data Mining by H. Liu
K-NN
New problems
lazy learning
large storage
An example
How good is k-NN?
6/25/2019
CSE 591: Data Mining by H. Liu

15. Naïve Bayes Classifier
This is a direct application of Bayes’ rule
P(C|X) = P(X|C)P(C)/P(X)
X - a vector of x1,x2,…,xn
That’s the best classifier you can build
But, there are problems
6/25/2019
CSE 591: Data Mining by H. Liu

16. CSE 591: Data Mining by H. Liu
NBC (2)
Assume conditional independence between xi’s
We have
An example
How good is it in reality?
6/25/2019
CSE 591: Data Mining by H. Liu

17. Classification via Neural Networks
Squash 
A perceptron
6/25/2019
CSE 591: Data Mining by H. Liu

18. What can a perceptron do?
Neuron as a computing device
To separate a linearly separable points
Nice things about a perceptron
distributed representation
local learning
weight adjusting
6/25/2019
CSE 591: Data Mining by H. Liu

19. CSE 591: Data Mining by H. Liu
Linear threshold unit
Basic concepts: projection, thresholding
W vectors evoke 1
W = [.11 .6]
L= [.7 .7] .5
6/25/2019
CSE 591: Data Mining by H. Liu

20. E.g. 1: solution region for AND problem
Find a weight vector that satisfies all the constraints
AND problem
0 0 0
0 1 0
1 0 0
1 1 1
6/25/2019
CSE 591: Data Mining by H. Liu

21. E.g. 2: Solution region for XOR problem?
0 0 0
0 1 1
1 0 1
1 1 0
6/25/2019
CSE 591: Data Mining by H. Liu

22. Learning by error reduction
Perceptron learning algorithm
If the activation level of the output unit is 1 when it should be 0, reduce the weight on the link to the ith input unit by r*Li, where Li is the ith input value and r a learning rate
If the activation level of the output unit is 0 when it should be 1, increase the weight on the link to the ith input unit by r*Li
Otherwise, do nothing
6/25/2019
CSE 591: Data Mining by H. Liu

23. Multi-layer perceptrons
Using the chain rule, we can back-propagate the errors for a multi-layer perceptrons.
Output layer
Hidden layer
Differences between DT and NN
Speed
Accuracy
Comprehensibility
Which one to use
Many successful applications of both approaches
Input layer
6/25/2019
CSE 591: Data Mining by H. Liu