1. Classification and Prediction
(baseado nos slides do livro: Data Mining: C & T)

2. Classification and Prediction
What is classification? What is prediction?
Issues regarding classification and prediction
Classification by decision tree induction
Bayesian Classification (TI)
Classification by Back Propagation (Neural Networks) (TI)
Support Vector Machines (SVM)
Associative Classification: Classification by association rule analysis
Other Classification Methods: K-Nearest Neighbor, Case-based reasoning, etc
Prediction: Regression (TI)
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

3. Classification vs. Prediction
predicts categorical class labels (discrete or nominal)
classifies data (constructs a model) based on the training set and the values (class labels) in a classifying attribute and uses it in classifying new data
Prediction
models continuous-valued functions, i.e., predicts unknown or missing values
Typical applications
Credit approval
Target marketing
Medical diagnosis
Fraud detection
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

4. Classification—A Two-Step Process (1)
Model construction: describing a set of predetermined classes
Each tuple/sample is assumed to belong to a predefined class, as determined by the class label attribute
The set of tuples used for model construction is the training set
The model is represented as classification rules, decision trees, or mathematical formulae
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

5. Classification—A Two-Step Process (2)
Model usage: for classifying future or unknown objects
Estimate accuracy of the model
The known label of test sample is compared with the classified result from the model
Accuracy rate is the percentage of test set samples that are correctly classified by the model
Test set is independent of training set, otherwise over-fitting will occur
If the accuracy is acceptable, use the model to classify data tuples whose class labels are not known
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

6. Classification Process (1): Model Construction
Algorithms
Training
Data
Classifier
(Model)
IF rank = ‘professor’
OR years > 6
THEN tenured = ‘yes’
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

7. Classification Process (2): Use the Model in Prediction
Classifier
Testing
Data
Unseen Data
(Jeff, Professor, 4)
Tenured?
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

8. Supervised vs. Unsupervised Learning
Supervised learning (classification)
The training data (observations, measurements, etc.) are accompanied by labels indicating the class of the observations
New data is classified based on the training set
Unsupervised learning (clustering)
The class labels of training data is unknown
Given a set of measurements, observations, etc. with the aim of establishing the existence of classes or clusters in the data
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

9. Classification and Prediction
What is classification? What is prediction?
Issues regarding classification and prediction
Classification by decision tree induction
Bayesian Classification
Classification by Back Propagation
Support Vector Machines
Associative Classification: Classification by association rule analysis
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

10. Sistemas de Apoio à Decisão
Data Preparation
Data cleaning
Preprocess data in order to reduce noise and handle missing values
Relevance analysis (feature selection)
Remove the irrelevant or redundant attributes
Data transformation
Generalize and/or normalize data
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

11. Evaluating classification methods
Accuracy: classifier accuracy and predictor accuracy
Speed and scalability
time to construct the model (training time)
time to use the model (classification/prediction time)
Robustness
handling noise and missing values
Scalability
efficiency in disk-resident databases
Interpretability
understanding and insight provided by the model
Other measures
e.g., goodness of rules, such as decision tree size or compactness of classification rules
Scalability evaluated by assessing the nb of I/O operations involved for a given classification algo. on data sets of increasingly large size.
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

12. Classification Accuracy: Estimating Error Rates
Partition: Training-and-testing
use two independent data sets, e.g., training set (2/3), test set (1/3)
used for data set with large number of samples
Cross-validation
divide the data set into k sub-samples
use k-1 sub-samples as training data and one sub-sample as test data—k-fold cross-validation
for data set with moderate size
Bootstrapping (leave-one-out)
for small size data
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

13. Increasing Classfier Accuracy: Bagging
General idea: averaging the prediction over a collection of classifiers
Training data
Altered Training data
……..
Aggregation ….
Classification method (CM)
Classifier C CM
Classifier C1 CM
Classifier C2
Classifier C*
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

14. Bagging: The Algorithm
Given a set S of s samples
Generate a bootstrap sample T from S. Cases in S may not appear in T or may appear more than once.
Repeat this sampling procedure, getting a sequence of k independent training sets
A corresponding sequence of classifiers C1, C2, …, Ck is constructed for each of these training sets, by using the same classification algorithm
To classify an unknown sample X, let each classifier predict or vote
The Bagged Classifier C* counts the votes and assigns X to the class with the “most” votes
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

15. Classification and Prediction
What is classification? What is prediction?
Issues regarding classification and prediction
Classification by decision tree induction
Bayesian Classification (TI)
Classification by Back Propagation (Neural Networks) (TI)
Support Vector Machines (SVM)
Associative Classification: Classification by association rule analysis
Other Classification Methods: K-Nearest Neighbor, Case-based reasoning, etc
Prediction: Regression (TI)
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

16. Decision Tree Induction: Training Dataset
This follows an example of Quinlan’s ID3 (Playing Tennis)
Aprendizagem de um conceito: inferir o valor de uma função do cjto de treino, a partir do seu
Input e output
Função alvo: buys_computer
Tarefa: prever o valor de buys_computer baseado nos valores dos outros atributos
Representação de hipóteses:
Hipótese – conjunção de restrições sobre os atributos
Se uma instância satisfaz as restrições de uma hipóteses e buys_computer(x)) = 1 é ex. Positivo,
Se buys_computer(x) = 0 então é exemplo negativo
Aprendizagem é a tarefa de procurar num espaço de hipóteses aquela que melhor
se adequa aos exemplos de treino
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

17. Sistemas de Apoio à Decisão
Output: A Decision Tree
for “buys_computer”
age?
overcast
student?
credit rating? no
yes
fair
excellent
<=30
>40
30..40
Representa um conceito (comprar computador) ou seja, prevê se um cliente é um potencial comprador de computador.
Árvore de decisão: representa uma disjunção de conjunções
Para classificar uma amostra desconhecida, os valores dos seus atributos são testados contra a árvore de decisão.
O caminho desde a raiz até ao nó folha é traçado e o nó folha dá a previsão de classe para essa amostra.
Àrvores de decisão são adequadas para problemas onde: - as instâncias são descritas por pares atributo-valor; a função alvo tem valores discretos (existe extensão para valores contínuos); são necessárias descrições com disjunção; os dados de treino podem conter erros e pode conter missing values
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

18. Algorithm for Decision Tree Induction (ID3)
Basic algorithm (a greedy algorithm)
Tree is constructed in a top-down recursive divide-and-conquer manner
At start, all the training examples are at the root
Attributes are categorical (if continuous-valued, they are discretized in advance)
Examples are partitioned recursively based on selected attributes
Test attributes are selected on the basis of a heuristic or statistical measure (e.g., information gain)
Conditions for stopping partitioning
All samples for a given node belong to the same class
There are no remaining attributes for further partitioning – majority voting is employed for classifying the leaf
There are no samples left
A ideia de qq. Algo de indução de árvores de decisão é encontrar a árvore que melhor se adequa
ao cjto de treino, isto é que melhor classifica o cjto de treino, dentro de todas as árvores possíveis.
ID3 favorece árvores mis pequenas e aquelas que colocam os atributos com maior ganho de informação
Mais perto da raiz.
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

19. Sistemas de Apoio à Decisão
Attribute Selection Measure: Information Gain (ID3/C4.5)
Select the attribute with the highest information gain
S contains si tuples of class Ci for i = {1, …, m}
information measures info required to classify any arbitrary tuple
entropy of attribute A with values {a1,a2,…,av}
information gained by branching on attribute A
Then how can we decide a test attribute on each node?
One of the popular methods is using information gain measure, which we covered in chapter 5.
It involves rather complicated equations, and I’ll not present the details here. Just basic ideas.
The basic idea is that we select the attribute with the highest information gain.
This information gain can be calculated from the expected information I and entropy of each attribute, E
I : the expected information needed to classify a given sample
E (entropy) : expected information based on the partitioning into subsets by A
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

20. Attribute Selection by Information Gain Computation
means “age <=30” has 5 out of 14 samples, with 2 yes’es and 3 no’s. Hence
Similarly:
Class P: buys_computer = “yes”
Class N: buys_computer = “no”
I(p, n) = I(9, 5) =0.940
Compute the entropy for age:
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

21. Extracting Classification Rules from Trees
Represent the knowledge in the form of IF-THEN rules
One rule is created for each path from the root to a leaf
Each attribute-value pair along a path forms a conjunction
The leaf node holds the class prediction
Rules are easier for humans to understand
Example:
IF age = “<=30” AND student = “no” THEN buys_computer = “no”
IF age = “<=30” AND student = “yes” THEN buys_computer = “yes”
IF age = “31…40” THEN buys_computer = “yes”
IF age = “>40” AND credit_rating = “excellent” THEN buys_computer = “no”
IF age = “<=30” AND credit_rating = “fair” THEN buys_computer = “yes”
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

22. Avoid Overfitting in Classification
Overfitting: An induced tree may overfit the training data
Too many branches, some may reflect anomalies due to noise or outliers
Poor accuracy for unseen samples
Two approaches to avoid overfitting
Prepruning: Halt tree construction early, do not split a node if this would result in the goodness measure falling below a threshold; difficult to choose an appropriate threshold
Postpruning: Remove branches from a “fully grown” tree—get a sequence of progressively pruned trees
Use a set of data different from the training data to decide which is the “best pruned tree”
Overfitting pode acontecer se o cjto de treino é pequeno de mais
Pode existir uma hipótese (árvore) que se comporte melhor com os dados completos,
Embora pior com os dados de treino.
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

23. Approaches to Determine the Final Tree Size
Separate training (2/3) and testing (1/3) sets
Use cross validation, e.g., 10-fold cross validation
Use all the data for training
but apply a statistical test (e.g., chi-square) to estimate whether expanding or pruning a node may improve the entire distribution
Use minimum description length (MDL) principle
halting growth of the tree when the encoding is minimized
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

24. Enhancements to Basic Decision Tree Induction
Allow for continuous-valued attributes
Dynamically define new discrete-valued attributes that partition the continuous attribute value into a discrete set of intervals
Handle missing attribute values
Assign the most common value of the attribute
Assign probability to each of the possible values
Attribute construction
Create new attributes based on existing ones that are sparsely represented
This reduces fragmentation, repetition, and replication
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

25. Computing Info. Gain for Continuous-Value Attributes
Let attribute A be a continuous-valued attribute
Must determine the best split point for A
Sort the value A in increasing order
Typically, the midpoint between each pair of adjacent values is considered as a possible split point
(ai+ai+1)/2 is the midpoint between the values of ai and ai+1
The point with the minimum expected information requirement for A is selected as the split-point for A
Split:
D1 is the set of tuples in D satisfying A ≤ split-point, and D2 is the set of tuples in D satisfying A > split-point
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

26. Classification in Large Databases
Classification: a classical problem extensively studied by statisticians and machine learning researchers
Scalability: Classify data sets with millions of examples and hundreds of attributes with reasonable speed
Why decision tree induction in data mining?
relatively faster learning speed (than other classification methods)
convertible to simple and easy to understand classification rules
can use SQL queries for accessing databases
comparable classification accuracy with other methods
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

27. Scalable Decision Tree Induction Methods
SLIQ (EDBT’96 — Mehta et al.): builds an index for each attribute and only class list and the current attribute list reside in memory
SPRINT (VLDB’96 — J. Shafer et al.): constructs an attribute list data structure
PUBLIC (VLDB’98 — Rastogi & Shim): integrates tree splitting and tree pruning: stop growing the tree earlier
RainForest (VLDB’98 — Gehrke, Ramakrishnan & Ganti): separates the scalability aspects from the criteria that determine the quality of the tree; builds an AVC-list (attribute, value, class label)
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

28. Presentation of Classification Results
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

29. Visualization of a Decision Tree in SGI/MineSet 3.0
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

30. Sistemas de Apoio à Decisão
Bibliografia
(Livro) Data Mining: Concepts and Techniques, J. Han & M. Kamber, Morgan Kaufmann, 2001 (Secções 7.1 a 7.3 – livro 2001, Secções 5.1 a 5.3 – draft)
(Livro) Machine Learning, T. Mitchell, McGraw Hill, 1997
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)

31. Data Cube-Based Decision-Tree Induction
Integration of generalization with decision-tree induction (Kamber et al’97).
Classification at primitive concept levels
E.g., precise temperature, humidity, outlook, etc.
Low-level concepts, scattered classes, bushy classification-trees
Semantic interpretation problems.
Cube-based multi-level classification
Relevance analysis at multi-levels.
Information-gain analysis with dimension + level.
2003/04
Sistemas de Apoio à Decisão
(LEIC Tagus)