1. ML410C Projects in health informatics – Project and information management Data Mining

2. Last time… Why do we need data analysis? What is data mining?
Examples where data mining has been useful
Data mining and other areas of computer science and mathematics
Some (basic) data mining tasks

3. The Knowledge Discovery Process
Knowledge Discovery in Databases (KDD) is the nontrivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data.
U.M. Fayyad, G. Piatetsky-Shapiro and P. Smyth, “From Data Mining to Knowledge Discovery in Databases”, AI Magazine 17(3): (1996)

4. CRISP-DM: CRoss Industry Standard Process for Data Mining
Shearer C., “The CRISP-DM model: the new blueprint for data mining”,
Journal of Data Warehousing 5 (2000) (see also

5. Today DATE TIME ROOM TOPIC MONDAY 2013-09-09 10:00-11:45 502
Introduction to data mining
WEDNESDAY
09:00-10:45
501
Decision trees, rules and forests
FRIDAY
Sal C
Evaluating predictive models and tools for data mining

6. Today What is classification Overview of classification methods
Decision trees
Forests

7. Predictive data mining
Our task
Input: data representing objects that have been assigned labels
Goal: accurately predict labels for new (previously unseen) objects

8. An example: email classification
Features (attributes)
Ex.
All caps
No. excl. marks
Missing date
No. digits in From:
Image fraction
Spam e1
yes no 3 e2
0.2 e3 1 e4 4
0.5 e5 2 e6
Examples (observations)

9. Decision tree
Spam = yes
Spam = yes
Spam = no

10. Rules
Spam = yes
Spam = no
Spam = yes
Spam = no
Spam = no

11. Forests 11

12. Classification What is the class of the following e-mail? No Caps: Yes
No. excl. marks: 0
Missing date: Yes
No. digits in From: 4
Image fraction: 0.3

13. Classification What is classification?
Issues regarding classification and prediction
Classification by decision tree induction
Classification by Naïve Bayes classifier
Classification by Nearest Neighbor
Classification by Bayesian Belief networks

14. Classification Classification: Typical Applications
predicts categorical class labels
classifies data (constructs a model) based on the training set and the values (class labels) in a classifying attribute
uses the model for classifying new data
Typical Applications
credit approval
target marketing
medical diagnosis
treatment effectiveness analysis

15. Why Classification? A motivating application
Credit approval
A bank wants to classify its customers based on whether they are expected to pay back their approved loans
The history of past customers is used to train the classifier
The classifier provides rules, which identify potentially reliable future customers

16. Why Classification? A motivating application
Credit approval
Classification rule:
If age = “ ” and income = high
then credit_rating = excellent
Future customers
Paul: age = 35, income = high  excellent credit rating
John: age = 20, income = medium  fair credit rating

17. Classification—A Two-Step Process
Model construction: describing a set of predetermined classes
Each tuple/sample is assumed to belong to a predefined class, as determined by the class label attribute
The set of tuples used for model construction: training set
The model is represented as classification rules, decision trees, or mathematical formulas

18. Classification—A Two-Step Process
Model usage: for classifying future or unknown objects
Estimate accuracy of the model
The known label of test samples is compared with the classified result from the model
Accuracy rate is the percentage of test set samples that are correctly classified by the model
Test set is independent of training set, otherwise over-fitting will occur

19. Classification Process (1): Model Construction
Algorithms
Training
Data
Classifier
(Model)
IF LDL = ‘high’
OR Gluc > 6 mmol/lit
THEN Heart attack = ‘yes’

20. Classification Process (2): Use the Model in Prediction
Classifier
Accuracy=?
Testing
Data
Unseen Data
(Jeff, high, 7.5)
Heart attack?

21. Supervised vs. Unsupervised Learning
Supervised learning (classification)
Supervision: The training data (observations, measurements, etc.) are accompanied by labels indicating the class of the observations
New data is classified based on the training set
Unsupervised learning (clustering)
The class labels of training data is unknown
Given a set of measurements, observations, etc. with the aim of establishing the existence of classes or clusters in the data

22. Issues regarding classification and prediction: Evaluating Classification Methods
Predictive accuracy
Speed
time to construct the model
time to use the model
Robustness
handling noise and missing values
Scalability
efficiency in disk-resident databases
Interpretability:
understanding and insight provided by the model
Goodness of rules (quality)
decision tree size
compactness of classification rules

23. Classification by Decision Tree Induction
A flow-chart-like tree structure
Internal node denotes a test on an attribute
Branch represents an outcome of the test
Leaf nodes represent class labels or class distribution
Decision tree generation consists of two phases
Tree construction
At start, all the training examples are at the root
Partition examples recursively based on selected attributes
Tree pruning
Identify and remove branches that reflect noise or outliers
Use of decision tree: Classifying an unknown sample
Test the attribute values of the sample against the decision tree

24. Training Dataset
Example

25. Output: A Decision Tree for “buys_computer”
age?
<=30
overcast
30..40
>40
student?
yes
credit rating? no
yes
excellent
fair no
yes no
yes

26. Algorithm for Decision Tree Induction
Basic algorithm (a greedy algorithm)
Tree is constructed in a top-down recursive divide-and-conquer manner
At start, all the training examples are at the root
Attributes are categorical (if continuous-valued, they are discretized in advance)
Samples are partitioned recursively based on selected attributes
Test (split) attributes are selected on the basis of a heuristic or statistical measure (e.g., information gain)
Conditions for stopping partitioning
All samples for a given node belong to the same class
There are no remaining attributes for further partitioning – majority voting is employed for classifying the leaf
There are no samples left

27. Algorithm for Decision Tree Induction (pseudocode)
Algorithm GenDecTree(Sample S, Attlist A)
create a node N
If all samples are of the same class C then label N with C; terminate;
If A is empty then label N with the most common class C in S (majority voting); terminate;
Select aA, with the highest information gain; Label N with a;
For each value v of a:
Grow a branch from N with condition a=v;
Let Sv be the subset of samples in S with a=v;
If Sv is empty then attach a leaf labeled with the most common class in S;
Else attach the node generated by GenDecTree(Sv, A-a)

28. Attribute Selection Measure: Information Gain
Let pi be the probability that an arbitrary tuple in D belongs to class Ci, estimated by |Ci, D|/|D|
- where Ci, D denotes the set of tuples that belong to class Ci
Expected information (entropy) needed to classify a tuple in D:
- where m is the number of classes
I : the expected information needed to classify a given sample
E (entropy) : expected information based on the partitioning into subsets by A

29. Attribute Selection Measure: Information Gain
Information needed (after using A to split D into v partitions) to classify D:
Information gained by branching on attribute A
I : the expected information needed to classify a given sample
E (entropy) : expected information based on the partitioning into subsets by A

30. Attribute Selection: Information Gain
Class P: buys_computer = “yes”
Class N: buys_computer = “no”

31. Splitting the samples using age
<=30
>40
labeled yes

32. Gini index
If a data set D contains examples from n classes, gini index, gini(D) is defined as
- where pj is the relative frequency of class j in D
If a data set D is split on A into two subsets D1 and D2, the gini index gini(D) is defined as

33. Gini index Reduction in Impurity:
The attribute that provides the smallest ginisplit(D) (or the largest reduction in impurity) is chosen to split the node

34. Gini index (CART, IBM IntelligentMiner)
Example:
D has 9 tuples in buys_computer = “yes” and 5 in “no”
Suppose that attribute “income” partitions D into 10 records (D1: {low, medium}) and 4 records (D2: {high}).

35. Gini index Then: = 0.45 and gini{medium,high} = 0.30
All attributes are assumed continuous-valued
May need other tools, e.g., clustering, to get the possible split values

36. Comparing Attribute Selection Measures
The two measures, in general, return good results but
Information gain:
biased towards multivalued attributes
Gini index:
biased to multivalued attributes
has difficulty when # of classes is large
tends to favor test sets that result in equal-sized partitions and purity in both partitions

37. Overfitting due to noise
Decision boundary is distorted by noise point

38. Overfitting due to insufficient samples
Why?

39. Overfitting due to insufficient samples
Lack of data points in the lower half of the diagram makes it difficult to predict correctly the class labels of that region
- Insufficient number of training records in the region causes the decision tree to predict the test examples using other training records that are irrelevant to the classification task

40. Overfitting and Tree Pruning
Overfitting: An induced tree may overfit the training data
Too many branches, some may reflect anomalies due to noise or outliers
Poor accuracy for unseen samples
Two approaches to avoid overfitting
Prepruning: Halt tree construction early—do not split a node if this would result in the goodness measure falling below a threshold
Difficult to choose an appropriate threshold
Postpruning: Remove branches from a “fully grown” tree—get a sequence of progressively pruned trees

41. Occam’s Razor “entia non sunt multiplicanda praeter ecessitatem”,
Given two models of similar generalization errors, one should prefer the simpler model over the more complex model
Therefore, one should include model complexity when evaluating a model
“entia non sunt multiplicanda praeter ecessitatem”,
which translates to:
“entities should not be multiplied beyond necessity”.

42. Pros and Cons of decision trees
Reasonable training time
Fast application
Easy to interpret
Easy to implement
Can handle large number of features
Cons
Cannot handle complicated relationship between features
simple decision boundaries
problems with lots of missing data

43. Some well-known decision tree learning implementations
CART Breiman L, Friedman JH, Olshen RA, Stone CJ (1984) Classification and Regression Trees. Wadsworth
ID3 Quinlan JR (1986) Induction of decision trees. Machine Learning 1:81–106
C4.5 Quinlan JR (1993) C4.5: Programs for machine learning. Morgan Kaufmann
J48 Implementation of C4.5 in WEKA

44. Handling missing values
Remove attributes with missing values
Remove examples with missing values
Assume most frequent value
Assume most frequent value given a class
Learn the distribution of a given attribute
Find correlation between attributes

45. Handling missing values
Example
A 1 …
Class e1
yes + e2 no e3 - e4 ? A1
yes no
e1 (w=1)
e3 (w=1)
e4 (w=2/3)
e2 (w=1)
e4 (w=1/3)

46. k-nearest neighbor classifiers
k-nearest neighbors of a record x are data points that have the k smallest distance to x

47. k-nearest neighbor classification
Given a data record x find its k closest points
Closeness: ?
Determine the class of x based on the classes in the neighbor list
Majority vote
Weigh the vote according to distance
e.g., weight factor, w = 1/d2

48. Characteristics of nearest-neighbor classifiers
No model building (lazy learners)
Lazy learners: computational time in classification
Eager learners: computational time in model building
Decision trees try to find global models, k-NN take into account local information
K-NN classifiers depend a lot on the choice of proximity measure

49. Condorcet’s jury theorem
If each member of a jury is more likely to be right than wrong,
then the majority of the jury, too, is more likely to be right than wrong
and the probability that the right outcome is supported by a majority of the jury is a (swiftly) increasing function of the size of the jury,
converging to 1 as the size of the jury tends to infinity
Condorcet, 1785 49

50. Condorcet’s jury theorem50

51. Random forests
Random forests (Breiman 2001) are generated by combining two techniques:
bagging (Breiman 1996)
the random subspace method (Ho 1998)
L. Breiman. Random forests. Machine Learning, 45(1):5–32, 2001
L. Breiman. Bagging predictors. Machine Learning, 24(2):123–140, 1996
T. K. Ho. The random subspace method for constructing decision forests, IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(8): , 1998 51

52. Bagging A bootstrap replicate E’ of a set of examples E is created by
Other
Bar
Fri/Sat
Hungry
Guests
Wait e2
yes no
full e3
some e4 e6
Ex.
Other
Bar
Fri/Sat
Hungry
Guests
Wait e1
yes no
some e2
full e3 e4 e5
none e6
A bootstrap replicate E’ of a set of examples E is created by
randomly selecting n = |E| examples from E with replacement.

53. Forests 53

54. Today What is classification Overview of classification methods
Decision trees
Forests

55. Next time DATE TIME ROOM TOPIC MONDAY 2013-09-09 10:00-11:45 502
Introduction to data mining
WEDNESDAY
09:00-10:45
501
Decision trees, rules and forests
FRIDAY
Sal C
Evaluating predictive models and tools for data mining