Overfitting Overfitting occurs when a statistical model describes random error or noise instead of the underlying relationship. Overfitting generally occurs.

Overfitting Overfitting occurs when a statistical model describes random error or noise instead of the underlying relationship. Overfitting generally occurs when a model is excessively complex, such as having too many parameters relative to the number of observations. A model which has been overfit will generally have poor predictive performance, as it can exaggerate minor fluctuations in the data.

Suppose we need to solve a classification problem We are not sure if we should us the.. Simple linear classifier or the Simple quadratic classifier How do we decide which to use? We do cross validation and choose the best one.

100 1020304050607080 90 100 10 20 30 40 50 60 70 80 90 100 1020304050607080 90 100 10 20 30 40 50 60 70 80 90 Simple linear classifier gets 81% accuracy Simple quadratic classifier 99% accuracy

Simple linear classifier gets 96% accuracy Simple quadratic classifier 97% accuracy

This problem is greatly exacerbated by having too little data Simple linear classifier gets 90% accuracy Simple quadratic classifier 95% accuracy

What happens as we have more and more training examples? The accuracy for all models goes up! The chance of making a mistake goes down The cost of the mistake (if made) goes down Simple linear 70% accuracy Simple quadratic 90% accuracy Simple linear 90% accuracy Simple quadratic 95% accuracy Simple linear 99% accuracy Simple quadratic 99% accuracy

One Solution: Charge Penalty for complex models For example, for the simple {polynomial} classifier, we could charge 1% for every increase in the degree of the polynomial 10 123456789 1 2 3 4 5 6 7 8 9 123456789 1 2 3 4 5 6 7 8 9 123456789 1 2 3 4 5 6 7 8 9 Accuracy = 90.5%Accuracy = 97.0%Accuracy = 97.05% Simple linear classifier gets 90.5% accuracy, minus 0, equals 90.5% Simple quadratic classifier 97.0% accuracy, minus 1, equals 96.0% Simple cubic classifier 97.05% accuracy, minus 2, equals 95.05%

One Solution: Charge Penalty for complex models For example, for the simple {polynomial} classifier, we could charge 1% for every increase in the degree of the polynomial. There are more principled ways to charge penalties In particular, there is a technique called Minimum Description Length (MDL)

Suppose you have a four feature problem, and you want to search over feature subsets. It happens to be the case that features 2 and 3, shown here Are all you need, and the other features are random

Suppose you have a four feature problem, and you want to search over feature subsets. It happens to be the case that features 2 and 3, shown here are all you need, and the other features are random 1234 3,42,41,42,31,31,2 2,3,41,3,41,2,41,2,3 1,2,3,4 0123401234

Insect ID AbdomenLengthAntennaeLength Insect Class 12.75.5Grasshopper 28.09.1Katydid 30.94.7Grasshopper 41.13.1Grasshopper 55.48.5Katydid 62.91.9Grasshopper 76.16.6Katydid 80.51.0Grasshopper 98.36.6Katydid 108.14.7Katydids My_Collection We have seen that we are given features… Suppose using these features we cannot get satisfactory accuracy results. So far, we have two tricks 1)Ask for more features 2)Remove irrelevant or redundant features There is another possibility…

Feature Generation Feature generation refers to any technique to make new features from existing features 10 12345678 9 1 2 3 4 5 6 7 8 9 Recall pigeon problem 2, and assume we are using the linear classifier Examples of class A 4 5 6 3 Examples of class B 5 2.5 2 5 5 3 2.5 3 Pigeon Problem 2 Using both features works poorly, using just X works poorly, using just Y works poorly..

Feature Generation Solution: Create a new feature Z Z = absolute_value(X-Y) 10 12345678 9 1 2 3 4 5 6 7 8 9 12345678 9 0 Z-axis

Recall this example? It was a teaching example to show that NN could use any distance measure IDNameClass 1GunopulosGreek 2PapadopoulosGreek 3KolliosGreek 4DardanosGreek 5 KeoghIrish 6GoughIrish 7GreenhaughIrish 8HadleighIrish It would not really work very well, unless we had LOTS more data…

AIKO AIMI AINA AIRI AKANE AKEMI AKI AKIKO AKIO AKIRA AMI AOI ARATA ASUKA ABERCROMBIE ABERNETHY ACKART ACKERMAN ACKERS ACKLAND ACTON ADAIR ADLAM ADOLPH AFFLECK ALVIN AMMADON Japanese Names Irish Names

AIKO 0.75 AIMI 0.75 AINA 0.75 AIRI 0.75 AKANE 0.6 AKEMI 0.6 ABERCROMBIE 0.45 ABERNETHY 0.33 ACKART 0.33 ACKERMAN 0.375 ACKERS 0.33 ACKLAND 0.28 ACTON 0.33 Japanese Names Irish Names Z = number of vowels / word length Vowels = I O U A E

I have a box of apples.. All bad All good 0 0.5 1 H(X) Pr(X = good) = p then Pr(X = bad) = 1 − p the entropy of X is given by 0 1 binary entropy function attains its maximum value when p = 0.5

Decision Tree Classifier Ross Quinlan Antenna Length 10 123456789 1 2 3 4 5 6 7 8 9 Abdomen Length Abdomen Length Abdomen Length > 7.1? no yes Katydid Antenna Length Antenna Length > 6.0? no yes Katydid Grasshopper

Antennae shorter than body? Cricket Foretiba has ears? KatydidsCamel Cricket Yes No 3 Tarsi? No Decision trees predate computers

Decision tree –A flow-chart-like tree structure –Internal node denotes a test on an attribute –Branch represents an outcome of the test –Leaf nodes represent class labels or class distribution Decision tree generation consists of two phases –Tree construction At start, all the training examples are at the root Partition examples recursively based on selected attributes –Tree pruning Identify and remove branches that reflect noise or outliers Use of decision tree: Classifying an unknown sample –Test the attribute values of the sample against the decision tree Decision Tree Classification

Basic algorithm (a greedy algorithm) –Tree is constructed in a top-down recursive divide-and-conquer manner –At start, all the training examples are at the root –Attributes are categorical (if continuous-valued, they can be discretized in advance) –Examples are partitioned recursively based on selected attributes. –Test attributes are selected on the basis of a heuristic or statistical measure (e.g., information gain) Conditions for stopping partitioning –All samples for a given node belong to the same class –There are no remaining attributes for further partitioning – majority voting is employed for classifying the leaf –There are no samples left How do we construct the decision tree?

Information Gain as A Splitting Criteria Select the attribute with the highest information gain ( information gain is the expected reduction in entropy ). Assume there are two classes, P and N –Let the set of examples S contain p elements of class P and n elements of class N –The amount of information, needed to decide if an arbitrary example in S belongs to P or N is defined as 0 log(0) is defined as 0

Information Gain in Decision Tree Induction Assume that using attribute A, a current set will be partitioned into some number of child sets The encoding information that would be gained by branching on A Note: entropy is at its minimum if the collection of objects is completely uniform

Person Hair Length WeightAgeClass Homer0”25036M Marge10”15034F Bart2”9010M Lisa6”788F Maggie4”201F Abe1”17070M Selma8”16041F Otto10”18038M Krusty6”20045M Comic8”29038?

Hair Length <= 5? yes no Entropy(4F,5M) = -(4/9)log 2 (4/9) - (5/9)log 2 (5/9) = 0.9911 Entropy(1F,3M) = -(1/4)log 2 (1/4) - (3/4)log 2 (3/4) = 0.8113 Entropy(3F,2M) = -(3/5)log 2 (3/5) - (2/5)log 2 (2/5) = 0.9710 Gain(Hair Length <= 5) = 0.9911 – (4/9 * 0.8113 + 5/9 * 0.9710 ) = 0.0911 Let us try splitting on Hair length

Weight <= 160? yes no Entropy(4F,5M) = -(4/9)log 2 (4/9) - (5/9)log 2 (5/9) = 0.9911 Entropy(4F,1M) = -(4/5)log 2 (4/5) - (1/5)log 2 (1/5) = 0.7219 Entropy(0F,4M) = -(0/4)log 2 (0/4) - (4/4)log 2 (4/4) = 0 Gain(Weight <= 160) = 0.9911 – (5/9 * 0.7219 + 4/9 * 0 ) = 0.5900 Let us try splitting on Weight

age <= 40? yes no Entropy(4F,5M) = -(4/9)log 2 (4/9) - (5/9)log 2 (5/9) = 0.9911 Entropy(3F,3M) = -(3/6)log 2 (3/6) - (3/6)log 2 (3/6) = 1 Entropy(1F,2M) = -(1/3)log 2 (1/3) - (2/3)log 2 (2/3) = 0.9183 Gain(Age <= 40) = 0.9911 – (6/9 * 1 + 3/9 * 0.9183 ) = 0.0183 Let us try splitting on Age

Weight <= 160? yes no Hair Length <= 2? yes no Of the 3 features we had, Weight was best. But while people who weigh over 160 are perfectly classified (as males), the under 160 people are not perfectly classified… So we simply recurse! This time we find that we can split on Hair length, and we are done!

Weight <= 160? yesno Hair Length <= 2? yes no We need don’t need to keep the data around, just the test conditions. Male Female How would these people be classified?

It is trivial to convert Decision Trees to rules… Weight <= 160? yesno Hair Length <= 2? yes no Male Female Rules to Classify Males/Females If Weight greater than 160, classify as Male Elseif Hair Length less than or equal to 2, classify as Male Else classify as Female Rules to Classify Males/Females If Weight greater than 160, classify as Male Elseif Hair Length less than or equal to 2, classify as Male Else classify as Female

Decision tree for a typical shared-care setting applying the system for the diagnosis of prostatic obstructions. Once we have learned the decision tree, we don’t even need a computer! This decision tree is attached to a medical machine, and is designed to help nurses make decisions about what type of doctor to call.

Garzotto M et al. JCO 2005;23:4322-4329 PSA = serum prostate-specific antigen levels PSAD = PSA density TRUS = transrectal ultrasound

Wears green? Yes No The worked examples we have seen were performed on small datasets. However with small datasets there is a great danger of overfitting the data… When you have few datapoints, there are many possible splitting rules that perfectly classify the data, but will not generalize to future datasets. For example, the rule “Wears green?” perfectly classifies the data, so does “Mothers name is Jacqueline?”, so does “Has blue shoes”… Male Female

Avoid Overfitting in Classification The generated tree may overfit the training data –Too many branches, some may reflect anomalies due to noise or outliers –Result is in poor accuracy for unseen samples Two approaches to avoid overfitting –Prepruning: Halt tree construction early—do not split a node if this would result in the goodness measure falling below a threshold Difficult to choose an appropriate threshold –Postpruning: Remove branches from a “fully grown” tree—get a sequence of progressively pruned trees Use a set of data different from the training data to decide which is the “best pruned tree”

10 12345678 9 1 2 3 4 5 6 7 8 9 12345678 9 1 2 3 4 5 6 7 8 9 Which of the “Pigeon Problems” can be solved by a Decision Tree? 1)Deep Bushy Tree 2)Useless 3)Deep Bushy Tree The Decision Tree has a hard time with correlated attributes ?

Advantages: –Easy to understand (Doctors love them!) –Easy to generate rules Disadvantages: –May suffer from overfitting. –Classifies by rectangular partitioning (so does not handle correlated features very well). –Can be quite large – pruning is necessary. –Does not handle streaming data easily Advantages/Disadvantages of Decision Trees

There now exists, perhaps tens of million of digitized pages of historical manuscripts dating back to the 12 th century, that feature one or more heraldic shields The images are often stained, faded or torn

Wouldn’t it be great if we could automatically hyperlink all similar shields to each other? For example, here we could link two occurrence of the Von Sax family shield. To do this, we need to consider shape, color and texture. Lets just consider shape for now… Manesse Codex an illuminated manuscript in codex form, copied and illustrated between 1304 and 1340 in Zurich

0100200300400 0 1 2 3 I II 151.7 156.1 Shaplet Dictionary I II Shield Decision Tree 2 0 1 PolishSpanish (Polish) (Spanish) SpanishPolishFrench SpanishPolishFrench Training data (subset) Using the entire shape is not a good idea, because the shields can have flourishes or tears Flourishes Tear An NSF funded project (IIS 0803410) is attempting to solve this by using parts of the shapes, called shaplets* Shaplets allow you to build decision trees for shapes * Ye and Keogh (2009) Time Series Shapelets: A New Primitive for Data Mining. SIGKDD 2009 Decision Tree for Shields

Naïve Bayes Classifier We will start off with a visual intuition, before looking at the math… Thomas Bayes 1702 - 1761

Antenna Length 10 123456789 1 2 3 4 5 6 7 8 9 Grasshoppers Katydids Abdomen Length Remember this example? Let’s get lots more data…

Antenna Length 10 123456789 1 2 3 4 5 6 7 8 9 Katydids Grasshoppers With a lot of data, we can build a histogram. Let us just build one for “Antenna Length” for now…

We can leave the histograms as they are, or we can summarize them with two normal distributions. Let us us two normal distributions for ease of visualization in the following slides…

p(c j | d) = probability of class c j, given that we have observed d 3 Antennae length is 3 We want to classify an insect we have found. Its antennae are 3 units long. How can we classify it? We can just ask ourselves, give the distributions of antennae lengths we have seen, is it more probable that our insect is a Grasshopper or a Katydid. There is a formal way to discuss the most probable classification…

10 2 P( Grasshopper | 3 ) = 10 / (10 + 2)= 0.833 P( Katydid | 3 ) = 2 / (10 + 2)= 0.166 3 Antennae length is 3 p(c j | d) = probability of class c j, given that we have observed d

Bayes Classifiers That was a visual intuition for a simple case of the Bayes classifier, also called: Idiot Bayes Naïve Bayes Simple Bayes We are about to see some of the mathematical formalisms, and more examples, but keep in mind the basic idea. previously unseen instance Find out the probability of the previously unseen instance belonging to each class, then simply pick the most probable class.

Bayes Classifiers Bayesian classifiers use Bayes theorem, which says p(c j | d ) = p(d | c j ) p(c j ) p(d) p(c j | d) = probability of instance d being in class c j, This is what we are trying to compute p(d | c j ) = probability of generating instance d given class c j, We can imagine that being in class c j, causes you to have feature d with some probability p(c j ) = probability of occurrence of class c j, This is just how frequent the class c j, is in our database p(d) = probability of instance d occurring This can actually be ignored, since it is the same for all classes

Assume that we have two classes malefemale c 1 = male, and c 2 = female. We have a person whose sex we do not know, say “drew” or d. malefemale malefemale Classifying drew as male or female is equivalent to asking is it more probable that drew is male or female, I.e which is greater p(male | drew) or p(female | drew) malemalemale p(male | drew) = p(drew | male ) p(male) p(drew) (Note: “Drew can be a male or female name”) What is the probability of being called “drew” given that you are a male? What is the probability of being a male? What is the probability of being named “drew”? (actually irrelevant, since it is that same for all classes) Drew Carey Drew Barrymore

p(c j | d) = p(d | c j ) p(c j ) p(d) Officer Drew NameSex DrewMale ClaudiaFemale DrewFemale Female AlbertoMale KarinFemale NinaFemale SergioMale This is Officer Drew (who arrested me in 1997). Is Officer Drew a Male or Female? Luckily, we have a small database with names and sex. We can use it to apply Bayes rule…

male p(male | drew) = 1/3 * 3/8 = 0.125 3/8 3/8 female p(female | drew) = 2/5 * 5/8 = 0.250 3/8 3/8 Officer Drew p(c j | d) = p(d | c j ) p(c j ) p(d) NameSex DrewMale ClaudiaFemale DrewFemale Female AlbertoMale KarinFemale NinaFemale SergioMale Female Officer Drew is more likely to be a Female.

Officer Drew IS a female! Officer Drew male p(male | drew) = 1/3 * 3/8 = 0.125 3/8 3/8 female p(female | drew) = 2/5 * 5/8 = 0.250 3/8 3/8

NameOver 170 CM EyeHair lengthSex DrewNoBlueShortMale ClaudiaYesBrownLongFemale DrewNoBlueLongFemale DrewNoBlueLongFemale AlbertoYesBrownShortMale KarinNoBlueLongFemale NinaYesBrownShortFemale SergioYesBlueLongMale p(c j | d) = p(d | c j ) p(c j ) p(d) So far we have only considered Bayes Classification when we have one attribute (the “antennae length”, or the “name”). But we may have many features. How do we use all the features?

To simplify the task, naïve Bayesian classifiers assume attributes have independent distributions, and thereby estimate p(d|c j ) = p(d 1 |c j ) * p(d 2 |c j ) * ….* p(d n |c j ) The probability of class c j generating instance d, equals…. The probability of class c j generating the observed value for feature 1, multiplied by.. The probability of class c j generating the observed value for feature 2, multiplied by..

p(d 1 |c j ) p(d 2 |c j ) p(d n |c j ) cjcj The Naive Bayes classifiers is often represented as this type of graph… Note the direction of the arrows, which state that each class causes certain features, with a certain probability …

Naïve Bayes is fast and space efficient We can look up all the probabilities with a single scan of the database and store them in a (small) table… SexOver190 cmMaleYes0.15 No0.85 FemaleYes0.01 No0.99 cjcj … p(d 1 |c j ) p(d 2 |c j ) p(d n |c j ) SexLong HairMaleYes0.05 No0.95 FemaleYes0.70 No0.30 SexMale Female

Naïve Bayes is NOT sensitive to irrelevant features... Suppose we are trying to classify a persons sex based on several features, including eye color. (Of course, eye color is completely irrelevant to a persons gender) Female p( Jessica | Female) = 9,000/10,000 * 9,975/10,000 * …. Male p( Jessica | Male) = 9,001/10,000 * 2/10,000 * …. p( Jessica |c j ) = p(eye = brown|c j ) * p( wears_dress = yes|c j ) * …. However, this assumes that we have good enough estimates of the probabilities, so the more data the better. Almost the same!

An obvious point. I have used a simple two class problem, and two possible values for each example, for my previous examples. However we can have an arbitrary number of classes, or feature values AnimalMass >10 kgCatYes0.15 No0.85 DogYes0.91 No0.09 PigYes0.99 No0.01 cjcj … p(d 1 |c j ) p(d 2 |c j ) p(d n |c j ) AnimalCat Dog Pig ColorCatBlack0.33 White0.23 Brown0.44 DogBlack0.97 White0.03 Brown0.90 PigBlack0.04 White0.01 Brown0.95

Naïve Bayesian Classifier p(d 1 |c j ) p(d 2 |c j ) p(d n |c j ) p(d|cj)p(d|cj) Solution Consider the relationships between attributes… SexOver 6 foot MaleYes0.15 No0.85 FemaleYes0.01 No0.99 SexOver 200 pounds MaleYes and Over 6 foot0.11 No and Over 6 foot0.59 Yes and NOT Over 6 foot0.05 No and NOT Over 6 foot0.35 FemaleYes and Over 6 foot0.01

The Naïve Bayesian Classifier has a piecewise quadratic decision boundary Grasshoppers Katydids Ants Adapted from slide by Ricardo Gutierrez-Osuna

10 12345678 9 1 2 3 4 5 6 7 8 9 100 1020304050607080 90 100 10 20 30 40 50 60 70 80 90 10 12345678 9 1 2 3 4 5 6 7 8 9 Which of the “Pigeon Problems” can be solved by a decision tree?

Dear SIR, I am Mr. John Coleman and my sister is Miss Rose Colemen, we are the children of late Chief Paul Colemen from Sierra Leone. I am writing you in absolute confidence primarily to seek your assistance to transfer our cash of twenty one Million Dollars ($21,000.000.00) now in the custody of a private Security trust firm in Europe the money is in trunk boxes deposited and declared as family valuables by my late father as a matter of fact the company does not know the content as money, although my father made them to under stand that the boxes belongs to his foreign partner. …

This mail is probably spam. The original message has been attached along with this report, so you can recognize or block similar unwanted mail in future. See http://spamassassin.org/tag/ for more details. Content analysis details: (12.20 points, 5 required) NIGERIAN_SUBJECT2 (1.4 points) Subject is indicative of a Nigerian spam FROM_ENDS_IN_NUMS (0.7 points) From: ends in numbers MIME_BOUND_MANY_HEX (2.9 points) Spam tool pattern in MIME boundary URGENT_BIZ (2.7 points) BODY: Contains urgent matter US_DOLLARS_3 (1.5 points) BODY: Nigerian scam key phrase ($NN,NNN,NNN.NN) DEAR_SOMETHING (1.8 points) BODY: Contains 'Dear (something)' BAYES_30 (1.6 points) BODY: Bayesian classifier says spam probability is 30 to 40% [score: 0.3728]

Advantages: –Fast to train (single scan). Fast to classify –Not sensitive to irrelevant features –Handles real and discrete data –Handles streaming data well Disadvantages: –Assumes independence of features Advantages/Disadvantages of Naïve Bayes

Summary of Classification We have seen 4 major classification techniques: Simple linear classifier, Nearest neighbor, Decision tree. There are other techniques: Neural Networks, Support Vector Machines, Genetic algorithms.. In general, there is no one best classifier for all problems. You have to consider what you hope to achieve, and the data itself… Let us now move on to the other classic problem of data mining and machine learning, Clustering…

Overfitting Overfitting occurs when a statistical model describes random error or noise instead of the underlying relationship. Overfitting generally occurs.

Similar presentations

Presentation on theme: "Overfitting Overfitting occurs when a statistical model describes random error or noise instead of the underlying relationship. Overfitting generally occurs."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Overfitting Overfitting occurs when a statistical model describes random error or noise instead of the underlying relationship. Overfitting generally occurs.

Similar presentations

Presentation on theme: "Overfitting Overfitting occurs when a statistical model describes random error or noise instead of the underlying relationship. Overfitting generally occurs."— Presentation transcript:

Similar presentations

About project

Feedback