Artificial Intelligence 8. Supervised and unsupervised learning Japan Advanced Institute of Science and Technology (JAIST) Yoshimasa Tsuruoka

Outline Supervised learning Naive Bayes classifier Unsupervised learning Clustering Lecture slides

Supervised and unsupervised learning Supervised learning – Each instance is assigned with a label – Classification, regression – Training data need to be created manually Unsupervised learning – Each instance is just a vector of attribute-values – Clustering – Pattern mining

Naive Bayes classifier Chapter 6.9 of Mitchell, T., Machine Learning (1997) Naive Bayes classifier – Output probabilities – Easy to implement – Assumes conditional independence between features – Efficient learning and classification

Thomas Bayes (1702 – 1761) The reverse conditional probability can be calculated using the original conditional probability and prior probabilities. Bayes’ theorem

Can we know the probability of having cancer from the result of a medical test? Bayes’ theorem

The probability of actually having cancer is not very high. Bayes’ theorem

Naive Bayes classifier Assume that features are conditionally independent Bayes’ theorem The denominator is constant. Conditional independence

Training data DayOutlookTemperatureHumidityWindPlayTennis D1SunnyHotHighWeakNo D2SunnyHotHighStrongNo D3OvercastHotHighWeakYes D4RainMildHighWeakYes D5RainCoolNormalWeakYes D6RainCoolNormalStrongNo D7OvercastCoolNormalStrongYes D8SunnyMildHighWeakNo D9SunnyCoolNormalWeakYes D10RainMildNormalWeakYes D11SunnyMildNormalStrongYes D12OvercastMildHighStrongYes D13OvercastHotNormalWeakYes D14RainMildHighStrongNo

Naive Bayes classifier Instance

Class prior probability Maximum likelihood estimation – Just counting the number of occurrences in the training data

Conditional probabilities of features Maximum likelihood

Class posterior probabilities Normalize

Smoothing Maximum likelihood estimation – Estimated probabilities are not reliable when n c is small m-estimate of probability ： prior probability ： equivalent sample size

Text classification with a Naive Bayes classifier Text classification – Automatic classification of news articles – Spam filtering – Sentiment analysis of product reviews – etc.

There were doors all round the hall, but they were all locked; and when Alice had been all the way down one side and up the other, trying every door, she walked sadly down the middle, wondering how she was ever to get out again.

Cannot be estimated reliably Ignore the position and apply m-estimate smoothing Conditional probabilities of words The probability of the second word of the document being the word “were”

Unsupervised learning No “correct” output for each instance Clustering – Merging “similar” instances into a group – Hierarchical clustering, k-means, etc.. Pattern mining – Discovering frequent patterns from a large amount of data – Association rules, graph mining, etc

Clustering Organize instances into groups whose members are similar in some way

Agglomerative clustering Define a distance between every pair of instances – E.g. cosine similarity Algorithm 1.Start with every instance representing a singleton cluster 2.The closest two clusters are merged into a single cluster 3.Repeat this process until all clusters are merged

Agglomerative clustering Example Dendrogram

Defining a distance between clusters single linkcomplete link group-averagecentroid

k-means algorithm Centroids Minimize Algorithm 1.Choose k centroids c 1,…c k randomely 2.Assign each instance to the cluster with the closest centroid 3.Update the centroids and go back to Step 2