Introduction to Supervised Machine Learning Concepts PRESENTED BY B. Barla Cambazoglu February 21, 2014

Guest Lecturer’s Background 2

Lecture Outline 3  Basic concepts in supervised machine learning  Use case: Sentiment-focused web crawling

Basic Concepts

What is Machine Learning? 5  Wikipedia: “Machine learning is a branch of artificial intelligence, concerning the construction and study of systems that can learn from data.”  Arthur Samuel: “Field of study that gives computers the ability to learn without being explicitly programmed.”  Tom M. Mitchell: “A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.”

Unsupervised versus Supervised Machine Learning 6  Unsupervised learning › Assumes unlabeled data (the desired output is not known) › Objective is to discover the structure in the data  Supervised learning › Trained on labeled data (the desired output is known) › Objective is to generate an output for previously unseen input data

Supervised Machine Learning Applications 7  Common › Spam filtering › Recommendation and ranking › Fraud detection › Stock price prediction  Not so common › Recognize the user of a mobile device based on how he holds and moves the phone › Predict whether someone is a psychopath based on his twitter usage › Identify whales in the ocean based on audio recordings › Predict in advance whether a product launch will be successful or not

Terminology 8  Instance  Label  Feature  Training set  Test set  Learning model  Accuracy Toy problem: To predict the income level of a person based on his/her facial attributes.

Instances 9

Categorical Labels 10

Numeric Labels 11 $12K $11K $9K $8K$7K $5K $1K $2K $3K$4K

Features 12 Blonde No White No Male 5cm Bald No White Yes Male 0cm White No Black Yes Male 3cm Dark Yes White No Female 12cm

Training Set 13 Blonde No White No Male 5cm Bald No White Yes Male 0cm White No Black Yes Male 3cm Dark Yes White No Female 12cm

Test Set 14 Dark No White No Female 14cm Dark No White Yes Male 6cm Dark No Black No Male 6cm Dark Yes White No Female 15cm

Training and Testing 15 Model Training Testing Prediction Test instance Set of training instances

Accuracy 16 Actual labels Predicted labels Accuracy = # of correct predictions / total number of predictions = 2 / 4 = 50%

Precision and Recall 17  In certain cases, there are two class labels and predicting a particular class correctly is more important than predicting the other.  A good example is top-k ranking in web search.  Performance measures: › Recall › Precision

Some Practical Issues 18  Problem: Missing feature values  Solution: › Training: Use the most frequently observed (or average) feature value in the instance’s class. › Testing: Use the most frequently observed (or average) feature value in the entire training set.  Problem: Class imbalance  Solution › Oversampling: Duplicate the training instances in the small class › Undersampling: User fewer instances from the bigger class

Majority Classifier 19  Training: Find the class with the largest number of instances.  Testing: For every test instance, predict that class as the label, independent of the features of the test instance. Model PredictionTesting Class Size

k-Nearest Neighbor Classifier 20  Training: None! (known as a lazy classifier).  Testing: Find the k instances that are most similar to the test instance and use majority voting to decide on the label. k = 3

Decision Tree Classifier 21  Training: Build a tree where leaves represent labels and branches represent features that lead to those labels.  Testing: Traverse the tree using the feature values of the test instance. BlackWhite Black Not black Yes No

Naïve Bayes Classifier 22  Training: For every feature value v and class c pair, we compute and store in a lookup table the conditional probability P(v | c).  Testing: For each class c, we compute: P( | ) = 0.40 P( | ) = 0.65 P( | ) = 0.78

Other Commonly Used Classifiers 23  Support vector machines  Boosted decision trees  Neural networks

Use Case: Sentiment-Focused Web Crawling G. Vural, B. B. Cambazoglu, and P. Senkul, “Sentiment-focused web crawling”, CIKM’12, pp

Problem 25  Early discovery of the opinionated content in the Web is important.  Use cases › Measuring brand loyalty or product adoption › Politics › Finance  We would like to design a sentiment-focused web crawler that aims to maximize the amount of sentimental/opinionated content fetched from the Web within a given amount of time.

Web Crawling 26  Subspaces › Downloaded pages › Discovered pages › Undiscovered pages

Sentiment-Focused Web Crawling 27  Challenge: to predict the sentimentality of an “unseen” web page, i.e., without having access to the page content.

Features 28  Assumption: Sentimental pages are more likely to be linked by other sentimental pages.  Idea: Build a learning model using features extracted from › Textual content of referring pages › Anchor text on the hyperlinks › URL of the target page

Labels 29  Our data (ClueWeb09-B) lacks ground-truth sentiment scores.  We created a ground-truth using the SentiStrength tool. › Assigns a sentiment score (between 0 and 8) to each web page as its label.  A small scale user-study is conducted with three judges to verify the suitability of this ground-truth. › 500 random pages sampled from the collection. › pages are labeled as sentimental or not sentimental.  Observations › 22% of the pages are labeled as sentimental. › High agreement between judges: the overlap is above 85%.

Learner and Performance Metric 30  As the learner, we use the LibSVM software in the regression mode.  We rebuild the prediction model at regular intervals throughout the crawling process.  As the main performance metric, we compute the total sentimentality score accumulated after fetching a certain number of pages.

Evaluated Crawlers 31  Proposed crawlers › based on the average sentiment score of referring page content › based on machine learning  Oracle crawlers › highest sentiment score › highest spam score › highest PageRank  Baseline crawlers › random › indegree-based › breadth first

Performance 32