1. Indian Statistical Institute Kolkata
Supervised Learning Regression, Classification Linear regression, k-NN classification
Debapriyo Majumdar
Data Mining – Fall 2014
Indian Statistical Institute Kolkata
August 11, 2014

2. An Example: Size of Engine vs Power
Power (bhp)
Engine displacement (cc)
An unknown car has an engine of size 1800cc. What is likely to be the power of the engine?

3. An Example: Size of Engine vs Power
Power (bhp)
Target
Variable
Engine displacement (cc)
Intuitively, the two variables have a relation
Learn the relation from the given data
Predict the target variable after learning

4. Exercise: on a simpler set of data pointsy 1 2 3 7 4 10
2.5 ? y x
Predict y for x = 2.5

5. Engine displacement (cc)
Linear Regression
Training set
Power (bhp)
Engine displacement (cc)
Assume: the relation is linear
Then for a given x (=1800), predict the value of y

6. Engine displacement (cc)
Linear Regression
Engine (cc)
Power (bhp)
800 60
1000 90
1200 80
100 75
1400
1500
120
1800
160
2000
140
170
2400
180
Power (bhp)
Engine displacement (cc)
Optional exercise
Linear regression
Assume y = a . x + b
Try to find suitable a and b

7. Exercise: using Linear Regressiony 1 2 3 7 4 10
2.5 ? y x
Define a regression line of your choice
Predict y for x = 2.5

8. Choosing the parameters right
Goal: minimizing the deviation from the actual data points y x
The data points: (x1, y1), (x2, y2), … , (xm, ym)
The regression line: f(x) = y = a . x + b
Least-square cost function: J = Σi ( f(xi) – yi )2
Goal: minimize J over choices of a and b

9. How to Minimize the Cost Function?b a
Goal: minimize J for all values of a and b
Start from some a = a0 and b = b0
Compute: J(a0,b0)
Simultaneously change a and b towards the negative gradient and eventually hope to arrive an optimal
Question: Can there be more than one optimal? Δ

10. Another example: Y
Training set
High blood sugar N
Age
Given that a person’s age is 24, predict if (s)he has high blood sugar
Discrete values of the target variable (Y / N)
Many ways of approaching this problem

11. Classification problemY
High blood sugar N ? 24
Age
One approach: what other data points are nearest to the new point?
Other approaches?

12. Classification Algorithms
The k-nearest neighbor classification
Naïve Bayes classification
Decision Tree
Linear Discriminant Analysis
Logistics Regression
Support Vector Machine

13. Classification or Regression?
Given data about some cars: engine size, number of seats, petrol / diesel, has airbag or not, price
Problem 1: Given engine size of a new car, what is likely to be the price?
Problem 2: Given the engine size of a new car, is it likely that the car is run by petrol?
Problem 3: Given the engine size, is it likely that the car has airbags?

14. Classification

15. Example: Age, Income and Owning a flat
Training set
Owns a flat
Does not own a flat
Monthly income (thousand rupees)
Age
Given a new person’s age and income, predict – does (s)he own a flat?

16. Example: Age, Income and Owning a flat
Training set
Owns a flat
Does not own a flat
Monthly income (thousand rupees)
Age
Nearest neighbor approach
Find nearest neighbors among the known data points and check their labels

17. Example: Age, Income and Owning a flat
Training set
Owns a flat
Does not own a flat
Monthly income (thousand rupees)
Age
The 1-Nearest Neighbor (1-NN) Algorithm:
Find the closest point in the training set
Output the label of the nearest neighbor

18. The k-Nearest Neighbor Algorithm
Training set
Owns a flat
Does not own a flat
Monthly income (thousand rupees)
Age
The k-Nearest Neighbor (k-NN) Algorithm:
Find the closest k point in the training set
Majority vote among the labels of the k points

19. The Maholanobis distance (1936)
Distance measures
How to measure distance to find closest points?
Euclidean: Distance between vectors x = (x1, … , xk) and y = (y1, … , yk)
Manhattan distance:
Generalized squared interpoint distance: S is the covariance matrix
The Maholanobis distance (1936)

20. Classification setup
Training data / set: set of input data points and given answers for the data points
Labels: the list of possible answers
Test data / set: inputs to the classification algorithm for finding labels
Used for evaluating the algorithm in case the answers are known (but known to the algorithm)
Classification task: Determining labels of the data points for which the label is not known or not passed to the algorithm
Features: attributes that represent the data

21. Evaluation Test set accuracy: the correct performance measure
Accuracy = #of correct answer / #of all answers
Need to know the true test labels
Option: use training set itself
Parameter selection (for k-NN) by accuracy on training set
Overfitting: a classifier performs too good on training set compared to new (unlabeled) test data

22. Better validation methods
Leave one out:
For each training data point x of training set D
Construct training set D – x, test set {x}
Train on D – x, test on x
Overall accuracy = average over all such cases
Expensive to compute
Hold out set:
Randomly choose x% (say 25-30%) of the training data, set aside as test set
Train on the rest of training data, test on the test set
Easy to compute, but tends to have higher variance

23. The k-fold Cross Validation Method
Randomly divide the training data into k partitions D1,…, Dk : possibly equal division
For each fold Di
Train a classifier with training data = D – Di
Test and validate with Di
Overall accuracy: average accuracy over all cases

24. References Data Mining Map: http://www.saedsayad.com/
Lecture videos by Prof. Andrew Ng, Stanford University
Available on Coursera (Course: Machine Learning)
Data Mining Map: