Outline Classification Linear classifiers Perceptron Multi-class classification Generative approach Naïve Bayes classifier 2

Classification: Oranges and Lemons 3

4

Classification problem 5

Linear classifiers 6

7

Decision boundary 8

Linear Decision boundary (Perceptron) 9

Linear Decision boundary (Decision Tree) 10 t1t3 t2 Income

Linear Decision boundary (K Nearest Neighbor) 11 O O O x x x Feature 1 Feature 2

Non-Linear Decision boundary 12 Decision Boundary Decision Region 1 Decision Region 2

Decision boundary Linear classifier 13

Non-linear decision boundary Choose non-linear features Classifier still linear in parameters 14

Linear boundary: geometry 15

SSE cost function for classification SSE cost function is not suitable for classification Sum of Squared Errors loss penalizes “too correct” predictions SSE also lack robustness to noise 16

SSE cost function for classification 17

Perceptron algorithm 18

Perceptron criterion 19

Batch gradient for descent Perceptron “Gradient Descent” to solve the optimization problem Batch Perceptron converges in finite number of steps for linearly separable data 20

Stochastic gradient descent for Perceptron 21

Convergence of Perceptron 22

Convergence of Perceptron 23

Multi-class classification 24

Multi-class classification One-vs-all (one-vs-rest) 25

Multi-class classification One-vs-one 26

Multi-class classification: ambiguity regions in which the classification is undefined Converting the multi-class problem to a set of two- class problems can lead to regions in which the classification is undefined 27

Probabilistic approach Bayes’ theorem 28

Bayes’ theorem 29

Bayes decision theory 30

Probabilistic classifiers Probabilistic classification approaches can be divided in two main categories Generative Discriminative 31

Discriminative vs. generative approach 32

Generative approach 33

Discriminative approach 34

Naïve Bayes classifier 35

Naïve Bayes classifier 36

Naïve Bayes: discrete example 37

38