2. Outline Classification Linear classifiers Perceptron Multi-class classification Generative approach Naïve Bayes classifier 2

3. Classification: Oranges and Lemons 3

4. 4

5. Classification problem 5

6. Linear classifiers 6

7. 7

8. Decision boundary 8

9. Linear Decision boundary (Perceptron) 9

10. Linear Decision boundary (Decision Tree) 10 t1t3 t2 Income

11. Linear Decision boundary (K Nearest Neighbor) 11 O O O x x x Feature 1 Feature 2

12. Non-Linear Decision boundary 12 Decision Boundary Decision Region 1 Decision Region 2

13. Decision boundary Linear classifier 13

14. Non-linear decision boundary Choose non-linear features Classifier still linear in parameters 14

15. Linear boundary: geometry 15

16. SSE cost function for classification SSE cost function is not suitable for classification Sum of Squared Errors loss penalizes “too correct” predictions SSE also lack robustness to noise 16

17. SSE cost function for classification 17

18. Perceptron algorithm 18

19. Perceptron criterion 19

20. Batch gradient for descent Perceptron “Gradient Descent” to solve the optimization problem Batch Perceptron converges in finite number of steps for linearly separable data 20

21. Stochastic gradient descent for Perceptron 21

22. Convergence of Perceptron 22

23. Convergence of Perceptron 23

24. Multi-class classification 24

25. Multi-class classification One-vs-all (one-vs-rest) 25

26. Multi-class classification One-vs-one 26

27. Multi-class classification: ambiguity regions in which the classification is undefined Converting the multi-class problem to a set of two- class problems can lead to regions in which the classification is undefined 27

28. Probabilistic approach Bayes’ theorem 28

29. Bayes’ theorem 29

30. Bayes decision theory 30

31. Probabilistic classifiers Probabilistic classification approaches can be divided in two main categories Generative Discriminative 31

32. Discriminative vs. generative approach 32

33. Generative approach 33

34. Discriminative approach 34

35. Naïve Bayes classifier 35

36. Naïve Bayes classifier 36

37. Naïve Bayes: discrete example 37

38. 38