1. Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley & Sons, 2000 with the permission of the authors and the publisher

2. Chap 5: Linear Discriminant Functions (Sections 1-5) 1. Introduction 2. Linear Discriminant Functions and Decisions Surfaces 3. Generalized Linear Discriminant Functions 4. The Two-Category Linearly Separable Case 5. Minimizing the Perceptron Criterion Function

3. 2 1. Introduction In chapter 3, the underlying probability densities were known (or given) The training sample was used to estimate the parameters of these probability densities (Maximum Likelihood estimations) In this chapter, we only know the proper forms for the discriminant functions: similar to non-parametric techniques They may not be optimal, but they are simple to use They provide us with linear classifiers

4. 3 2. Linear Discriminant Functions and Decisions Surfaces The linear discriminant function is a linear combination of the components of x g(x) = w t x + w 0 (1) where w is the weight vector and w 0 the bias A two-category classifier with a discriminant function of the form (1) uses the following rule: Decide  1 if g(x) > 0 and  2 if g(x) < 0  Decide  1 if w t x > -w 0 and  2 otherwise If g(x) = 0  x is assigned to either class

5. 4

6. 5 The equation g(x) = 0 defines the decision surface that separates points assigned to the category  1 from points assigned to the category  2 When g(x) is linear, the decision surface is a hyperplane Algebraic measure of the distance from x to the hyperplane (interesting result!)

7. 6

8. 7 In conclusion, a linear discriminant function divides the feature space by a hyperplane decision surface The orientation of the surface is determined by the normal vector w and the location of the surface is determined by the bias

9. 8 The multi-category case We define c linear discriminant functions and assign x to  i if g i (x) > g j (x)  j  i; in case of ties, the classification is undefined In this case, the classifier is a “linear machine” A linear machine divides the feature space into c decision regions, with g i (x) being the largest discriminant if x is in the region R i For a two contiguous regions R i and R j ; the boundary that separates them is a portion of hyperplane H ij defined by: g i (x) = g j (x)  (w i – w j ) t x + (w i0 – w j0 ) = 0 w i – w j is normal to H ij and

10. 9

11. 10

12. 11 It is easy to show that the decision regions for a linear machine are convex, this restriction limits the flexibility and accuracy of the classifier

13. 12 3. Generalized Linear Discriminant Functions Decision boundaries which separate between classes may not always be linear The complexity of the boundaries may sometimes request the use of highly non-linear surfaces A popular approach to generalize the concept of linear decision functions is to consider a generalized decision function as: g(x) = w 1 f 1 (x) + w 2 f 2 (x) + … + w N f N (x) + w N+1 (1) where f i (x), 1  i  N are scalar functions of the pattern x, x  R n (Euclidean Space)

14. 13

15. 14 Augmented feature vector

16. 15 4. Two-Category Linearly Separable Case For two categories a “normalization” simplifies the problem Replace all samples of category two by their negatives Then we can look for a weight vector yielding a positive dot product A margin can also be included Solutions can be found with various gradient descent algorithms

17. 16

18. 17

19. 18

20. 19 5. Minimizing the Perceptron Criterion Function There are a number of Perceptron learning algorithms We will learn an early version of the two-category Fixed-Increment Single-Sample Perceptron algorithm It is like algorithm 4 in the textbook only without the normalization making the desired dot product positive The two classes will be the positive class and the negative class and we want the dot product to be positive for the positive class and negative (or zero) for the negative class This is the original (classic) Perceptron learning algorithm

21. 20

22. 21 Four learning criteria: 1. Total number of patterns misclassified 2. Perceptron criterion (Eq. 16) 3. Squared error (Eq. 32) 4. Squared error with margin (Eq. 33)

23. 22

24. 23