1. Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley & Sons, 2000 with the permission of the authors and the publisher

2. Chapter 2 (part 3) Bayesian Decision Theory (Sections 2-6,2-9) Discriminant Functions for the Normal Density Bayes Decision Theory – Discrete Features

3. Pattern Classification, Chapter 2 (Part 3) 2 Discriminant Functions for the Normal Density We saw that the minimum error-rate classification can be achieved by the discriminant function g i (x) = ln P(x |  i ) + ln P(  i ) Case of multivariate normal

4. Pattern Classification, Chapter 2 (Part 3) 3 Case  i =  2. I ( I stands for the identity matrix)

5. Pattern Classification, Chapter 2 (Part 3) 4 A classifier that uses linear discriminant functions is called “a linear machine” The decision surfaces for a linear machine are pieces of hyperplanes defined by: g i (x) = g j (x)

6. Pattern Classification, Chapter 2 (Part 3) 5

7. 6 The hyperplane separating R i and R j always orthogonal to the line linking the means!

8. Pattern Classification, Chapter 2 (Part 3) 7

9. 8

10. 9 Case  i =  (covariance of all classes are identical but arbitrary!) Hyperplane separating R i and R j (the hyperplane separating R i and R j is generally not orthogonal to the line between the means!)

11. Pattern Classification, Chapter 2 (Part 3) 10

12. Pattern Classification, Chapter 2 (Part 3) 11

13. Pattern Classification, Chapter 2 (Part 3) 12 Case  i = arbitrary The covariance matrices are different for each category (Hyperquadrics which are: hyperplanes, pairs of hyperplanes, hyperspheres, hyperellipsoids, hyperparaboloids, hyperhyperboloids)

14. Pattern Classification, Chapter 2 (Part 3) 13

15. Pattern Classification, Chapter 2 (Part 3) 14

16. Pattern Classification, Chapter 2 (Part 3) 15 Bayes Decision Theory – Discrete Features Components of x are binary or integer valued, x can take only one of m discrete values v 1, v 2, …, v m Case of independent binary features in 2 category problem Let x = [x 1, x 2, …, x d ] t where each x i is either 0 or 1, with probabilities: p i = P(x i = 1 |  1 ) q i = P(x i = 1 |  2 )

17. Pattern Classification, Chapter 2 (Part 3) 16 The discriminant function in this case is:

18. Pattern Classification, Chapter 2 (Part 3) 17 Bayesian Belief Network Features Causal relationships Statistically independent Bayesian belief nets Causal networks Belief nets

19. Pattern Classification, Chapter 2 (Part 3) 18 x 1 and x 3 are independent

20. Pattern Classification, Chapter 2 (Part 3) 19 Structure Node Discrete variables Parent, Child Nodes Direct influence Conditional Probability Table Set by expert or by learning from training set (Sorry, learning is not discussed here)

21. Pattern Classification, Chapter 2 (Part 3) 20

22. Pattern Classification, Chapter 2 (Part 3) 21 Examples

23. Pattern Classification, Chapter 2 (Part 3) 22

24. Pattern Classification, Chapter 2 (Part 3) 23

25. Pattern Classification, Chapter 2 (Part 3) 24 Evidence e

26. Pattern Classification, Chapter 2 (Part 3) 25 Ex. 4. Belief Network for Fish AB X CD P(a) P(b) P(c|x) P(d|x) P(x|a,b) b 1 =north Atlantic, 0.6 b 2 =south Atlantic, 0.4 a 1 =winter, 0.25 a 2 =spring, 0.25 a 3 =summer,0.25 a 4 =autumn, 0.25 x 1 =salmon x 2 =sea bass d 1 =wide, d 2 =thin x1 0.3, 0.7 x2 0.6, 0.4 c 1 =light,c 2 =medium, c 3 =dark x 1 0.6, 0.2, 0.2 x 2 0.2, 0.3, 0.5 x1x2 a1b1 0.5 a1b2 0.70.3 a2b1 0.60.4 a2b2 0.80.2 a3b1 0.40.6 a3b2 0.10.9 a4b1 0.20.8 a4b2 0.30.7

27. Pattern Classification, Chapter 2 (Part 3) 26 Belief Network for Fish Fish was caught in the summer in the north Atlantic and is a see bass that is dark and thin P(a 3,b 1,x 2,c 3,d 2 ) = P(a 3 )P(b 1 )P(x 2 |a 3,b 1 )P(c 3 |x 2 )P(d 2 |x 2 ) =0.25*0.6*0.4*0.5*0.4 =0.012

28. Pattern Classification, Chapter 2 (Part 3) 27 Light, south Atlantic, fish?

29. Pattern Classification, Chapter 2 (Part 3) 28 Normalize

30. Pattern Classification, Chapter 2 (Part 3) 29 Conditionally Independent

31. Pattern Classification, Chapter 2 (Part 3) 30 Medical Application Medical diagnosis Uppermost nodes: biological agent (virus or bacteria) Intermediate nodes: diseases (flu or emphysema) Lowermost nodes: symptoms (high temperature or coughing) Finds the most likely disease or cause By entering measured values

32. Pattern Classification, Chapter 2 (Part 3) 31 Exercise 50 (based on Ex. 4) (a) December 20, north Atlantic, thin P(a1)=P(a4)=0.5, P(b1)=1, P(d2)=1 Fish? Error rate? (b) Thin, medium lightness Season? Probability? (c) Thin, medium lightness, north atlantic Season?, probability?