1. Bayesian Classification with a brief introduction to pattern recognition Modified from slides by Michael L. Raymer, Ph.D.

2. 8/29/03M. Raymer – WSU, FBS2 The pattern recognition paradigm Fruit on an assembly line  Oranges, grapefruit, lemons, cherries, apples Sensors measure:  Red intensity  Yellow intensity  Mass (kg)  Approximate volume At the end of the line, a gate switches to deposit the fruit into the correct bin

3. 8/29/03M. Raymer – WSU, FBS3 Training the algorithm Red = 2.125 Yellow = 6.143 Mass = 134.32 Volume = 24.21 Apple Sensors, scales, etc…

4. 8/29/03M. Raymer – WSU, FBS4 Training (2) Red = 2.125 Yellow = 6.143 Mass = 134.32 Volume = 24.21 Apple Classifier

5. 8/29/03M. Raymer – WSU, FBS5 Testing Red = 2.125 Yellow = 6.143 Mass = 134.32 Volume = 24.21 ?? Classifier !

6. 8/29/03M. Raymer – WSU, FBS6 Pattern Matrix V1V2V3V4V5 Ex 13.062.056.397.846.75 Ex 28.250.722.520.509.08 Ex 32.729.325.687.837.86 Ex 47.371.302.970.613.49 Ex 50.731.466.606.080.78 Ex 64.855.084.878.068.65 Ex 75.891.236.382.816.84 Ex 80.526.574.083.620.59 Ex 95.663.656.876.907.93 Ex 103.920.731.013.572.47 Ex 118.841.422.793.403.19 Ex 125.634.328.080.824.74 Class 1 1 1 2 2 2 3 3 3 4 4 4

7. 8/29/03M. Raymer – WSU, FBS7 Nearest Neighbor Classification Mass (normalized) 012345678910 12345678910 Red Intensity (normalized) ?

8. 8/29/03M. Raymer – WSU, FBS8 Evaluating Accuracy Training data Testing data

9. 8/29/03M. Raymer – WSU, FBS9 Problems with KNN classifiers Lots of memorization Slow (lots of distance calculations) Incorrect features cause problems Features are assumed to all be of equal importance in classification Odd exemplars (e.g. green/yellow apples) cause problems What value for k?

10. 8/29/03M. Raymer – WSU, FBS10 Distributions Bayesian classifiers start with an estimate of the distribution of the features Binomial Distribution (Discrete) Gaussian Distribution (Continuous)

11. 8/29/03M. Raymer – WSU, FBS11 Density Estimation Parametric  Assume a Gaussian (e.g.) distribution.  Estimate the parameters ( ,  ). Non-parametric  Histogram sampling  Bin size is critical  Gaussian smoothing can help

12. 8/29/03M. Raymer – WSU, FBS12 The Gaussian distribution Multivariate (d-dimensional): Univariate: A parametric Bayesian classifier must estimate  and  from the training samples.  

13. 8/29/03M. Raymer – WSU, FBS13 Making decisions Once you have the distributions for  Each feature and  Each class You can ask questions like… If I have an apple, what is the probability that the diameter will be between 3.2 and 3.5 inches?

14. 8/29/03M. Raymer – WSU, FBS14 More decisions… Non-parametricParametric Diameter Count  

15. 8/29/03M. Raymer – WSU, FBS15 A Simple Example You are given a fruit with a diameter of 4” – is it a pear or an apple? To begin, we need to know the distributions of diameters for pears and apples.

16. 8/29/03M. Raymer – WSU, FBS16 Maximum Likelihood P(x)P(x) Class- Conditional Distributions 1” 2” 3” 4” 5” 6”

17. 8/29/03M. Raymer – WSU, FBS17 What are we asking? If the fruit is an apple, how likely is it to have a diameter of 4”? If the fruit is a xenofruit from planet Xircon, how likely is it to have a diameter of 4”? Is this the right question to ask?

18. 8/29/03M. Raymer – WSU, FBS18 A Key Problem We based this decision on (class conditional) What we really want to use is (posterior probability) What if we found the fruit in a pear orchard? We need to know the prior probability of finding an apple or a pear!

19. 8/29/03M. Raymer – WSU, FBS19 Statistical decisions… If a fruit has a diameter of 4”, how likely is it to be an apple? Apples 4” Fruit

20. 8/29/03M. Raymer – WSU, FBS20 “Inverting” the question Given an apple, what is the probability that it will have a diameter of 4”? Given a 4” diameter fruit, what is the probability that it is an apple?

21. 8/29/03M. Raymer – WSU, FBS21 Prior Probabilities Prior probability + Evidence  Posterior Probability Without evidence, what is the “prior probability” that a fruit is an apple?

22. 8/29/03M. Raymer – WSU, FBS22 The heart of it all Bayes Rule

23. 8/29/03M. Raymer – WSU, FBS23 Bayes Rule or

24. 8/29/03M. Raymer – WSU, FBS24 Example Revisited Is it an ordinary apple or an uncommon pear?

25. 8/29/03M. Raymer – WSU, FBS25 Bayes Rule Example

26. 8/29/03M. Raymer – WSU, FBS26 Bayes Rule Example

27. 8/29/03M. Raymer – WSU, FBS27 Solution

28. 8/29/03M. Raymer – WSU, FBS28 Marginal Distributions

29. 8/29/03M. Raymer – WSU, FBS29 Combining Marginals Assuming independent features: If we assume independence and use Bayes rule, we have a Naïve Bayes decision maker (classifier).

30. 8/29/03M. Raymer – WSU, FBS30 Bayes Decision Rule Provably optimal when the features (evidence) follow Gaussian distributions, and are independent.

31. 8/29/03M. Raymer – WSU, FBS31 Likelihood Ratios When deciding between two possibilities, we don’t need the exact probabilities. We only need to know which one is greater. The denominator for all the classes is always equal.  Can be eliminated  Useful when there are many possible classes

32. 8/29/03M. Raymer – WSU, FBS32 Likelihood Ratio Example 

33. 8/29/03M. Raymer – WSU, FBS33 Likelihood Ratio Example

34. 8/29/03M. Raymer – WSU, FBS34 In-class example: OrangesGrapefruit

35. 8/29/03M. Raymer – WSU, FBS35 Example (cont’d) After observing several hundred fruit pass down the assembly line, we observe that  72% are oranges  28% are grapefruit Fruit ‘x’  Red intensity = 8.2  Mass = 7.6 What shall we predict for the class of fruit ‘x’?

36. 8/29/03M. Raymer – WSU, FBS36 The whole enchilada and… (Naïve assumption) Repeat for grapefruit and predict the more probable class.

37. 8/29/03M. Raymer – WSU, FBS37 The whole enchilada (2)

38. 8/29/03M. Raymer – WSU, FBS38 The whole enchilada (3)

39. 8/29/03M. Raymer – WSU, FBS39 Conclusion Predict that fruit ‘x’ is a grapefruit, despite the relative scarcity of grapefruits on the conveyor belt.

40. 8/29/03M. Raymer – WSU, FBS40 Abbreviated Since the denominator is the same for all classes, we can just compare: and

41. 8/29/03M. Raymer – WSU, FBS41 Likelihood comparison