1. PatReco: Bayes Classifier and Discriminant Functions Alexandros Potamianos Dept of ECE, Tech. Univ. of Crete Fall 2009-2010

2. PatReco: Problem Solving 1.Data Collection 2.Data Analysis 3.Feature Selection 4.Model Selection 5.Model Training 6.Classification 7.Classifier Evaluation

3. Bayes Classifier  Classes: ω 1, ω 2, … ω n  Sample: x = (x 1, x 2, … x d ) [ d-Dimensional features ]  Model: p(x|ω 1 ), p(x|ω 2 ), … p(x|ω n ) p(ω 1 ), p(ω 2 ), … p(ω n ) ω  Bayes classifier ( classify sample x to class ω ): ω ω = arg max ωi p(ω i |x) = arg max ωi p(x|ω i ) p(ω i )

7. Bayes Error  Classes: ω 1, ω 2, … ω n  Sample: x = (x 1, x 2, … x d ) [ d-Dimensional features ]  Model: p(x|ω 1 ), p(x|ω 2 ), … p(x|ω n ) p(ω 1 ), p(ω 2 ), … p(ω n )  Decision regions: Ω 1, Ω 2,... Ω n  Bayes error ( probability of wrong classification ): P(error) P(error) = 1 - P(correct) = = 1 -  i  Ωi p ( x|ω 1 ) p(ω 1 ) dx

9. Discriminant Functions  Define class boundaries (instead of class characteristics)  Dualism: Parametric class description  Bayes classifier  Decision boundary  Parametric Discriminant Functions

10. Normal Density  1D  Multi-D Full covariance Diagonal covariance Diagonal covariance + univariate  Mixture of Gaussians Usually diagonal covariance

12. Gaussian Discriminant Functions  Same variance ALL classes Hyper-planes  Different variance among classes Hyper-quadratics (hyper-parabolas, hyper- ellipses etc.)

15. Hyper-Planes  When the covariance matrix is common across Gaussian classes The decision boundary is a hyper-plane that is vertical to the line connecting the means of the Gaussian distributions If the a-priori probabilities of classes are equal the hyper-planes cuts the line connecting the Gaussian means in the middle  Euclidean classifier

16. Gaussian Discriminant Functions  Same variance ALL classes Hyper-planes  Different variance among classes Hyper-quadratics (hyper-parabolas, hyper- ellipses etc.)

21. Hyper-Quadratics  When the Gaussian class variances are different the boundary can be hyper-plane, multiple hyper-planes, hyper-sphere, hyper- parabola, hyper-elipsoid etc. The boundary in general in NOT vertical to the Gaussian mean connecting line If the a-priori probabilities of classes are equal the resulting classifier is a Mahalanobois classifier

22. Conclusions  Parametric statistical models describe class characteristics x by modeling the observation probabilities p(x|class)  Discriminant functions describe class boundaries parametrically  Parametric statistical models have an equivalent parametric discriminant function  For Gaussian p(x|class) distributions the decision boundaries are hyper-planes or hyper-quadratics