1. Quadratic Classifiers (QC) J.-S. Roger Jang ( 張智星 ) CS Dept., National Taiwan Univ. http://mirlab.org/jangjang@mirlab.org 2010 Scientific Computing

2. 2 Bayes Classifier Bayes classifier  A probabilistic framework for classification problem Conditional probability Bayes theorem 2015/12/4 2

3. 2010 Scientific Computing 3 2015/12/4 3 PDF Modeling Goal: Find a PDF (probability density function) that can best describe a given dataset Steps: Select a class of parameterized PDF Identify the parameters via MLE (maximum likelihood estimate) based on a given set of sample data Commonly used PDFs: Multi-dimensional Gaussian PDF Gaussian mixture models (GMM)

4. 2010 Scientific Computing 4 2015/12/4 4 PDF Modeling for Classification Procedure for classification based on PDF Training stage: PDF modeling of each class based on the training dataset Test stage: For each entry in the test dataset, pick the class with the max. PDF Commonly used classifiers: Quadratic classifier (n-dim. Gaussian PDF) Gaussian-mixture-model classifier (GMM PDF)

5. 2010 Scientific Computing 5 2015/12/4 5 1D Gaussian PDF Modeling 1D Gaussian PDF: MLE of  and  Detailed derivation

6. 2010 Scientific Computing 6 2015/12/4 6 1D Gaussian PDF Modeling via MLE MLE: Maximum Likelihood Estimate Given a set of observations, find the parameters of the PDF such that the overall likelihood is maximized. Detailed derivation

7. 2010 Scientific Computing 7 2015/12/4 7 1D Gaussian PDF Modeling via MLE Normal dist. estimated by normal dist. Uniform dist. estimated by normal dist.

8. 2010 Scientific Computing 8 2015/12/4 8 d-dim. Gaussian PDF Modeling d-dim. Gaussian PDF g(x, ,  ) MLE of  and  : Detailed derivation

9. 2010 Scientific Computing 9 2015/12/4 9 d-dim. Gaussian PDF Modeling d-dim. Gaussian PDF g(x, ,  ) Likelihood of x in class j (governed by g(x,  j,  j ))

10. 2010 Scientific Computing 10 2015/12/4 10 2D Gaussian PDF Bivariate normal density: Density functionContours

11. 2010 Scientific Computing 11 2D Gaussian PDF Modeling gaussianMle.m 2015/12/4 11

12. 2010 Scientific Computing 12 Steps of QC Training stage Select a type of Gaussian PDF Identify the PDF of each class Test stage Assign each sample to the class with the highest PDF value 2015/12/4 12

13. 2010 Scientific Computing 13 Characteristics of QC If each class is modeled by an Gaussian PDF, the decision boundary between any two classes is a quadratic function. That is why it is called quadratic classifier. How to prove it? Different selections of the covariance matrix: Constant times an identity matrix Diagonal matrix Full matrix (hard to use if the input dimension is large) 2015/12/4 13

14. 2010 Scientific Computing 14 QC Results on Iris Dataset (I) Dataset: IRIS dataset with the last two inputs 2015/12/4 14

15. 2010 Scientific Computing 15 QC Results on Iris Dataset(II) PDF for each class: 2015/12/4 15

16. 2010 Scientific Computing 16 QC Results on Iris Dataset (III) Decision boundaries among classes: 2015/12/4 16