 # Industrial Engineering College of Engineering Bayesian Kernel Methods for Binary Classification and Online Learning Problems Theodore Trafalis Workshop.

## Presentation on theme: "Industrial Engineering College of Engineering Bayesian Kernel Methods for Binary Classification and Online Learning Problems Theodore Trafalis Workshop."— Presentation transcript:

Industrial Engineering College of Engineering Bayesian Kernel Methods for Binary Classification and Online Learning Problems Theodore Trafalis Workshop on Clustering and Search Techniques in Large Scale Networks LATNA, Nizhny Novgorod, Russia, November 4, 2014

Part I. Bayesian Kernel Methods for Gaussian Processes

Why Bayesian Learning? Returns a probability Incorporates power of kernel methods with advantages of Bayesian updating Can incorporate prior knowledge into estimation Can “learn” fairly quickly if Gaussian process Can be used for regression or classification

Outline 1.Bayesics 2.Relevance Vector Machine 3.Laplace Approximation 4.Results

Bayes’ Rule Calculated valuePosterior PriorLikelihood

Logistic Likelihood

Prior Assume t(x) = {t(x 1 ), …, t(x m )} is a Gaussian process (normally distributed) Let t = K α

Maximize Posterior Goal: Find optimal values for α

Minimize Negative Log = 1 if y i = 0 = 1 if y i = 1

Minimize Negative Log

Relevance Vector Machine Combines Bayesian approach with sparseness of support vector machine Previously Hyperparameter 1/Variance(α i )

Non-Informative (Flat) Prior Let a = b ≈ 0

Maximize Posterior

Laplace Approximation Newton-Raphson Method cici C ii

Iteration

Optimizing Hyperparameter Need a closed-form expression for If α|y,s were normally distributed, then at optimal Use Gaussian approximation

SVM and RVM Comparison Similar accuracy with fewer “support” vectors

Conclusion Posterior Likelihood x Prior Gaussian process ▫ Makes math easier ▫ Assumes that density is centered around mode Relevance Vector Machine ▫ Similar accuracy to Support Vector Machine ▫ Fewer data points for RVM compared to SVM In part 2 we discuss ▫ Non-Gaussian process ▫ Markov Chain Monte Carlo solution

References B. Schӧlkopf and A.J. Smola, 2002. “Chapter 16: Bayesian Kernel Methods.” Learning With Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. Cambridge: MIT Press. C.M. Bishop and M.E. Tipping, 2003. “Bayesian Regression and Classification.” In J. Suykens, G. Horvath, S. Basu, C. Micchelli, and J. Vandewalle, eds. Advances in Learning Theory: Methods Models and Applications. Amsterdam: IOS Press.

Likelihood for Classification Logistic Probit

Likelihood for Regression

RVM for Regression where

Incremental Updating for Regression

Industrial Engineering College of Engineering Part II. Bayesian Kernel Methods Using Beta Distributions Theodore Trafalis Workshop on Clustering and Search Techniques in Large Scale Networks LATNA, Nizhny Novgorod, Russia, November 4, 2014

Summary of Part 1 Bayesian method: Posterior Likelihood x Prior Gaussian process ▫ Makes math easier ▫ Assumes that density is centered around mode Relevance Vector Machine Solution concept: posterior maximization

Current Bayesian Kernel methods Combine Bayesian probability with Kernel Methods n data points, m attributes per data point X is n x m matrix y is n x 1 vector of 0s and 1s  (X) is a function of X used to predict y Posterior Prior Likelihood

28 MacKenzie and Trafalis Support Vector Machines

What’s new in part 2 Beta distributions as priors Adaptation of beta-binomial updating formula Comparison of beta kernel classifiers with existing SVM classifiers Online learning

Outline 1.Beta Distribution 2.Other Priors 3.Markov Chain Monte Carlo 4.Test Case

Likelihood Logistic Likelihood Bernoulli Likelihood

Beta Distribution Prior

33 MacKenzie and Trafalis Shape of beta distribution

Beta-binomial conjugate Prior Likelihood Posterior Number of ones Number of trials

α and β Let α i and β i be a function of x i Assume

α and β Let α i and β i be a function of x i Assume

α and β Let α i and β i be a function of x i Assume

α and β Let α i and β i be a function of x i Assume

Applying beta-binomial to data mining Prior Posterior Number of zeros in training set Parameter to be tuned

Classification Rule

41 MacKenzie and Trafalis Testing on data sets Beta prior is uniform:  = 1  = 1 Rates represent mean values of percent of ones or zeros correctly classified

42 MacKenzie and Trafalis Online learning Updated probabilities for one data point from tornado data y = 0 y = 1 Each trial uses 100 data points to update prior

Conclusions Adapting the beta-binomial updating rule to a kernel-based classifier can create a fast and accurate data mining algorithm User can set prior and weights to reflect imbalanced data sets Results are comparable to weighted SVM Online learning combines previous and current information

Options for Prior Distributions α and β must be greater than 0 Assume and are independent Some choices

Kernel Function

Directed Acyclic Graph μαμα sr σασα μβμβ σβσβ γ x K+K+ K-K- αβ θ

Markov Chain Monte Carlo (MCMC) Simulation tool used for calculating posterior distributions Gibbs Sampler: iterates using conditional distributions

Markov Chain Monte Carlo (MCMC) Simulation tool used for calculating posterior distributions Gibbs Sampler: iterates using conditional distributions

Markov Chain Monte Carlo (MCMC) Simulation tool used for calculating posterior distributions Gibbs Sampler: iterates using conditional distributions

Markov Chain Monte Carlo (MCMC) Simulation tool used for calculating posterior distributions Gibbs Sampler: iterates using conditional distributions Software ▫ Bayesian Inference Using Gibbs Sampling (BUGS) ▫ Just Another Gibbs Sampler (JAGS)

Toy Example

Parameters for Priors

Large Gamma Needed

Results

Test Data Automatically Calculated

Comparison

Conclusion Advantages of Beta-Bayesian Method ▫ Incorporate non-Gaussian process ▫ Results of example equal to SVM ▫ Testing data automatically calculated with MCMC Disadvantages ▫ MCMC slow algorithm ▫ Analytical solution may not be possible ▫ Difficult to determine prior distributions Future Work ▫ Real data ▫ More comparisons with existing methods

References Cameron A. MacKenzie, Theodore B. Trafalis and Kash Barker, “A Bayesian Beta Kernel Model for Binary Classiﬁcation and Online Learning Problems”, Statistical Analysis and Data Mining, Vol. (In press 2014)

Download ppt "Industrial Engineering College of Engineering Bayesian Kernel Methods for Binary Classification and Online Learning Problems Theodore Trafalis Workshop."

Similar presentations