Download presentation

Presentation is loading. Please wait.

Published byEugene Lambert Modified over 4 years ago

1
Industrial Engineering College of Engineering Bayesian Kernel Methods for Binary Classification and Online Learning Problems Theodore Trafalis Workshop on Clustering and Search Techniques in Large Scale Networks LATNA, Nizhny Novgorod, Russia, November 4, 2014

2
Part I. Bayesian Kernel Methods for Gaussian Processes

3
Why Bayesian Learning? Returns a probability Incorporates power of kernel methods with advantages of Bayesian updating Can incorporate prior knowledge into estimation Can “learn” fairly quickly if Gaussian process Can be used for regression or classification

4
Outline 1.Bayesics 2.Relevance Vector Machine 3.Laplace Approximation 4.Results

5
Bayes’ Rule Calculated valuePosterior PriorLikelihood

6
Logistic Likelihood

7
Prior Assume t(x) = {t(x 1 ), …, t(x m )} is a Gaussian process (normally distributed) Let t = K α

8
Maximize Posterior Goal: Find optimal values for α

9
Minimize Negative Log = 1 if y i = 0 = 1 if y i = 1

10
Minimize Negative Log

11
Relevance Vector Machine Combines Bayesian approach with sparseness of support vector machine Previously Hyperparameter 1/Variance(α i )

12
Non-Informative (Flat) Prior Let a = b ≈ 0

13
Maximize Posterior

14
Laplace Approximation Newton-Raphson Method cici C ii

15
Iteration

16
Optimizing Hyperparameter Need a closed-form expression for If α|y,s were normally distributed, then at optimal Use Gaussian approximation

17
SVM and RVM Comparison Similar accuracy with fewer “support” vectors

18
Conclusion Posterior Likelihood x Prior Gaussian process ▫ Makes math easier ▫ Assumes that density is centered around mode Relevance Vector Machine ▫ Similar accuracy to Support Vector Machine ▫ Fewer data points for RVM compared to SVM In part 2 we discuss ▫ Non-Gaussian process ▫ Markov Chain Monte Carlo solution

19
References B. Schӧlkopf and A.J. Smola, 2002. “Chapter 16: Bayesian Kernel Methods.” Learning With Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. Cambridge: MIT Press. C.M. Bishop and M.E. Tipping, 2003. “Bayesian Regression and Classification.” In J. Suykens, G. Horvath, S. Basu, C. Micchelli, and J. Vandewalle, eds. Advances in Learning Theory: Methods Models and Applications. Amsterdam: IOS Press.

21
Likelihood for Classification Logistic Probit

22
Likelihood for Regression

23
RVM for Regression where

24
Incremental Updating for Regression

25
Industrial Engineering College of Engineering Part II. Bayesian Kernel Methods Using Beta Distributions Theodore Trafalis Workshop on Clustering and Search Techniques in Large Scale Networks LATNA, Nizhny Novgorod, Russia, November 4, 2014

26
Summary of Part 1 Bayesian method: Posterior Likelihood x Prior Gaussian process ▫ Makes math easier ▫ Assumes that density is centered around mode Relevance Vector Machine Solution concept: posterior maximization

27
Current Bayesian Kernel methods Combine Bayesian probability with Kernel Methods n data points, m attributes per data point X is n x m matrix y is n x 1 vector of 0s and 1s (X) is a function of X used to predict y Posterior Prior Likelihood

28
28 MacKenzie and Trafalis Support Vector Machines

29
What’s new in part 2 Beta distributions as priors Adaptation of beta-binomial updating formula Comparison of beta kernel classifiers with existing SVM classifiers Online learning

30
Outline 1.Beta Distribution 2.Other Priors 3.Markov Chain Monte Carlo 4.Test Case

31
Likelihood Logistic Likelihood Bernoulli Likelihood

32
Beta Distribution Prior

33
33 MacKenzie and Trafalis Shape of beta distribution

34
Beta-binomial conjugate Prior Likelihood Posterior Number of ones Number of trials

35
α and β Let α i and β i be a function of x i Assume

36
α and β Let α i and β i be a function of x i Assume

37
α and β Let α i and β i be a function of x i Assume

38
α and β Let α i and β i be a function of x i Assume

39
Applying beta-binomial to data mining Prior Posterior Number of zeros in training set Parameter to be tuned

40
Classification Rule

41
41 MacKenzie and Trafalis Testing on data sets Beta prior is uniform: = 1 = 1 Rates represent mean values of percent of ones or zeros correctly classified

42
42 MacKenzie and Trafalis Online learning Updated probabilities for one data point from tornado data y = 0 y = 1 Each trial uses 100 data points to update prior

43
Conclusions Adapting the beta-binomial updating rule to a kernel-based classifier can create a fast and accurate data mining algorithm User can set prior and weights to reflect imbalanced data sets Results are comparable to weighted SVM Online learning combines previous and current information

44
Options for Prior Distributions α and β must be greater than 0 Assume and are independent Some choices

45
Kernel Function

46
Directed Acyclic Graph μαμα sr σασα μβμβ σβσβ γ x K+K+ K-K- αβ θ

47
Markov Chain Monte Carlo (MCMC) Simulation tool used for calculating posterior distributions Gibbs Sampler: iterates using conditional distributions

48
Markov Chain Monte Carlo (MCMC) Simulation tool used for calculating posterior distributions Gibbs Sampler: iterates using conditional distributions

49
Markov Chain Monte Carlo (MCMC) Simulation tool used for calculating posterior distributions Gibbs Sampler: iterates using conditional distributions

50
Markov Chain Monte Carlo (MCMC) Simulation tool used for calculating posterior distributions Gibbs Sampler: iterates using conditional distributions Software ▫ Bayesian Inference Using Gibbs Sampling (BUGS) ▫ Just Another Gibbs Sampler (JAGS)

51
Toy Example

52
Parameters for Priors

53
Large Gamma Needed

54
Results

55
Test Data Automatically Calculated

56
Comparison

57
Conclusion Advantages of Beta-Bayesian Method ▫ Incorporate non-Gaussian process ▫ Results of example equal to SVM ▫ Testing data automatically calculated with MCMC Disadvantages ▫ MCMC slow algorithm ▫ Analytical solution may not be possible ▫ Difficult to determine prior distributions Future Work ▫ Real data ▫ More comparisons with existing methods

58
References Cameron A. MacKenzie, Theodore B. Trafalis and Kash Barker, “A Bayesian Beta Kernel Model for Binary Classiﬁcation and Online Learning Problems”, Statistical Analysis and Data Mining, Vol. (In press 2014)

Similar presentations

© 2020 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google