Introduction to Radial Basis Function Networks

Contents Overview The Models of Function Approximator The Radial Basis Function Networks RBFN’s for Function Approximation The Projection Matrix Learning the Kernels Bias-Variance Dilemma The Effective Number of Parameters Model Selection

RBF Linear models have been studied in statistics for about 200 years and the theory is applicable to RBF networks which are just one particular type of linear model. However, the fashion for neural networks which started in the mid-80 has given rise to new names for concepts already familiar to statisticians

Typical Applications of NN Pattern Classification Function Approximation Time-Series Forecasting

Function Approximation Unknown f Fonksiyon yaklaşımı yöntemleri genel olarak ikiye ayrılır. 1. Bir aralık üzerinde bilinen bir fonksiyona başka bir fonksiyon ile yaklaşım 2. Belli noktalarda değerleri bilinen bir fonksiyona başka bir fonksiyon ile yaklaşım. En iyi yaklaşım polinom yaklaşım fonksiyonları ve kare yaklaşım ek olarak belirtilen durumlar arasındaki ilişkiyi genel olarak doğrusal bir ilişki değildir. Approximator ˆ f

Introduction to Radial Basis Function Networks The Model of Function Approximator

Linear Models Weights Fixed Basis Functions

Linearly weighted output Linear Models y 1 2 m x1 x2 xn w1 w2 wm x = Linearly weighted output Output Units Decomposition Feature Extraction Transformation Hidden Units Inputs Feature Vectors

Linearly weighted output Linear Models Can you say some bases? y Linearly weighted output Output Units w1 w2 wm Decomposition Feature Extraction Transformation Hidden Units 1 2 m Inputs Feature Vectors x = x1 x2 xn

Example Linear Models Are they orthogonal bases? Polynomial Fourier Series

Single-Layer Perceptrons as Universal Aproximators 1 2 m x1 x2 xn w1 w2 wm x = With sufficient number of sigmoidal units, it can be a universal approximator. Hidden Units

Radial Basis Function Networks as Universal Aproximators y 1 2 m x1 x2 xn w1 w2 wm x = With sufficient number of radial-basis-function units, it can also be a universal approximator. Hidden Units

Non-Linear Models Weights Adjusted by the Learning process

Introduction to Radial Basis Function Networks The Radial Basis Function Networks

Radial Basis Functions Three parameters for a radial function: i(x)= (||x  xi||) xi Center Distance Measure Shape r = ||x  xi|| 

Typical Radial Functions Gaussian Hardy-Multiquadratic (1971) Inverse Multiquadratic r verinin seçilen merkeze(center) uzaklıklığı Standard sapma E = (standardsapma)^2 – değişim C – shape parameter that affects the shape of surface. : Bir şekil parametresidir (örneğin, bir oranı parametre olarak veya sadece bu, aşağıdakilerden birini ya da her ikisinin bir fonksiyonu), bir yer parametresi de bir ölçü parametresi, ne bir olasılık dağılımının bir parametredir. Böyle bir parametre sadece (bir konum parametresi yaptığı gibi) o kayması ya da (bir ölçek parametresi yaptığı gibi) bu küçülen / germe yerine dağıtım şeklini etkileyen gerekir.

Gaussian Basis Function (=0.5,1.0,1.5)

Inverse Multiquadratic

Basis {i: i =1,2,…} is `near’ orthogonal. Most General RBF + + +

Properties of RBF’s On-Center, Off Surround Analogies with localized receptive fields found in several biological structures, e.g., visual cortex; ganglion cells

The Topology of RBF As a function approximator x1 x2 xn y1 ym Output Units Interpolation Hidden Units Projection Interpolation : ekleme yapma, ara değerleri bulma Inputs Feature Vectors

The Topology of RBF As a pattern classifier. x1 x2 xn y1 ym Output Units Classes Hidden Units Subclasses Inputs Feature Vectors

Introduction to Radial Basis Function Networks RBFN’s for Function Approximation

Radial Basis Function Networks Radial basis function (RBF) networks are feed-forward networks trained using a supervised training algorithm. The activation function is selected from a class of functions called basis functions. They usually train much faster than BP. They are less susceptible to problems with non-stationary inputs Bunlar, sabit olmayan girişli sorunlarına karşı daha az hassas olan networks

Radial Basis Function Networks Popularized by Broomhead and Lowe (1988), and Moody and Darken (1989), RBF networks have proven to be a useful neural network architecture. The major difference between RBF and BP is the behavior of the single hidden layer. Rather than using the sigmoidal or S-shaped activation function as in BP, the hidden units in RBF networks use a Gaussian or some other basis kernel function. S-shaped- Sigmoid function Gaussian function Kernel (statistics) Polynomial kernel Radial basis function Laplacian

The idea y Unknown Function to Approximate Training Data x

Basis Functions (Kernels) The idea y Unknown Function to Approximate Training Data x Basis Functions (Kernels)

Basis Functions (Kernels) The idea y Function Learned x Basis Functions (Kernels)

Basis Functions (Kernels) The idea Nontraining Sample y Function Learned x Basis Functions (Kernels)

The idea Nontraining Sample y Function Learned x

Radial Basis Function Networks as Universal Aproximators Training set x1 x2 xn w1 w2 wm x = Goal for all k

Learn the Optimal Weight Vector Training set x1 x2 xn x = Goal for all k w1 w2 wm

Regularization Training set If regularization is unneeded, set Goal for all k

Learn the Optimal Weight Vector Minimize

Learn the Optimal Weight Vector Define

Learn the Optimal Weight Vector Define

Learn the Optimal Weight Vector

Learn the Optimal Weight Vector Design Matrix Variance Matrix

Introduction to Radial Basis Function Networks The Projection Matrix

The Empirical-Error Vector Unknown Function

The Empirical-Error Vector Unknown Function Error Vector

Sum-Squared-Error Error Vector If =0, the RBFN’s learning algorithm is to minimize SSE (MSE). Sum-Squared-Error Error Vector

The Projection Matrix Error Vector

Introduction to Radial Basis Function Networks Learning the Kernels Çekirdeği Öğrenme genel çekirdek yöntemleri ve makine öğrenme için temel bir konudur. Uygun çekirdek seçme sorusu DVM özellikle, çekirdek yöntemleri başından beri gündeme gelmiştir. Bu alanda önemli gelişmeler hem makine öğrenme teknikleri uygularken kullanıcıların gereksinimlerini azaltmak ve daha iyi performans elde etmenize yardımcı olacaktır

RBFN’s as Universal Approximators xn y1 yl 1 2 m w11 w12 w1m wl1 wl2 wlml Training set Kernels

What to Learn? x1 x2 xn y1 yl Weights wij’s Centers j’s of j’s 1 2 m w11 w12 w1m wl1 wl2 wlml Weights wij’s Centers j’s of j’s Widths j’s of j’s Number of j’s  Model Selection

One-Stage Learning

The simultaneous updates of all three sets of parameters may be suitable for non-stationary environments or on-line setting. One-Stage Learning

Two-Stage Training x1 x2 xn y1 yl Step 2 Step 1 w11 w12 w1m wl1 wl2 1 2 m w11 w12 w1m wl1 wl2 wlml Step 2 Determines wij’s. E.g., using batch-learning. Step 1 Determines Centers j’s of j’s. Widths j’s of j’s. Number of j’s.

Train the Kernels

Unsupervised Training + + + + +

Methods Subset Selection Clustering Algorithms Mixture Models Random Subset Selection Forward Selection Backward Elimination Clustering Algorithms KMEANS LVQ Mixture Models GMM

Subset Selection

Random Subset Selection Randomly choosing a subset of points from training set Sensitive to the initially chosen points. Using some adaptive techniques to tune Centers Widths #points

Clustering Algorithms Partition the data points into K clusters.

Clustering Algorithms Is such a partition satisfactory?

Clustering Algorithms How about this?

Clustering Algorithms 1 + + 2 + 4 + 3

Introduction to Radial Basis Function Networks Bias-Variance Dilemma Bias-Variance Dilemma-Önyargı-Varyans İkilemi

Questions -How should the user choose the kernel? Problem similar to that of selecting features for other learning algorithms. Poor choice ---learning made very difficult. Good choice ---even poor learners could succeed. The requirement from the user is thus critical. can this requirement be lessened? is a more automatic selection of features possible?

Goal Revisit Minimize Prediction Error Minimize Empirical Error Ultimate Goal  Generalization Minimize Prediction Error Goal of Our Learning Procedure Deneysel hatayı Minimize etmek Minimize Empirical Error

Badness of Fit Underfitting Overfitting A model (e.g., network) that is not sufficiently complex can fail to detect fully the signal in a complicated data set, leading to underfitting. Produces excessive bias in the outputs. Overfitting A model (e.g., network) that is too complex may fit the noise, not just the signal, leading to overfitting. Produces excessive variance in the outputs.

Underfitting/Overfitting Avoidance Model selection Jittering Early stopping Weight decay Regularization Ridge Regression Bayesian learning Combining networks Model seçimi Değişimi erken durdurma Ağırlık çürüme düzenlileştirme Ridge Regresyon Bayes öğrenme ağları birleştiren

Best Way to Avoid Overfitting Use lots of training data, e.g., 30 times as many training cases as there are weights in the network. for noise-free data, 5 times as many training cases as weights may be sufficient. Don’t arbitrarily reduce the number of weights for fear of underfitting.

Badness of Fit Underfit Overfit

Badness of Fit Underfit Overfit

Bias-Variance Dilemma However, it's not really a dilemma. Bias-Variance Dilemma Underfit Overfit Large bias Small bias Small variance Large variance

Bias-Variance Dilemma More on overfitting Easily lead to predictions that are far beyond the range of the training data. Produce wild predictions in multilayer perceptrons even with noise-free data. Underfit Overfit Large bias Small bias Small variance Large variance

Bias-Variance Dilemma It's not really a dilemma. Bias-Variance Dilemma Bias Variance fit more overfit underfit

Bias-Variance Dilemma The mean of the bias=? The variance of the bias=? Bias-Variance Dilemma noise The true model bias bias bias Solution obtained with training set 3. Solution obtained with training set 2. Solution obtained with training set 1. Sets of functions E.g., depend on # hidden nodes used.

Bias-Variance Dilemma The mean of the bias=? The variance of the bias=? Bias-Variance Dilemma noise Variance The true model bias Sets of functions E.g., depend on # hidden nodes used.

Model Selection Variance Reduce the effective number of parameters. Reduce the number of hidden nodes. Model Selection noise Variance The true model bias Sets of functions E.g., depend on # hidden nodes used.

Bias-Variance Dilemma Goal: Bias-Variance Dilemma noise The true model bias Sets of functions E.g., depend on # hidden nodes used.

Bias-Variance Dilemma Goal: Bias-Variance Dilemma Goal: constant

Bias-Variance Dilemma

Bias-Variance Dilemma Goal: Bias-Variance Dilemma bias2 variance Minimize both bias2 and variance noise Cannot be minimized

Model Complexity vs. Bias-Variance Goal: Model Complexity vs. Bias-Variance bias2 variance noise Model Complexity (Capacity)

Bias-Variance Dilemma Goal: Bias-Variance Dilemma bias2 variance noise Model Complexity (Capacity)

Example (Polynomial Fits)

Example (Polynomial Fits)

Example (Polynomial Fits) Degree 1 Degree 5 Degree 10 Degree 15

Introduction to Radial Basis Function Networks The Effective Number of Parameters

Variance Estimation Mean In general, not available. Variance

Variance Estimation Mean Variance Loss 1 degree of freedom

Simple Linear Regression

Simple Linear Regression Minimize Simple Linear Regression

Mean Squared Error (MSE) Minimize Mean Squared Error (MSE) Loss 2 degrees of freedom

Variance Estimation m: #parameters of the model Loss m degrees of freedom m: #parameters of the model

The Number of Parameters x1 x2 xn x = #degrees of freedom: w1 w2 wm

The Effective Number of Parameters () x1 x2 xn x = The projection Matrix w1 w2 wm

The Effective Number of Parameters () Facts: The Effective Number of Parameters () Pf) The projection Matrix

Regularization SSE The effective number of parameters: Penalize models with large weights SSE

The effective number of parameters  =m. Regularization Without penalty (i=0), there are m degrees of freedom to minimize SSE (Cost). The effective number of parameters  =m. Penalize models with large weights SSE

The effective number of parameters: Regularization With penalty (i>0), the liberty to minimize SSE will be reduced. The effective number of parameters  <m. Penalize models with large weights SSE

Variance Estimation Loss  degrees of freedom The effective number of parameters: Variance Estimation Loss  degrees of freedom

The effective number of parameters: Variance Estimation

Introduction to Radial Basis Function Networks Model Selection

Model Selection Goal Criteria Main Tools (Estimate Model Fitness) Choose the fittest model Criteria Least prediction error Main Tools (Estimate Model Fitness) Cross validation Projection matrix Methods Weight decay (Ridge regression) Pruning and Growing RBFN’s

Empirical Error vs. Model Fitness Ultimate Goal  Generalization Minimize Prediction Error Goal of Our Learning Procedure Minimize Empirical Error (MSE) Minimize Empirical Error : Ampirik(Deneysel) Hata Minimize Minimize Prediction Error

Estimating Prediction Error When you have plenty of data use independent test sets E.g., use the same training set to train different models, and choose the best model by comparing on the test set. When data is scarce, use Cross-Validation Bootstrap Veriler bağımsız test setleri kullanmak çok zaman E. g., aynı eğitim kümesi farklı model tren, ve test kümesi üzerinde karşılaştırılarak en iyi model seçmek için kullanın. Bootstrap Örneklemesi Örneklemeyi tekrarlamak ve çok kez yapmak, zaman, emek, maliyet, mümkün omama gibi sebeplereden dolayı elverişli değildir. Elimizde, sadece bir veri (data, gözlemler) bulunmaktadır. Verinin alındığı kitlenin ortalamasını tahmin etmek istemişsek, elimizde tahmin olarak örnek ortalaması bulunmaktadır. Kitle dağılımı hakkında hiçbir varsayım yapılmamışsa, küçük hacimli örneklemede, kitle ortalaması için güven aralığı söyleyemeyiz, aralık tahmini yapamayız. Bu gibi sorunların altından kalmak için elimizdeki veri üzerinde “yeniden n hacimlik örneklemeler” yapılıp, ilgili istatistiğin değeri çok kez gözlenip, dağılımı hakkında fikir elde edilebilir.

Cross Validation Simplest and most widely used method for estimating prediction error. Partition the original set into several different ways and to compute an average score over the different partitions, e.g., K-fold Cross-Validation Leave-One-Out Cross-Validation Generalize Cross-Validation Veri madenciliği çalışmalarında, uygulanan yöntemin başarınının sınanması için, veri kümesini eğitim ve test kümeleri olarak ayırılmaktadır. Bu ayırma işlemi çeşitli şekillerde yapılabilir. Örneğin veri kümesinin %66’lık bir kısımını eğitim %33’lük bir kısmını ise test için ayırmak ve eğitim kümesi ile sistem eğitildikten sonra test kümesi ile başarısının sınanması kullanılabilecek yöntemlerden birsidir. Bu eğitim ve test kümelerinin rastgele olarak atanması da farklı bir yöntemdir. Fakat bu yazının konusu eğitim ve test kümelerinin nasıl katlanarak değiştirildiğidir. K-katlamalı çarpraz doğrulama yönteminde öncelikle bir k değeri seçilir. Ben yazıda anlatması kolay olsun diye 3 olarak seçeceğim. Ancak belirtmem gerekir ki literatürde en çok tercih edilen k değeri 10’dur.

K-Fold CV Split the set, say, D of available input-output patterns into k mutually exclusive subsets, say D1, D2, …, Dk. Train and test the learning algorithm k times, each time it is trained on D\Di and tested on Di.

K-Fold CV Available Data

K-Fold CV . . . . . . . . . . . . . . . Estimate 2 Test Set Training Set D1 Available Data D2 D3 . . . Dk D1 D2 D3 . . . Dk Estimate 2 D1 D2 D3 . . . Dk D1 D2 D3 . . . Dk D1 D2 D3 . . . Dk

A special case of k-fold CV. Leave-One-Out CV Split the p available input-output patterns into a training set of size p1 and a test set of size 1. Average the squared error on the left-out pattern over the p possible ways of partition.

Error Variance Predicted by LOO A special case of k-fold CV. Error Variance Predicted by LOO Available input-output patterns. Training sets of LOO. Function learned using Di as training set. The estimate for the variance of prediction error using LOO: Error-square for the left-out element.

Error Variance Predicted by LOO A special case of k-fold CV. Error Variance Predicted by LOO Given a model, the function with least empirical error for Di. Available input-output patterns. Training sets of LOO. Function learned using Di as training set. As an index of model’s fitness. We want to find a model also minimize this. The estimate for the variance of prediction error using LOO: Error-square for the left-out element.

Error Variance Predicted by LOO A special case of k-fold CV. Error Variance Predicted by LOO Available input-output patterns. Training sets of LOO. Function learned using Di as training set. How to estimate? Are there any efficient ways? The estimate for the variance of prediction error using LOO: Error-square for the left-out element.

Error Variance Predicted by LOO Error-square for the left-out element.

Generalized Cross-Validation

More Criteria Based on CV GCV (Generalized CV) Akaike’s Information Criterion UEV (Unbiased estimate of variance) FPE (Final Prediction Error) BIC (Bayesian Information Criterio)

More Criteria Based on CV

More Criteria Based on CV

Regularization SSE Standard Ridge Regression, Penalize models with large weights SSE

Regularization SSE Standard Ridge Regression, Penalize models with large weights SSE

Solution Review Used to compute model selection criteria

Example Width of RBF r = 0.5

Example Width of RBF r = 0.5

Example Width of RBF r = 0.5 How the determine the optimal regularization parameter effectively?

Optimizing the Regularization Parameter Re-Estimation Formula

Local Ridge Regression Re-Estimation Formula

Example — Width of RBF

Example — Width of RBF

Example — Width of RBF There are two local-minima. Using the about re-estimation formula, it will be stuck at the nearest local minimum. That is, the solution depends on the initial setting.

Example — Width of RBF There are two local-minima.

Example — Width of RBF There are two local-minima.

Example — Width of RBF RMSE: Root Mean Squared Error In real case, it is not available.

Example — Width of RBF RMSE: Root Mean Squared Error In real case, it is not available.

Local Ridge Regression Standard Ridge Regression Local Ridge Regression

Local Ridge Regression Standard Ridge Regression j  implies that j() can be removed. Local Ridge Regression

The Solutions Linear Regression Standard Ridge Regression Local Ridge Regression Used to compute model selection criteria

Optimizing the Regularization Parameters Incremental Operation P: The current projection Matrix. Pj: The projection Matrix obtained by removing j().

Optimizing the Regularization Parameters Solve Subject to

Optimizing the Regularization Parameters Solve Subject to

Optimizing the Regularization Parameters Remove j() Solve Subject to

The Algorithm Initialize i’s. e.g., performing standard ridge regression. Repeat the following until GCV converges: Randomly select j and compute Perform local ridge regression If GCV reduce & remove j()