Ch. 4: Radial Basis Functions Stephen Marsland, Machine Learning: An Algorithmic Perspective. CRC 2009 based on slides from many Internet sources Longin Jan Latecki Temple University

Perceptorn

In RBFN

architecture Input layerHidden layerOutput layer x1x1 x2x2 x3x3 xnxn h1h1 h2h2 h3h3 hmhm f(x) W1W1 W2W2 W3W3 WmWm

Three layers Input layer – Source nodes that connect to the network to its environment Hidden layer – Hidden units provide a set of basis function – High dimensionality Output layer – Linear combination of hidden functions architecture

Radial basis function h j (x) = exp( -(x-c j ) 2 / r j 2 ) f(x) =  w j h j (x) j=1 m Wherec j is center of a region, r j is width of the receptive field architecture

7 Function Approximation with Radial Basis Functions RBF Networks approximate functions using (radial) basis functions as the building blocks.

8 Exact Interpolation RBFs have their origins in techniques for performing exact function interpolation [Bishop, 1995]: – Find a function h(x) such that h(x n ) = t n  n=1,... N Radial Basis Function approach (Powel 1987): – Use a set of N basis functions of the form  (||x-x n ||), one for each point,where  (.) is some non-linear function. – Output: h(x) =  n w n  (||x-x n ||)

9 Exact Interpolation Goal (exact interpolation): – Find a function h(x) such that h(x n ) = t n  n=1,... N Radial Basis Function approach (Powel 1987): – Use a set of N basis functions of the form  (||x-x n ||), one for each point,where  (.) is some non-linear function. – Output: h(x) =  n w n  (||x-x n ||) w 1  (||x 1 -x 1 ||) + w 2  (||x 1 -x 2 ||) w N  (||x 1 -x N ||) = t 1 w 1  (||x 2 -x 1 ||) + w 2  (||x 2 -x 2 ||) w N  (||x 2 -x N ||) = t 2  W = T... w 1  (||x N -x 1 ||) + w 2  (||x N -x 2 ||) w N  (||x N -x N ||)= t N

10 Exact Interpolation

11 Exact Interpolation

12 Due to noise that may be present in the data exact interpolation is rarely useful. By introducing a number of modifications, we arrive at RBF networks: Complexity rather than the size of the data is what is important – Number of the basis functions need not be equal to N Centers need not be constrained by the input Each basis function can have its own adjustable width parameter  Bias parameter may be included in the linear sum.

13 Illustrative Example - XOR Problem 22 11

14 Function Approximation via Basis Functions and RBF Networks Using nonlinear functions, we can convert a nonlinearly separable problem into a linearly separable one. From a function approximation perspective, this is equivalent to implementing a complex function (corresponding to the nonlinearly separable decision boundary) using simple functions (corresponding to the linearly separable decision boundary) Implementing this procedure using a network architecture, yields the RBF networks, if the nonlinear mapping functions are radial basis functions. Radial Basis Functions: – Radial: Symmetric around its center – Basis Functions: Also called kernels, a set of functions whose linear combination can generate an arbitrary function in a given function space.

15 RBF Networks

16 RBF Networks

17 Network Parameters What do these parameters represent?  : The radial basis function for the hidden layer. This is a simple nonlinear mapping function (typically Gaussian) that transforms the d- dimensional input patterns to a (typically higher) H- dimensional space. The complex decision boundary will be constructed from linear combinations (weighted sums) of these simple building blocks. u Ji : The weights joining the first to hidden layer. These weights constitute the center points of the radial basis functions. Also called prototypes of data.  : The spread constant(s). These values determine the spread (extend) of each radial basis function. W jk : The weights joining hidden and output layers. These are the weights which are used in obtaining the linear combination of the radial basis functions. They determine the relative amplitudes of the RBFs when they are combined to form the complex function. ||x-u J ||: the Euclidean distance between the input x and the prototype vector u J. Activation of the hidden unit is determined according to this distance through 

18

19 Training RBF Networks Approach 1: Exact RBF Approach 2: Fixed centers selected at random Approach 3: Centers are obtained from clustering Approach 4: Fully supervised training

20 Training RBF Networks Approach 1: Exact RBF Guarantees correct classification of all training data instances. Requires N hidden layer nodes, one for each training instance. No iterative training is involved: w are obtained by solving a set of linear equations Non-smooth, bad generalization

21 Exact Interpolation

22 Exact Interpolation

23 Too Many Receptive Fields? In order to reduce the artificial complexity of the RBF, we need to use fewer number of receptive fields. Approach 2: Fixed centers selected at random. Use M < N data points as the receptive field centers. Fast but may require excessive centers Approach 3: Centers are obtained from unsupervised learning (clustering). Centers no longer has to coincide with data points This is the most commonly used procedure, providing good results.

24 Approach 2 Approach 3 Approach 3.b

25 Determining the Output Weights through learning (LMS)

26 RBFs for Classification

Homework Problem 4.1, p. 117 Problem 4.2, p. 117