Artificial Neural Network Lecturer note Azhari, Dr Artificial Neural Network Script & Programming in Matlab Computer Science Department, Gadjah Mada University, @2010

Multilayer Neural Network

Learning rules in this toolbox fall into two broad categories: A learning rule is defined as a procedure for modifying the weights and biases of a network. The learning rule is applied to train the network to perform some particular task. Learning rules in this toolbox fall into two broad categories: supervised learning, and unsupervised learning.

Supervised learning the learning rule is provided with a set of examples (the training set) of proper network behavior: {p1,t1},{p2,t2},…,{pQ,tQ} where pQ is an input to the network, and tQ is the corresponding correct (target) output. As the inputs are applied to the network, the network outputs are compared to the targets. The learning rule is then used to adjust the weights and biases of the network in order to move the network outputs closer to the targets.

Unsupervised learning In unsupervised learning, the weights and biases are modified in response to network inputs only. There are no target outputs available. Most of these algorithms perform clustering operations. They categorize the input patterns into a finite number of classes.

The perceptron & Learning rule The objective is to reduce the error e, which is the difference between the neuron response a and the target vector t. The perceptron learning rule learnp calculates desired changes to the perceptron’s weights and biases, given an input vector p and the associated error e. The target vector t must contain values of either 0 or 1, because perceptrons (with hardlim transfer functions) can only output these values.

The perceptron & Learning rule There are three conditions that can occur for a single neuron once an input vector p is presented and the network’s response a is calculated:

The perceptron & Learning rule The perceptron learning rule can be written in terms of the error e = t – a : For the case of a layer of neurons can be written as: Secara umum

Example: Learning rule Matlab code: net = newp([-2 2;-2 2],[0 1]); net.b{1} = [0]; w = [1 -0.8]; net.IW{1,1} = w; p = [1; 2]; t = [1]; a = sim(net,p) % a = 0 e = t-a % e = 1 – 0  1 dw = learnp(w,p,[],[],[],[],e,[],[],[]) % dw = [1 2] w = w + dw % w = [2.0000 1.2000] The process of finding new weights (and biases) can be repeated until there are no errors.

Example Training train Assume: Step 1 The output a does not equal the target value t1, so use the perceptron rule to find the incremental changes to the weights and biases based on the error. calculate the new weights and bias using the perceptron update rules Step 2

W(4) = [-3, -1] b(4) = 0 Final step: W(6) = [–2 –3] b(6) = 1

Train syntax

Example train From page: 16-300

Matlab code: with epoch =1 net = newp([-2 2;-2 2],[0 1]); p = [2; 2]; t = [0]; net.trainParam.epochs = 1; net = train(net,p,t); w = net.iw{1,1}, b = net.b{1} % w = [-2 -2 ] & b = -1 p = [[2;2] [1;-2] [-2;2] [-1;1]] t = [0 1 0 1] %w = [-3 -1 ] & b = 0 % continue with epoch =1000 a = sim(net,p) % a = [0 0 1 1] net.trainParam.epochs = 1000; net = train(net,p,t); w = net.iw{1,1}, b = net.b{1} % w = [-2 -3 ] & b = 1 % a = [0 1 0 1 ] error = a-t % error = [0 0 0 0]

Feedforward backpropagation net = newff(PR,[S1 S2...SNl],{TF1 TF2...TFNl},BTF,BLF,PF) PR = Rx2 matrix of min and max values for R input elements S = number of output vector (or number of neorons in each hidden layer) TF = Transfer function of ith layer, default = 'tansig' BTF = Backpropagation network training function, default = 'traingdx' BLF = Backpropagation weight/bias learning function, default = 'learngdm' PF = performance function, default = 'mse' xk+1 = xk - akgk The transfer functions TFi can be any differentiable transfer function such as TANSIG, LOGSIG, or PURELIN. The training function BTF can be any of the backprop training functions such as TRAINLM, TRAINBFG, TRAINRP, TRAINGD, etc.

input vector and vector target Matlab Code: input=[-1 -1 2 2; 0 5 0 5] Example newff input vector and vector target Matlab Code: input=[-1 -1 2 2; 0 5 0 5] target=[-1 -1 1 1]   net=newff ([-1 2;0 5],[3 1],{'tansig' 'purelin'},’traingdm’); net.trainParam.epochs=30; %(number of epochs) net.trainParam.lr=0.3; %(learning rate) net.trainParam.mc=0.6; %(momentum) net=train (net,input,target); output =sim(net,input); [target; output]  [-1 2;0 5] is the minimum and maximum values of vector input One hidden layer, 3 nodes in hidden layer, with tangent sigmoid as transfer function in hidden layer and linear function for output layer, and with gradient descent with momentum backpropagation training function

Example: The XOR problem Single hidden layer: 3 Sigmoid neurons 2 inputs, 1 output x1 x2 y Example 1 Example 2 1 Example 3 Example 4 Desired: I/O table (XOR):

create a matrix with the inputs ”1 1”, ”1 0”, ”0 1” and ”0 0”; XOR network create a matrix with the inputs ”1 1”, ”1 0”, ”0 1” and ”0 0”; and target ‘0 1 1 0‘ Matlab code: net = newff([0 1; 0 1],[2 1],{’logsig’,’logsig’}); input = [1 1 0 0; 1 0 1 0]; target = [0 1 1 0]; net.trainParam.show=NaN; net = train(net,input,target); output = sim(net,input) net.IW{1,1} net.LW{2,1} Apply the default training algorithm Levenberg-Marquardt backpropagation trainlm Output = 0.0000 1.0000 1.0000 0.0000 ans = 25.9797 -25.7624