1. of the Artificial Neural Networks.
Topic 3.
Learning Rules
of the Artificial Neural Networks.

2. Multilayer Perceptron.
The first layer is the input layer,
and the last layer is the output layer.
All other layers with no direct connections from or to the outside are called hidden layers.

3. Multilayer Perceptron.
The input is processed and relayed from one layer to the next, until the final result has been computed.
This process represents the feedforward scheme.

4. Multilayer Perceptron.
structural credit assignment problem: when an error is made at the output of a network, how is credit (or blame) to be assigned to neurons deep within the network?
One of the most popular techniques to train the hidden neurons is error backpropagation,
whereby the error of output units is propagated back to yield estimates of how much a given hidden unit contributed to the output error.

5. Multilayer Perceptron.
The error function of multilayer perceptron:
The best performance of the network corresponds to the minimum of the total squared error, and during the network training, we adjust the weights of connections in order to get to that minimum.

6. Multilayer Perceptron.
Combination of the weights, including that of hidden neurons, which minimises the error function E is considered to be a solution of multiple layer perceptron learning problem .

7. Multilayer Perceptron.
The error function of multilayer perceptron:
The backpropagation algorithm looks for the minimum of the multi-variable error function E in the space of weights of connections w using the method of gradient descent.

8. Multilayer Perceptron.
Following calculus, a local minimum of a function of two or more variables is defined by equality to zero of its gradient:
where
is partial derivative of the error function E with respect to the weight
of connection between h-th unit in the layer k and t-th unit in the
previous layer number k-1.

9. Multilayer Perceptron.
We would like to go in the direction opposite to to most rapidly minimise E.
Therefore, during the iterative process of gradient descent each weight of connection, including the hidden ones, is updated:
using the increment
here C represents the learning rate.

10. Multilayer Perceptron.
where
Since calculus-based methods of minimisation rest on the taking of derivatives, their application to network training requires the error function E be a differentiable function

11. Multilayer Perceptron.
Since calculus-based methods of minimisation rest on the taking of derivatives, their application to network training requires the error function E be a differentiable function,
which requires the network output Xjp to be differentiable,
which requires the activation functions f(S) to be differentiable:
where

12. Multilayer Perceptron.
Since calculus-based methods of minimisation rest on the taking of derivatives, their application to network training
requires the error function E be a differentiable function,
which requires the network output Xjp to be differentiable,
which requires the activation functions f(S) to be differentiable:
This provides a powerful motivation for using continuous and differentiable activation functions f(w,a).
where

13. Multilayer Perceptron.
Since calculus-based methods of minimisation rest on the taking of derivatives, their application to network training
requires the activation functions f(S) to be differentiable.
To make a multiple layer perceptron to be “able to learn” here is a useful generic sigmoid activation function associated with a hidden or output neuron:
where

14. Multilayer Perceptron.
Since calculus-based methods of minimisation rest on the taking of derivatives, their application to network training
requires the activation functions f(S) to be differentiable.
To make a multiple layer perceptron to be “able to learn” here is a useful generic sigmoid activation function associated with a hidden or output neuron:
Important thing about the generic sigmoid function is that it is differentiable, with a very simple and easy to compute derivative
where

15. Multilayer Perceptron.
Since calculus-based methods of minimisation rest on the taking of derivatives, their application to network training
requires the activation functions f(S) to be differentiable.
To make a multiple layer perceptron to be “able to learn” here is a useful generic sigmoid activation function associated with a hidden or output neuron:
If all activation functions f(S) in the network are differentiable then, according to the chain rule of calculus, differentiating the error function E with respect to the weight of connection in consideration we can express the corresponding partial derivative of the error function
where

16. Multilayer Perceptron.
Then….
where

17. Multilayer Perceptron.
where

18. Multilayer Perceptron.
where
Thus, correction to the hidden weight of connection between h-th unit in the k-th layer and t-th unit in the previous (k-1)-th layer can be found by

19. Multilayer Perceptron Learning rule!!!
where
The correction is defined by
the output layer errors ejp,
derivatives of activation functions of all
neurons in the upper layers with numbers p > k,
derivative of activation function of the neuron h itself in the layer k,
activation function of connected neuron t in the previous layer (k-1).

20. Multilayer Perceptron Learning rule!!!
where
We can easily measure the output errors of the network, and it is us to define all the activation functions.
If we also know the derivatives of the activation functions, then we can easily find all the corrections to weights of connections of all neurons in the network, including the hidden ones, during the second run back through the network.

21. Multilayer Perceptron Training.
The training process of multilayer perceptron consists of two phases.
Initial values of the weights of connections set up randomly.
Then, during the first, feedforward phase, starting from the input layer and further layer-by-layer, outputs of every unit in the network are computed together with the corresponding derivatives.
Figure: Directions of two basic signal flows in multilayer perceptron: forward propagation of function signals and back-propagation of error signals.

22. Multilayer Perceptron Training.
The training process of multilayer perceptron consists of two phases.
Initial values of the weights of connections set up randomly.
Then, during the first, feedforward phase, starting from the input layer and further layer-by-layer, outputs of every unit in the network are computed together with the corresponding derivatives.
In the second, feedback phase corrections to all weights of connections of all units including the hidden ones are computed using the outputs and derivatives computed during the feedforward phase.
Figure: Directions of two basic signal flows in multilayer perceptron: forward propagation of function signals and back-propagation of error signals.

23. Multilayer Perceptron Training.
To understand the second, error back-propagation
phase of computing corrections to the weights, let us
follow an example of a small three-layer perceptron.
input layer
hidden layer
output layer
Layer N 1 2
Unit N

24. Multilayer Perceptron Training.
To understand the second, error back-propagation
phase of computing corrections to the weights, let us
follow an example of a small three-layer perceptron.
input layer
hidden layer
output layer
Layer N 1 2
Unit N
Suppose that we have found all outputs and corresponding derivatives of activation functions of all computing units including the hidden ones in the network.

25. Multilayer Perceptron Training.
input layer
hidden layer
output layer
Layer N 1 2
Unit N
We shall mark
values of the layer in consideration,
values of the layer previous to the one in consideration,

26. Multilayer Perceptron Training.
input layer
hidden layer
output layer
Layer N 1 2
Unit N
Weight of connection between unit
number 1 (first lower index) in the output
layer (layer number 2 shown as the upper
index) and unit number 0 (second lower
index) in the previous layer (number 1=2-1)
after presentation of a training pattern
would have a correction

27. Multilayer Perceptron Training.
input layer
hidden layer
output layer
Layer N 1 2
Unit N
Analogously, corrections to all six weights of connections between the output layer and the hidden layer are obtained as

28. Multilayer Perceptron Training. Corrections to hidden units connections.
input layer
hidden layer
output layer
Layer N 1 2
Unit N
We shall mark
values of the layer in consideration,
values of the layer previous to the one in consideration,
values of the layers above the one in consideration,

29. Multilayer Perceptron Training. Corrections to hidden units connections.
input layer
hidden layer
output layer
Layer N 1 2
Unit N
Weight of connection between unit number 1 (first lower index) in the hidden layer (layer number 1 shown in the upper index) and unit number 0 in the previous input layer (second lower index) would have a correction

30. Multilayer Perceptron Training. Corrections to hidden units connections.
input layer
hidden layer
output layer
Layer N 1 2
Unit N
Analogously, for all six weights of connections between the hidden layer and the input layer:

31. Multilayer Perceptron Training. Corrections to hidden units connections.
input layer
hidden layer
output layer
Layer N 1 2
Unit N
In this way going backwards through the network, one obtain the corrections to all weights …,

32. Multilayer Perceptron Training. Corrections to hidden units connections.
input layer
hidden layer
output layer
Layer N 1 2
Unit N
In this way going backwards through the network, one obtain the corrections to all weights …,
then update the weights.

33. Multilayer Perceptron Training. Corrections to hidden units connections.
input layer
hidden layer
output layer
Layer N 1 2
Unit N
In this way going backwards through the network, one obtain the corrections to all weights …,
then update the weights.
After that, with the new weights go forward to get new outputs…

34. Multilayer Perceptron Training. Corrections to hidden units connections.
input layer
hidden layer
output layer
Layer N 1 2
Unit N
In this way going backwards through the network, one obtain the corrections to all weights …,
then update the weights.
After that, with the new weights go forward to get new outputs…
Find new error, go backwards and so on…

35. Multilayer Perceptron Training.
In this way going backwards through the network, one obtain the corrections to all weights …, then update the weights.
After that, with the new weights go forward to get new outputs…
Find new error, go backwards and so on…
Hopefully, sooner or later the iterative procedure will come to output with the minimum error, i.e. the absolute minimum of the error function E.

36. Multilayer Perceptron Training.
In this way going backwards through the network, one obtain the corrections to all weights …, then update the weights.
After that, with the new weights go forward to get new outputs…
Find new error, go backwards and so on…
Hopefully, sooner or later the iterative procedure will come to output with the minimum error, i.e. the absolute minimum of the error function E.
Unfortunately, as a function of many variables, the error function might have more than one minimum, and one may get not to the absolute minimum but to a relative one.

37. Multilayer Perceptron Training.
Unfortunately, as a function of many variables, the
error function might have more than one minimum, and one may get not to the absolute minimum but to a relative one.
If it happens, the error function stops to decrease regardless of number of iteration.
Some measures must be taken to get out of the function relative minimum, for example, adding small random values, i.e. “noise”, to one or more of the weights.
Then the iterative procedure starts from that new point to get to the absolute minimum eventually.

38. Multilayer Perceptron Training.
Finally, after successful training, perceptron is able to produce the desired responses to all input patterns of the training set.

39. Multilayer Perceptron Training.
Finally, after successful training, perceptron is able to produce the desired responses to all input patterns of the training set.
Then all the network weights of connections are fixed,

40. Multilayer Perceptron Training.
Finally, after successful training, perceptron is able to produce the desired responses to all input patterns of the training set.
Then all the network weights of connections are fixed,
and the network is presented with inputs it must “recognise”, i.e. not the training set inputs.

41. Multilayer Perceptron Training.
Finally, after successful training, perceptron is able to produce the desired responses to all input patterns of the training set.
Then all the network weights of connections are fixed,
and the network is presented with inputs it must “recognise”, i.e. not the training set inputs.
If an input in consideration produces an output similar to one of the training set, such input is said to belong to the same type or cluster of inputs as the corresponding one of the training set.

42. Multilayer Perceptron Training.
Then all the network weights of connections are fixed,
and the network is presented with inputs it must “recognise”, i.e. not the training set inputs.
If an input in consideration produces an output similar to one of the training set, such input is said to belong to the same type or cluster of inputs as the corresponding one of the training set.
If the network produces an output not similar to any of the training set, then such an input is said not been recognised.

43. Multilayer Perceptron Training. Conclusion.
In 1969 Minsky and Papert not just found the solution to the XOR problem in a form of multilayer perceptron, they also gave a very thorough mathematical analysis of the time it takes to train such networks.
Minsky and Papert emphasized that training times increase very rapidly for certain problems as the number of input lines and weights of connections increases.

44. Multilayer Perceptron Training. Conclusion.
Minsky and Papert emphasized that training times increase very rapidly for certain problems as the number of input lines and weights of connections increases.
The difficulties were seized upon by opponents of the subject. In particular, this was true of those working in the field of artificial intelligence (AI), who at that time did not want to concern themselves with the underlying “wetware” of the brain, but only with the functional aspects – regarded by them solely as logical processing.
Due to the limitations of funding, competition between AI and neural network communities could have only one victor.

45. Multilayer Perceptron Training. Conclusion.
Due to the limitations of funding, competition between AI and neural network communities could have only one victor.
Neural networks then went into a relative quietude for more then fifteen years, with only a few devotees still working on it.

46. Multilayer Perceptron Training. Conclusion.
Due to the limitations of funding, competition between AI and neural network communities could have only one victor.
Neural networks then went into a relative quietude for more then fifteen years, with only a few devotees still working on it.
Then new vigour came from various sources. One was from the increasing power of computers, allowing simulations of otherwise intractable problems.

47. Multilayer Perceptron Training. Conclusion.
New vigour came from various sources. One was from the increasing power of computers, allowing simulations of otherwise intractable problems.
Finally, established by the mid 80s the backpropagation algorithm solved the difficulty of training hidden neurons.

48. Multilayer Perceptron Training. Conclusion.
New vigour came from various sources. One was from the increasing power of computers, allowing simulations of otherwise intractable problems.
Finally, established by the mid 80s the backpropagation algorithm solved the difficulty of training hidden neurons.
Nowadays, Perceptron is an effective tool for recognising protein and amino-acid sequences and processing other complex biological data.