Download presentation

Presentation is loading. Please wait.

Published byKyle Abbs Modified about 1 year ago

1
Hazırlayan NEURAL NETWORKS Backpropagation Algorithm PROF. DR. YUSUF OYSAL

2
Backpropagation Algorithm NEURAL NETWORKS – Backpropagation Algorithm Backpropagation Algorithm has two phases: Forward pass phase: computes ‘functional signal’, feed forward propagation of input pattern signals through network Backward pass phase: computes ‘error signal’, propagates the error backwards through network starting at output units (where the error is the difference between actual and desired output values) 2

3
Backpropagation Algorithm NEURAL NETWORKS – Backpropagation Algorithm Step 1: Determine the architecture – how many input and output neurons; what output encoding ? – hidden neurons and layers. Step 2: Initialize all weights and biases to small random values, typically ∈ [-1,1], choose a learning rate η. Step 3: Repeat until termination criteria satisfied – Present a training example and propagate it through the network (forward pass) – Calculate the actual output Inputs applied Multiplied by weights Summed ‘Squashed’ by sigmoid activation function Output passed to each neuron in next layer – Adapt weights starting from the output layer and working backwards (backward pass) 3

4
Parameter Update Rules NEURAL NETWORKS – Backpropagation Algorithm Weight change 4 w pq (t) - weight from node p to node q at time t Error propagation for output neuron i Error propagation for hidden neuron j (the sum is over the i nodes in the layer above the node j)

5
Backpropagation - Example NEURAL NETWORKS – Backpropagation Algorithm 5 Training set p1 = [ ] T class 1banana p2 = [ ] T class 2orange Network architecture How many inputs? How many hidden neurons? Heuristic: n=(inputs+output_neurons)/2 How many output neurons? What encoding of the outputs? 10 for class 1, 01 for class 2 Initial weights and learning rate Let’s η = 0.1 and the weights are set as in the figure.

6
Backpropagation - Example NEURAL NETWORKS – Backpropagation Algorithm 6 1. Forward pass for example 1 calculate the outputs o6 and o7 o1=0.6, o2=0.1, target output 1 0, i.e. class 1 Activations of the hidden units: net3= o1*w13+ o2*w23+b3=0.6* *(-0.2)+0.1=0.14 o3=1/(1+e-net3) =0.53 net4= o1 *w14+ o2*w24+b4=0.6*0+0.1* =0.22 o4=1/(1+e-net4) =0.55 net5= o1 *w15+ o2*w25+b5=0.6* *(-0.4)+0.5=0.64 o5=1/(1+e-net5) =0.65 Activations of the output units: net6= o3 *w36+ o4*w46+ o5*w56 +b6=0.53*(-0.4)+0.55* * =0.13 o6=1/(1+e-net6) =0.53 net7= o3 *w37+ o4*w47+ o5*w57 +b7=0.53* *(-0.1)+0.65*(-0.2)+0.6=0.52 o7=1/(1+e-net7) =0.63

7
Backpropagation - Example NEURAL NETWORKS – Backpropagation Algorithm 7 2. Backward pass for example 1 Calculate the output errors δ6 and δ7 (note that d6=1, d7=0 for class 1) δ6 = (d6-o6) * o6 * (1-o6)=(1-0.53)*0.53*(1-0.53)=0.12 δ7 = (d7-o7) * o7 * (1-o7)=(0-0.63)*0.63*(1-0.63)=-0.15 Calculate the new weights between the hidden and output units (η=0.1) Δw36= η * δ6 * o3 = 0.1*0.12*0.53=0.006 w36 new = w36 old + Δw36 = = Δw37= η * δ7 * o3 = 0.1*-0.15*0.53= w37 new = w37 old + Δw37 = =-0.19 Similarly for w46 new, w47 new, w56 new and w57 new For the biases b6 and b7 (remember: biases are weights with input 1): Δb6= η * δ6 * 1 = 0.1*0.12=0.012 b6 new = b6 old + Δb6 = = Similarly for b7

8
Backpropagation - Example NEURAL NETWORKS – Backpropagation Algorithm 8 Calculate the errors of the hidden units δ3, δ4 and δ5 δ3 = o3 * (1-o3) * (w36* δ6 +w37 * δ7 ) = 0.53*(1-0.53)(-0.4* *(-0.15))= Similarly for δ4 and δ5 Calculate the new weights between the input and hidden units (η=0.1) Δw13= η * δ3 * o1 = 0.1*(-0.019)*0.6= w13 new = w13 old + Δw13 = = Similarly for w23 new, w14 new, w24 new, w15 new and w25 new ; b3 new, b4 new and b6 new 3. Repeat the same procedure for the other training examples Forward pass for example 2, backward pass for example 2… Note: it’s better to apply input examples in random order 4. At the end of the epoch – check if the stopping criteria is satisfied: if yes: stop training if not, continue training: epoch++ go to step 1

9
Backpropagation Algorithm NEURAL NETWORKS – Backpropagation Algorithm 9 Not optimal - is guaranteed to find a minimum but it might be a local minimum! Backpropagation’s error space: many local and 1 global minimum => the generalized gradient descent may not find the global minimum If the algorithm converges to a local minimum,the trajectory is trapped in a valley and diverges from the optimal solution, try different initializations. If the algorithm is slow to converge as there are flat surfaces over the path, increase the learning rate or smooth out the trajectory by averaging the updates to the parameters

10
Overtraining Problem NEURAL NETWORKS – Backpropagation Algorithm 10 Based on them the network should be able to generalize what it has learned to the total population of examples. Overtraining (overfitting): the error on the training set is very small but when a new data is presented to the network, the error is high. => the network has memorized the training examples but has not learned to generalize to new situations! Reasons of overtraining: Training examples are noisy Number of the free parameters is bigger than the number of training examples Preventing Overtraining Use network that is just large enough to provide an adequate fit The network should not have more free parameters than there are training examples! Use an early stopping method K-fold cross validation may be used

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google