Download presentation

Presentation is loading. Please wait.

Published byIsaak Brent Modified about 1 year ago

1
Artificial Neural Networks (1) Dr. Hala Farouk

2
Biological Neurons of the Brain

3
History of Neural Networks In 1943, Warren McCulloch and Walter Pitts introduced one of the first artificial neurons. The main feature of their neuron model is that a weighted sum of input signals is compared to a threshold to determine the neuron output. When the sum is greater than or equal to the threshold, the output is 0. They went on to show that networks of these neurons could, in principle, compute any arithmetic or logical function. Unlike biological networks, the parameters of their networks had to be designed, as no training method was available. In the late 1950s, Frank Rosenblatt introduces a learning rule for training perceptron networks to solve pattern recognition problems. Perceptron learn from their mistakes even if they were initialized with random weights. In the 1980s, improved multi-layer perceptron networks were introduced.

4
Analogy

5
Single-Input Neuron The output depends on the transfer function f chosen by the designer W and b are parameters adjustable by some learning rule Weight w Bias b

6
Learning Rules By learning we mean a procedure for modifying the weights and biases of a network The training algorithm falls into three broad categories Supervised Learning Here we need a set of example for the NN. {p1,t1}, {p2,t2},…, {p Q,t Q } Where p q is the input and t q is the corresponding correct output. Reinforcement Learning Similar to supervised but instead of the given correct output, only a grade (score) is given. This is currently much less common than supervised. Unsupervised Learning Here the weights and biases are modified in response to network inputs only. Most of these algorithms perform clustering operation.

7
Typical Transfer Functions

8
The log-sigmoid transfer function is commonly used in multilayer networks that are trained using the back-propagation algorithm, in part because this function is differentiable.

9
Example on Single-Input Neuron The input to a single input neuron is 2.0, its weights is 2.3 and its bias is -3. What is the net input to the transfer function? n=w p + b = (2.3) (2) + (-3) = 1.6 What is the neuron output, if it has the following transfer functions? Hard limit Linear Log-sigmoid a= hardlim (1.6) =1.0 a= purelin (1.6) =1.6 a= logsig (1.6) = 1/ (1+e -1.6 )=0.8320

10
Multi-Input Neuron n= w 1,1 p 1 + w 1,2 p 2 + w 1,3 p 3 + … +w 1,R p R + b 1 n=W p + b First Index for the Neuron Second Index for the Input

11
Multi-Input Neuron The number of input is set by the external specifications of the problem. If you want to design a neural network that is to predict kite- flying conditions and the inputs are air temperature, wind velocity, and humidity, then there would be three inputs to the network (R=3).

12
Example on Multi-Input Neuron Given a two-input neuron with the following parameters: b=1.2, W=[3 2] and p=[-5 6] T, calculate the neuron output for the following transfer functions: A symmetrical hard limit transfer fn. A saturating linear transfer fn. A hyperbolic tangent sigmoid transfer fn. n= W p + b = [ 3 2 ] * [ -5 6] T + (1.2)= -1.8 a= hardlims (-1.8)=-1 a= satlin(-1.8)=0 a= transig(-1.8)=0.9468

13
S Neurons with R Inputs each

14

15
Example on Single-Neuron Perceptron A i = hardlim (n i ) = hardlim( i w T p + b i ) Given w 1,1 =1, w 1,2 =1, b= - 1 The decision boundary (line at which n=0) n= 1 w T p + b 1 =w 1,1 p 1 +w 1,2 p 2 +b =p 1 +p 2 -1 = 0 On one side of this boundary, the output will be 0; on the line and on the other side the output is 1 Test with any point to know the direction The boundary is always orthogonal to 1 w

16
Adjusting the Bias If the decision boundary line is given then we can adjust the bias according to the following equation: 1 w T p + b 1 = 0

17
Design Problem Lets design an AND gate using NN 1) Draw the input space Black dots --> output=1 White dots --> output=0

18
Design Problem cont. 2. Select a decision boundary there are infinite number of lines that separates the black dots from the white dots. BUT it’s reasonable to choose a line halfway 3. Choose weight vector that is orthogonal to the decision boundary the weight vector can be of any length so there are infinite possibilities One choice is 1 w T =[2 2]

19
Design Problem cont. 4. Find the bias Pick any point on the decision boundary line and substitute in 1 w T p + b 1 = 0 For example, p=[1.5 0] T 1 w T p + b 1 = [2 2]*[1.5 0] T +b = 3 + b =0 b= Test the network

20
Multiple-Neuron Perceptron There will be one decision boundary for EACH neuron. The decision boundaries will be defined by i w T p + b i = 0

21
Example on Supervision Learning The problem has two inputs and one output Therefore two inputs and one neuron

22
Example on Supervision Learning cont. Lets start without any bias, so only two parameters w 1,1 and w 1,2 Removing the bias, then decision boundary passes through the origin

23
Example on Supervision Learning cont. We want a learning rule that will find a weight vector that points in one of these directions (length of vector is not important) So lets start with random weights 1 w T =[ ] Then present it to the network with p1. a=hardlim( [ ]*[1 2] T a=hardlim( -0.6 ) = 0 The NN has made a mistake!

24
Example on Supervision Learning cont. Why did it make a mistake? If we adjust the weights but adding p 1 to it this would make 1 w point more in the direction of p 1 so that it would hopefully not be classified falsely into the wrong zone w1w SO If ( t=1 and a=0 ), then 1 w new = 1 w old + p

25
Example on Supervision Learning cont. Test the next input p 2 with new weights a=hardlim( [ ]*[-1 2] T a=hardlim( 0.4 ) = 1 Again a mistake. This time we want w to move away from p w1w 1 w new = 1 w old + p = [ ] T + [ 1 2 ] T = [ ] T SO If ( t=0 and a=1 ), then 1 w new = 1 w old - p

26
Example on Supervision Learning cont. Test the next input p 3 with new weights a=hardlim( [ ]*[0 -1] T a=hardlim( 0.8 ) = 1 Again a mistake. We want w to move away from p w1w 1 w new = 1 w old + p = [ ] T - [- 1 2 ] T = [ ] T SO If ( t=0 and a=1 ), then 1 w new = 1 w old - p

27
Example on Supervision Learning cont. Test the next input p 1 again with new weights a=hardlim( [ ]*[1 2] T a=hardlim( 3.4 ) = 1 Now correct. Repeat for all other inputs w1w 1 w new = 1 w old + p = [ ] T - [0 -1 ] T = [ ] T SO If ( t=a 1 ) then w new = 1 w old

28
Multilayer Network Two Hidden LayersOne Output Layer

29
Recurrent Network It is a network with feedback. Some of its outputs is connected to its inputs a(1)=satlins ( W a(0) + b ) a(2)=satlins ( W a(1) + b ), …

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google