Presentation is loading. Please wait.

Presentation is loading. Please wait.

Artificial Neural Networks (1) Dr. Hala Farouk. Biological Neurons of the Brain.

Similar presentations


Presentation on theme: "Artificial Neural Networks (1) Dr. Hala Farouk. Biological Neurons of the Brain."— Presentation transcript:

1 Artificial Neural Networks (1) Dr. Hala Farouk

2 Biological Neurons of the Brain

3 History of Neural Networks  In 1943, Warren McCulloch and Walter Pitts introduced one of the first artificial neurons.  The main feature of their neuron model is that a weighted sum of input signals is compared to a threshold to determine the neuron output.  When the sum is greater than or equal to the threshold, the output is 0.  They went on to show that networks of these neurons could, in principle, compute any arithmetic or logical function.  Unlike biological networks, the parameters of their networks had to be designed, as no training method was available.  In the late 1950s, Frank Rosenblatt introduces a learning rule for training perceptron networks to solve pattern recognition problems. Perceptron learn from their mistakes even if they were initialized with random weights.  In the 1980s, improved multi-layer perceptron networks were introduced.

4 Analogy

5 Single-Input Neuron The output depends on the transfer function f chosen by the designer W and b are parameters adjustable by some learning rule Weight w Bias b

6 Learning Rules  By learning we mean a procedure for modifying the weights and biases of a network  The training algorithm falls into three broad categories  Supervised Learning  Here we need a set of example for the NN.  {p1,t1}, {p2,t2},…, {p Q,t Q }  Where p q is the input and t q is the corresponding correct output.  Reinforcement Learning  Similar to supervised but instead of the given correct output, only a grade (score) is given. This is currently much less common than supervised.  Unsupervised Learning  Here the weights and biases are modified in response to network inputs only. Most of these algorithms perform clustering operation.

7 Typical Transfer Functions

8 The log-sigmoid transfer function is commonly used in multilayer networks that are trained using the back-propagation algorithm, in part because this function is differentiable.

9 Example on Single-Input Neuron  The input to a single input neuron is 2.0, its weights is 2.3 and its bias is -3. What is the net input to the transfer function?  n=w p + b = (2.3) (2) + (-3) = 1.6  What is the neuron output, if it has the following transfer functions?  Hard limit  Linear  Log-sigmoid a= hardlim (1.6) =1.0 a= purelin (1.6) =1.6 a= logsig (1.6) = 1/ (1+e -1.6 )=0.8320

10 Multi-Input Neuron n= w 1,1 p 1 + w 1,2 p 2 + w 1,3 p 3 + … +w 1,R p R + b 1 n=W p + b First Index for the Neuron Second Index for the Input

11 Multi-Input Neuron  The number of input is set by the external specifications of the problem.  If you want to design a neural network that is to predict kite- flying conditions and the inputs are air temperature, wind velocity, and humidity, then there would be three inputs to the network (R=3).

12 Example on Multi-Input Neuron  Given a two-input neuron with the following parameters: b=1.2, W=[3 2] and p=[-5 6] T, calculate the neuron output for the following transfer functions:  A symmetrical hard limit transfer fn.  A saturating linear transfer fn.  A hyperbolic tangent sigmoid transfer fn.  n= W p + b = [ 3 2 ] * [ -5 6] T + (1.2)= -1.8  a= hardlims (-1.8)=-1  a= satlin(-1.8)=0  a= transig(-1.8)=0.9468

13 S Neurons with R Inputs each

14

15 Example on Single-Neuron Perceptron  A i = hardlim (n i ) = hardlim( i w T p + b i )  Given w 1,1 =1, w 1,2 =1, b= - 1  The decision boundary (line at which n=0)  n= 1 w T p + b 1 =w 1,1 p 1 +w 1,2 p 2 +b =p 1 +p 2 -1 = 0  On one side of this boundary, the output will be 0; on the line and on the other side the output is 1  Test with any point to know the direction  The boundary is always orthogonal to 1 w

16 Adjusting the Bias  If the decision boundary line is given then we can adjust the bias according to the following equation: 1 w T p + b 1 = 0

17 Design Problem  Lets design an AND gate using NN 1) Draw the input space Black dots --> output=1 White dots --> output=0

18 Design Problem cont. 2. Select a decision boundary there are infinite number of lines that separates the black dots from the white dots. BUT it’s reasonable to choose a line halfway 3. Choose weight vector that is orthogonal to the decision boundary the weight vector can be of any length so there are infinite possibilities One choice is 1 w T =[2 2]

19 Design Problem cont. 4. Find the bias Pick any point on the decision boundary line and substitute in 1 w T p + b 1 = 0 For example, p=[1.5 0] T 1 w T p + b 1 = [2 2]*[1.5 0] T +b = 3 + b =0 b= Test the network

20 Multiple-Neuron Perceptron  There will be one decision boundary for EACH neuron.  The decision boundaries will be defined by i w T p + b i = 0

21 Example on Supervision Learning  The problem has two inputs and one output  Therefore two inputs and one neuron

22 Example on Supervision Learning cont.  Lets start without any bias, so only two parameters w 1,1 and w 1,2  Removing the bias, then decision boundary passes through the origin

23 Example on Supervision Learning cont.  We want a learning rule that will find a weight vector that points in one of these directions (length of vector is not important)  So lets start with random weights  1 w T =[ ]  Then present it to the network with p1.  a=hardlim( [ ]*[1 2] T  a=hardlim( -0.6 ) = 0 The NN has made a mistake!

24 Example on Supervision Learning cont.  Why did it make a mistake?  If we adjust the weights but adding p 1 to it this would make 1 w point more in the direction of p 1 so that it would hopefully not be classified falsely into the wrong zone w1w SO If ( t=1 and a=0 ), then 1 w new = 1 w old + p

25 Example on Supervision Learning cont.  Test the next input p 2 with new weights  a=hardlim( [ ]*[-1 2] T  a=hardlim( 0.4 ) = 1  Again a mistake.  This time we want w to move away from p w1w 1 w new = 1 w old + p = [ ] T + [ 1 2 ] T = [ ] T SO If ( t=0 and a=1 ), then 1 w new = 1 w old - p

26 Example on Supervision Learning cont.  Test the next input p 3 with new weights  a=hardlim( [ ]*[0 -1] T  a=hardlim( 0.8 ) = 1  Again a mistake.  We want w to move away from p w1w 1 w new = 1 w old + p = [ ] T - [- 1 2 ] T = [ ] T SO If ( t=0 and a=1 ), then 1 w new = 1 w old - p

27 Example on Supervision Learning cont.  Test the next input p 1 again with new weights  a=hardlim( [ ]*[1 2] T  a=hardlim( 3.4 ) = 1  Now correct.  Repeat for all other inputs w1w 1 w new = 1 w old + p = [ ] T - [0 -1 ] T = [ ] T SO If ( t=a 1 ) then w new = 1 w old

28 Multilayer Network Two Hidden LayersOne Output Layer

29 Recurrent Network  It is a network with feedback.  Some of its outputs is connected to its inputs  a(1)=satlins ( W a(0) + b )  a(2)=satlins ( W a(1) + b ), …


Download ppt "Artificial Neural Networks (1) Dr. Hala Farouk. Biological Neurons of the Brain."

Similar presentations


Ads by Google