Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Perceptron. Prehistory W.S. McCulloch & W. Pitts (1943). A logical calculus of the ideas immanent in nervous activity, Bulletin of Mathematical Biophysics,

Similar presentations

Presentation on theme: "The Perceptron. Prehistory W.S. McCulloch & W. Pitts (1943). A logical calculus of the ideas immanent in nervous activity, Bulletin of Mathematical Biophysics,"— Presentation transcript:

1 The Perceptron

2 Prehistory W.S. McCulloch & W. Pitts (1943). A logical calculus of the ideas immanent in nervous activity, Bulletin of Mathematical Biophysics, 5, x+y-2 1 * -2 y * +1 x * +1 if sum<0 : 0 else : 1 inputs weights sum output x & yyx output inputs Truth Table for Logical AND This seminal paper pointed out that simple artificial neurons could be made to perform basic logical operations such as AND, OR and NOT.

3 Nervous Systems as Logical Circuits Groups of these neuronal logic gates could carry out any computation, even though each neuron was very limited. x+y-1 1 * -1 y * +1 x * +1 if sum<0 : 0 else : 1 inputs weights sum output x | yyx output inputs Truth Table for Logical OR Could computers built from these simple units reproduce the computational power of biological brains? Were biological neurons performing logical operations?

4 The Perceptron It obeyed the following rule: If the sum of the weighted inputs exceeds a threshold, output 1, else output -1. output inputs weights sum Σx i w i * Frank Rosenblatt (1962). Principles of Neurodynamics, Spartan, New York, NY. Subsequent progress was inspired by the invention of learning rules inspired by ideas from neuroscience… Rosenblatts Perceptron could automatically learn to categorise or classify input vectors into types. 1 if Σ input i * weight i > threshold -1 if Σ input i * weight i < threshold

5 Classifier Consider a network as a Classifier Network parameters are adapted so that it discriminates between classes For m classes, the classifier partitions the feature space into m decision regions The line or curve separating the classes is the decision boundary. In more than 2 dimensions this is a surface (e.g., a hyperplane) For 2 classes can view net output as a discriminant function y(x, w) where: y(x, w) = 1 if x in C 1 y(x, w) = - 1 if x in C 2 Need some training data with known classes to generate an error function for the network Need a (supervised) learning algorithm to adjust the weights

6 Linear discriminant functions A linear discriminant function is a mapping which partitions feature space using a linear function (a straight line, or a hyperplane) Thus in 2 dimensions the decision boundary is a straight line Simple form of classifier: separate the two classes using a straight line in feature space

7 The Perceptron as a Classifier For d-dimensional data perceptron consists of d-weights, a bias and a thresholding activation function. For 2D data we have: w1w1 w2w2 w0w0 a = w 0 + w 1 x 1 + w 2 x 2 y=g(a) {-1, +1} 1. Weighted Sum of the inputs 2. Pass thru Heaviside function: T(a)= -1 if a = 0 x1x1 1 Output = class decision If we group the weights as a vector w we therefore have the net output y given by: y = g(w. x + w 0 ) x2x2 View the bias as another weight from an input which is constantly on

8 Interpretation of weights Since Heaviside function is thresholded on 0 the decision boundary is at: w ||w|| w0w0 So perceptron functions as a linear discriminant function a = w. x + w 0 = w 0 + w 1 x 1 + w 2 x 2 = 0 Rearranging we get: x 2 = - (w 0 + w 1 x 1 )/w 2 unless w 2 =0 when we have x 1 = - w 0 /w 1 or w 2 = w 1 =0 when classification depends on sign of w 0

9 Perceptron can be extended to discriminate between k classes by having k output nodes: x is in class C j if y j (x)>= y k for all k Resulting decision boundaries divide the feature space into convex decision regions ykyk y j = g( w jk x k + w k0 ) w 11 y1y1 w kd w k1 w k0 x1x1 xdxd Weight to output j from input k is w jk 1 C1C1 C2C2 C3C3

10 Other activation functions can also be used (usually chosen to be monotonic). NB discriminant is still linear. Use of the sigmoidal logistic activation function: g(a) = 1/(1 + e -a ) together with data drawn from Gaussian or Bernoulli class- conditional distributions (P(x | C k )) means that the network outputs can be interpreted as the posterior probabilities P(C k | x) Generalised linear discriminants Linear discriminants can be made more general by including non- linear functions (basis functions) k which to transform the input data. Thus the outputs become: y j = g( w jk k + w k0 )

11 Network Learning Standard procedure for training the weights is by gradient descent For this process we have a set of training data from known classes to be used in conjunction with an error function E(w) (eg sum of squares error) to specify an error for each instantiation of the network Then do: w new = w old - E(w) So: where: (w) is a vector representing the gradient and is the learning rate (small, positive) 1. This moves us downhill in direction E(w) (steepest downhill since E(w) is the direction of steepest increase) 2. How far we go is determined by the value of

12 Moving Downhill: Move in direction of negative derivative E(w) w1w1 d E(w)/ dw 1 w1w1 Decreasing E(w) d E(w)/dw 1 > 0 w 1 <= w 1 - d E(w)/dw 1 i.e., the rule decreases w 1

13 E(w) w1w1 d E(w)/ dw 1 w1w1 Decreasing E(w) d E(w)/dw 1 < 0 w 1 <= w 1 - d E(w)/dw 1 i.e., the rule increases w 1 Moving Downhill: Move in direction of negative derivative

14 Illustration of Gradient Descent w1w1 w0w0 E(w)

15 Illustration of Gradient Descent w1w1 w0w0 E(w)

16 Illustration of Gradient Descent w1w1 w0w0 E(w) Direction of steepest descent = direction of negative gradient

17 Illustration of Gradient Descent w1w1 w0w0 E(w) Original point in weight space New point in weight space

18 General Gradient Descent Algorithm Define an objective function E(w) Algorithm: –pick an initial set of weights w, e.g. randomly –evaluate E(w) at w note: this can be done numerically or in closed form –update all the weights w new = w old - (w) –check if E(w) is approximately 0 if so, we have converged to a flat minimum if not, we move again in weight space

19 Equivalent to hill-climbing Can be problems knowing when to stop Local minima –can have multiple local minima (note: for perceptron, E(w) only has a single global minimum, so this is not a problem) –gradient descent goes to the closest local minimum: solution: random restarts from multiple places in weight space

20 Sequential Gradient Descent In standard gradient descent (batch version) get network output for all data points and estimate error gradient from difference between outputs and targets (for current weights) Sequential gradient descent: get an approximation to the full gradient based on the ith training vector x i only use: where E i is the error due to x i This allows us to update the weights as we cycle through each input - tends to be faster in practice - dont have to store all outputs and vectors - can be used to adapt weights on-line - can track slow moving changes in the data - stochasticity can help to escape from local minima

21 Error function Need to define an error function to start the training procedure Also need to define target functions t i for each input pattern x i in the training data set X: t i = 1 if pattern x i is in C 1 and –1 if x i is in C 2 An obvious starting point is to use number of training patterns that are currently misclassified Equivalent to sum of squares error function: E(w) = |y(x i ) – t i | = (1/4) (y(x i ) – t i ) 2 However, thinking about the resulting Error Surface highlights some bad properties of this error for gradient descent

22 In particular a smooth change in the weights w will not result in a smooth change in the error: E 4 5 w w w Either weight change has no effect on error Or a pattern is reclassified causing a discontinuity in the error surface This means we get no info from the error gradient (not great for a gradient descent procedure …) x o x o ie we cannot distinguish this: From this:

23 Therfore want an error function which takes into account the distance of misclassified patterns from the boundary Use the Perceptron Criterion: E perc (w) = - w.x i t i for all misclassified x i Since if x i is in C 1 but classified in C 2 then t i = +1 and w.x i =0 so w.x i t i = - w.x i < 0 w xixi w d w.x i Also from a geometrical interpretation of the weights we can show that w.x i is proportional to the absolute distance to the decision boundary

24 Applying the sequential gradient descent algorithm to this error function we get: w(t+1) = w(t) + x j t j for all misclassified x j Equivalently we can use: w(t+1) = w(t) + x j (t j – y j ) Which is a form of the adaline learning algorithm The Perceptron Convergence Theorem (Rosenblatt, 1962) states that this algorithm is guaranteed to converge to a solution for linearly separable data. The idea of the proof is to consider ||w(t+1)-w||-||w(t)-w|| which is a decreasing function of t (See eg Bishop (1995))

25 x x o o x3x3 w.x <0w.x > 0 (t 3 – y 3 ) = - 2 w(t)

26 x x o o x3x3 w.x <0w.x > 0 (t 3 – y 3 ) = - 2 x 3 (t 3 – y 3 ) w(t)

27 x x o o x3x3 w.x <0w.x > 0 w(t+1) = w(t) + x 3 (t 3 – y 3 )

28 x x o o x3x3 w.x <0w.x > 0 w(t+1)

29 The Fall of the Perceptron Marvin Minsky & Seymour Papert (1969). Perceptrons, MIT Press, Cambridge, MA. Before long researchers had begun to discover the Perceptrons limitations. Unless input categories were linearly separable, a perceptron could not learn to discriminate between them. Unfortunately, it appeared that many important categories were not linearly separable. E.g., those inputs to an XOR gate that give an output of 1 (namely 10 & 01) are not linearly separable from those that do not (00 & 11).

30 The Fall of the Perceptron Successful Unsuccessful Many Hours in the Gym per Week Few Hours in the Gym per Week Footballers Academics In this example, a perceptron would not be able to discriminate between the footballers and the academics… …despite the simplicity of their relationship: Academics = Successful XOR Gym This failure caused the majority of researchers to walk away.

31 The simple XOR example masks a deeper problem Consider a perceptron classifying shapes as connected or disconnected and taking inputs from the dashed circles in 1. In going from 1 to 2, change of right hand end only must be sufficient to change classification (raise/lower linear sum thru 0) Similarly, the change in left hand end only must be sufficient to change classification Therefore changing both ends must take the sum even further across threshold Problem is because of single layer of processing local knowledge cannot be combined into global knowledge. So add more layers...

32 THE PERCEPTRON CONTROVERSY There is no doubt that Minsky and Papert's book was a block to the funding of research in neural networks for more than ten years. The book was widely interpreted as showing that neural networks are basically limited and fatally flawed. What IS controversial is whether Minsky and Papert shared and/or promoted this belief ? Following the rebirth of interest in artificial neural networks, Minsky and Papert claimed that they had not intended such a broad interpretation of the conclusions they reached in the book Perceptrons. However, Jianfeng was present at MIT in 1974, and reached a different conclusion on the basis of the internal reports circulating at MIT. What were Minsky and Papert actually saying to their colleagues in the period after the publication of their book?

33 Minsky and Papert describe a neural network with a hidden layer as follows: GAMBA PERCEPTRON: A number of linear threshold systems have their outputs connected to the in- puts of a linear threshold system. Thus we have a linear threshold function of many linear threshold functions. Minsky and Papert then state: Virtually nothing is known about the computational capabilities of this latter kind of machine. We believe that it can do little more than can a low order perceptron. (This, in turn, would mean, roughly, that although they could recognize (sp) some relations between the points of a picture, they could not handle relations between such relations to any significant extent.) That we cannot understand mathematically the Gamba perceptron very well is, we feel, symptomatic of the early state of development of elementary computational theories.

34 In summary, Minsky and Papert, with intellectual honesty, confessed that they were not not able to prove that even with hidden layers, feed-forward neural nets were useless, but they expressed strong confidence that they were quite inadequate computational learning devices. NB Minsky and Papert restrict discussion to "linear threshold" rather than the sigmoid threshold functions prevalent in ANN. Conclusion? Dont believe everything you hear …

Download ppt "The Perceptron. Prehistory W.S. McCulloch & W. Pitts (1943). A logical calculus of the ideas immanent in nervous activity, Bulletin of Mathematical Biophysics,"

Similar presentations

Ads by Google