1. 2806 Neural Computation Single Layer Perceptron Lecture 3 2005 Ari Visa

2. Agenda n Some historical notes n Linear adaptive filters n Rosenblatt’s perceptron n Conclusions

3. Some historical notes Rosenblatt’s perceptron 1958: a single neuron with adjustable synaptic weights and bias Perceptron convergence theorem 1962 (Rosenblatt): If patterns used to train the perceptron are drawn from two linearly separable classes, then the algorithm converges and positions the decision surface in the form of hyperplane between two classes.

4. Some historical notes n A single neuron -> an adaptive filter n Widrow & Hoff (1960): least-mean square algorithm (LMS) = delta rule n The problem of designing an optimum linear filter: Kolmogorov 1942, Wiener 1949, Zadeh 1953, Gabor 1954

5. Linear Adaptive Filters Consider a dynamic system with an unknown mathematical characterization A set of labeled input-output data generated by the system at discrete instants of time at some uniform rate is available

6. Linear Adaptive Filters The problem is also known as system identification (in control theory). An adaptive filter: 1) Filtering process (involves the computation of two signals; an output y(i) and an error signal e(i) = d(i)-y(i)) 2) Adaptive process (the automatic adjustment of the synaptic weights of the neuron in accordance with the error signal e(i)).

7. Linear Adaptive Filters n Note that y(i) = v(i) =  k m w k (i)x k (i) is written into matrix form y(i) = x T (i)w(i) n The manner in which e(i) = d(i)-y(i) is used to control the adjusments to the synaptic weights is determined by the cost function used to derive the adaptive filtering algorithm.

8. Linear Adaptive Filters n unconstrained optimization problem: n Minimize the cost function E(w) with respect to the weight vector w. n Operation analysis -> the necessary condition for optimality is  E(w*) = 0. n The strategy is to use iterative steepest descent: Starting with an initial guess w(0), generate a sequence of weight vectors w(1), w(2),..., such that the cost function E(w) is reduced at each iteration of the algorithm as E(w(n+1) < Ew(n)).

9. Linear Adaptive Filters n 1) Method of Steepest Descent n The direction of steepest descent is in direction opposite to the gradient vector g =  E(w) n w(n+1) = w(n) –  g(n) n  is the stepsize or learning-rate parameter

10. Linear Adaptive Filters n The method of steepest descent converges to the optimal solution w* slowly. n The learning-rate parameter  has an influence on the convergence behaviour: - When  is small, the transient response of the algorithm is overdamped (3.2a) - When  is large, the transient response of the algorithm is underdamped (3.2b) – - When  exceeds a certain critical value, the algorithm becomes unstable.

11. Linear Adaptive Filters n 2) Newton’s method n The idea is to minimize the quadratic approximation of the cost function E(w) around the current point w(n). n Using a second-order Taylor series expansion of the cost function around the point w(n). n  Ew(n)  g T (n)  w(n) +½  w T (n) H(n)  w(n) n g(n) is the m-by-1 gradient vector of the cost function E(w) evaluated at the point w(n). The matrix H(n) is the m-by-m Hessian matrix of E(w) (second derivative), H =  ²E(w)

12. Linear Adaptive Filters n H =  ²E(w) requires the cost function E(w) to be twice continuously differentiable with respect to the elements of w. n Differentiating  Ew(n)  g T (n)  w(n) +½  w T (n) H(n)  w(n) with respect to  w, the change  E(n) is minimized when n g(n) + H(n)  w(n) = 0 →  w(n) = H -1 (n)g(n) n w(n+1) = w(n) +  w(n) n w(n+1) = w(n)+H -1 (n)g(n) n where H -1 (n) is the inverse of the Hessian of E(w).

13. Linear Adaptive Filters n Newton’s method converges quickly asymtotically and does not exhibit the zigzagging behavior. n Newton’s method requires that the Hessian H(n) has to be a positive definite matrix for all n!

14. Linear Adaptive Filters n 3) Gauss-Newton Method n It is applicable to a cost function that is expressed as the sum of error squares. n E(w) = ½  i=1 n e ² (i), note that all the error terms are calculated on the basis of a weight vector w that is fixed over the entire observation interval 1  i  n. n The error signal e(i) is a function of the adjustable weight vector w. Given an operating point w(n), we linearize the dependence of e(i) on w by writing e’(i,w) = e(i) + [  e(i)/  w] T w=w(n) (w-w(n)), i=1,2,...,n

15. Linear Adaptive Filters e’(n,w) = e(n) + J(n)(w-w(n)) where e(n) is the error vector e(n) = [e(1),e(2),...,e(n)] T and J(n) is the n-by-m Jacobian matrix of e(n) (The Jacobian J(n) is the transpose of the m-by-n gradient matrix  e(n), where  e(n) =[  e(1),  e(2),...,  e(n)]. w(n+1) = arg min w {½  e’(n,w)  ²}= ½  e(n)  ² +e T (n)J(n)(w-w(n)) + ½(w- w(n)) T J T (n)J(n)(w-w(n)) Differentiating the expression with respect w and setting the result equal to zero

16. Linear Adaptive Filters J T (n)e(n) + J T (n)e(n) (w-w(n)) = 0 w(n+1) = w(n) – ( J T (n)J(n)) -1 J T (n)e(n) The Gauss-Newton requires only the Jacobian matrix of the error function e(n). For the Gauss-Newton iteration to be computable, the matrix product J T (n)J(n) must be nonsigular. J T (n)J(n) is always nonnegative definite but to ensure that it is nonsingular, the Jacobian J(n) must have row rank n. → add the diagonal matrix  I to the matrix J T (n)J(n), the parameter  is a small positive constant.

17. Linear Adaptive Filters n J T (n)J(n)+  I ; positive definite for all n. n -> The Gauss-Newton method is implemented in the following form: w(n+1) = w(n) – ( J T (n)J(n) +  I ) -1 J T (n)e(n) n This is the solution to the modified cost function: n E(w) = ½{  w-w(0)  ²+  i=1 n e ² (i)} n where w(0) is the initial value of w.

18. Linear Adaptive Filters Linear Least-Square Filter: - the single neuron is linear - the cost function e(w) used to design the filter consists of the sum of error squares e(n) = d(n) –X(n)w(n), where d(n) is the n-by- 1 desired response vector and X(n) is the n- by-m data matrix.

19. Linear Adaptive Filters w(n+1) =w(n) + ( X T (n)X(n)) -1 X T (n)(d(n) – X(n)w(n)) = ( X T (n)X(n)) -1 X T (n)d(n) where (X T (n)X(n)) -1 X T (n) is recognized as the pseudoinverse of the data matrix X(n) The weight vector w(n+1) solves the linear least-squares problem defined over an observation interval of duration n.

20. Linear Adaptive Filters Wiener Filter: ergodic environment that is also stationary -> We may substitute long- term sample averages/time-averages for expectations/ensemble averages Correlation matrix of the input vector x(i): R x Cross-correlation vector between the input vector x(i) and desired response d(i): rxd

21. Linear Adaptive Filters R x = E[x(i)x T (i)] = lim n  1/n X T (n)X(n) r xd = E[x(i)d(i)] = lim n  1/n X T (n)d(n) w o = lim n  w(n+1) = R x -1 r xd where R x -1 is the inverse of the correlation matrix R x. The weight vector w o is called the Wiener solution to the linear optimum filtering problem.

22. Linear Adaptive Filters For an ergodic process, the linear least-squares filter asymtotically approaches the Wiener filter as the number of observations approaches infinity. n The least-mean square (LMS) algorithm is based on the use of instantaneous values for the cost function. n E(w) = ½ e ² (n), where e(n) is the error signal measured at time n.

23. Linear Adaptive Filters n e(n) = d(n) - x T (n)w(n) -> an estimate for the gradient vector g^(n) = -x(n)e(n) n -> w^(n+1) = w^(n) +  x(n)e(n) n where  is the learning -rate parameter

24. Linear Adaptive Filters The feedback loop around the weight vector w^(n) in the LMS algorithm behaves like a low-pass filter. The inverse of the learning-rate parameter  is a measure of the memory of the LMS algorithm.

25. Linear Adaptive Filters w^(n+1) = w^(n) +  x(n)[d(n) - x T (n) w^(n)] w^(n) = z -1 [w^(n+1)] A stochastic feedback system The learning-rate parameter  and the input vector x(n) that determine the transmittance of the feedback loop

26. Linear Adaptive Filters The convergence issue that matters is convergence in the mean square E[e²(n)]  constant as n  Assumptions: 1) the successive input vectors are statistically independent of each other 2) At time step n the input vector is statistically independent of all previous samples of the desired response.

27. Linear Adaptive Filters 3) At time step n, the desired response is dependent on the input, but statistically independent of all previous values of the desired response. 4) The input vector and the desired response are drawn from Gaussian distributed populations. The convergence issue is guaranted if 0 <  < 2/(sum of mean-square values of the sensor inputs) The LMS algorithm is model independent and therefore robust, which means that small model uncertainy and small disturbances can only result in small estimation errors.

28. Linear Adaptive Filters LMS algorithm is slow in convergence and sensitive to variations in the eigenstructure of the input. The LMS algorithm typically requires a number of iteration equal to about ten times the dimensionality of the input space for it to reach a steady-state condition. The LMS algorithm is particularly sensitive to variations in the condition number (=eigenvalue spread) of the correlarion matrix R x of the input vector x.  (R x )  max  min

29. Linear Adaptive Filters The learning curve is a plot of the mean-square value of the estimation error E av (n) versus the number of iterations n. Rate of convergence; the number of iterations n required for E av (n) to decrease to some arbitrarily chosen value Misadjustment M = E(  )/E min -1

30. Linear Adaptive Filters Difficulties encountered with the LMS algorithm may be attributed to the fact that the learning-rate parameter is maintained constant.  (n) = c/n where c is a conctant. Search-then-converge schedule  (n) =  0 /(1+(n/t)) where t is a search time constant

31. Perceptron n Perceptron is based on a nonlinear neuron. n v =  i=1 m w i x i + b n In the simplest form of the perceptron there are two decision regions reparated by a hyperplane.

32. Perceptron x(n) = [+1,x 1 (n),x 2 (n),...,x m (n)] T w(n) = [b(n),w 1 (n),w 2 (n),...,x m (n)] T v(n) = w T (n)x(n) defines a hyperplane

33. Perceptron 1. If the nth member of the training set, x(n), is correctly classified by the weight vector w(n) computed at the nth iteration of the algorithm, no correction is made to the weight vector of the perceptron in accordance with the rule: n w(n+1) =w(n) if w T x(n) >0 and x(n) belongs to class L 1 n w(n+1) =w(n) if w T x(n)  0 and x(n) belongs to class L 1

34. Perceptron n 2. Otherwise the weight vector of the perceptron is uppdated in accordance with the rule n w(n+1) =w(n) -  (n)x(n) if w T (n)x(n) >0 and x(n) belongs to class L 2 n w(n+1) =w(n) +  (n)x(n) if w T (n)x(n)  0 and x(n) belongs to class L 1 n where the learning-rate parameter  (n) controls the adjustment applied to the weight vector at iteration n.

35. Perceptron Fixed-increment convergence theorem: Let the subsets of training vectors H 1 and H 2 be linearly separable. Let the inputs presented to the perceptron originate from these two subsets. The perceptron converges after some n 0 iterations, in the sense that w(n 0 ) =w(n 0 +1)=w(n 0 +2)=... is a solution vector for n 0  n max.

36. Perceptron n The Perceptron Convergence Algorithm

37. Summary The LMS algorithm uses a linear neuron. Continuous learning takes place. The perceptron uses the McCulloch-Pitts formal model of a neuron. The learning process is performed for a finite number of iterations and then stops.