1 Geometric Methods for Learning and Memory A thesis presented by Dimitri Nowicki To Universite Paul Sabatier in partial fulfillment for the degree of Doctor es Science in the subject of Applied Mathematics

2 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose

3 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose

4 Models and Algorithms requiring Geometric Approach Kalman–like filters Blind Signal Separation Feed-Forward Neural Networks Independent Component Analysis

5 Introduction Riemannian spaces Lie groups and homogeneous spaces Metric spaces without any Riemannian structure Spaces emerging in learning problems

6 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose

7 Outline Some facts from Riemannian geometry Optimization algorithms –Smooth –Nonsmooth Implementation –The case of Submanifolds –Computing exponential maps –Computing Hessian etc.

8 Some concepts from Riemannian Geometry Geodesics

9 Exponential map

10 Parallel transport Computing parallel transport using an exponential map Where u such that

11 Newton Method for Geometric optimization The modified Newton operator

12 Wolfe condition for Riemannian manifolds

13 Global convergence of modified Newton method

14 Nonsmooth methods The subgradient:

15 The r -algorithm. Here

16 Problem of constrained optimization Equality constraints

17 Classical (extrinsic) methods The Lagrangian Newton-Lagrange method Sequential quadratic programming

18 Classical methods Penalty functions and the augmented Lagrangian

19 Advantages of Geometric methods Dimension of the manifold is n-m against n+m in the case of Lagrangian-based methods We may have convex function in the manifold even if the Lagrangian is non- convex Geometric Hessian may be positive- definite even if the classical one is not

20 Implementation: The case of Submanifolds

21 Hamilton Equations for the Geodesics The Lagrangian: The Hamiltonian:

22 Hamilton Equations for the Geodesics

23 Lagrange equation are also constrained Hamiltonian We can rewrite Lagrange equations in the form:

24 Symplectic Numerical Integration A transformation is called symplectic if it preserves following differential 2-form:

25 Implicit Runge-Kutta Integrators The IRK method is called symplectic if associated transformation preserves y=(x,p)

26 The Gauss method of order 4 i=1i=2 j=11/4 j=2 1/4 1/2

27 Backward error analysis

28 Covariant Derivative on the Submanifold

29 Computing the constrained Hessian Direct computation Mixed computation where

30 Example of geometric iterations

31 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose

32 Neural Associative memory Hopfield-type auto-associative memory. Memorized vectors are bipolar: vk {-1, 1} n, k=1…m. Suppose these vectors are columns of n m matrix V. Then synaptic matrix C of the memory is given by: Associative recall is performed using following procedure: the input vector x 0 is a starting point of the iterations: where f is a monotonic odd function such that

33 Attraction radius We will call the stable fixed point of this discrete-time dynamical system an attractor. The maximum Hamming distance between x 0 and a memorized pattern v k such that the examination procedure still converges to v k is called an attraction radius.

34 Problem statement

35 Generalized averaging on the manifold argmin

36 Computing generalized average on the Grassmann manifold Generalized averaging as an optimization problem Transforming objective function:

37 Statistical estimation

38 Statistical estimation

39 Experimental results: the simulated data n=256– for all experiments Nature of the data

40 Experimental results: simulated data

41 Experimental results: simulated data Frequencies of attractors of associative clustering network for different m, p=8

42 Experimental results: simulated data Frequencies of attractors of associative clustering network for different p, and m=p

43 Experimental results: simulated data Distinction coefficients of attractors of associative clustering network for different p, and m=p

44 The MNIST database: data description Gray-scale images classes: digits from 0 to 9 Training sample: images Test sample:10000 images Before entering to the network images were tresholded to obtain 784- dimensional bipolar vectors

45 Experimental results: the MNIST database Example of handwritten digits from MNIST database

46 Experimental results: the MNIST database Generalized images of digits found by the network

47 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose

48 Kernel AM The main algorithm

49 Kernel AM The Basic Algorithm (Continued)

50 Algorithm Scheme

51 Experimental Results Gaussian Kernel

52 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose

53 Model of Signal

54 Signal Trajectories in the phase space

55 The Manifold

56

57 Example of Signal Processing

58 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose

59 Application for Real-Life Problem Electronic Nose: QCM Setup overview Variance Distribution between principal Components

60 Chemical images in space spanned by first 3 PCs