Download presentation

Presentation is loading. Please wait.

Published byZachary Byrd Modified over 3 years ago

1
1 Geometric Methods for Learning and Memory A thesis presented by Dimitri Nowicki To Universite Paul Sabatier in partial fulfillment for the degree of Doctor es Science in the subject of Applied Mathematics

2
2 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose

3
3 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose

4
4 Models and Algorithms requiring Geometric Approach Kalman–like filters Blind Signal Separation Feed-Forward Neural Networks Independent Component Analysis

5
5 Introduction Riemannian spaces Lie groups and homogeneous spaces Metric spaces without any Riemannian structure Spaces emerging in learning problems

6
6 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose

7
7 Outline Some facts from Riemannian geometry Optimization algorithms –Smooth –Nonsmooth Implementation –The case of Submanifolds –Computing exponential maps –Computing Hessian etc.

8
8 Some concepts from Riemannian Geometry Geodesics

9
9 Exponential map

10
10 Parallel transport Computing parallel transport using an exponential map Where u such that

11
11 Newton Method for Geometric optimization The modified Newton operator

12
12 Wolfe condition for Riemannian manifolds

13
13 Global convergence of modified Newton method

14
14 Nonsmooth methods The subgradient:

15
15 The r -algorithm. Here

16
16 Problem of constrained optimization Equality constraints

17
17 Classical (extrinsic) methods The Lagrangian Newton-Lagrange method Sequential quadratic programming

18
18 Classical methods Penalty functions and the augmented Lagrangian

19
19 Advantages of Geometric methods Dimension of the manifold is n-m against n+m in the case of Lagrangian-based methods We may have convex function in the manifold even if the Lagrangian is non- convex Geometric Hessian may be positive- definite even if the classical one is not

20
20 Implementation: The case of Submanifolds

21
21 Hamilton Equations for the Geodesics The Lagrangian: The Hamiltonian:

22
22 Hamilton Equations for the Geodesics

23
23 Lagrange equation are also constrained Hamiltonian We can rewrite Lagrange equations in the form:

24
24 Symplectic Numerical Integration A transformation is called symplectic if it preserves following differential 2-form:

25
25 Implicit Runge-Kutta Integrators The IRK method is called symplectic if associated transformation preserves y=(x,p)

26
26 The Gauss method of order 4 i=1i=2 j=11/4 j=2 1/4 1/2

27
27 Backward error analysis

28
28 Covariant Derivative on the Submanifold

29
29 Computing the constrained Hessian Direct computation Mixed computation where

30
30 Example of geometric iterations

31
31 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose

32
32 Neural Associative memory Hopfield-type auto-associative memory. Memorized vectors are bipolar: vk {-1, 1} n, k=1…m. Suppose these vectors are columns of n m matrix V. Then synaptic matrix C of the memory is given by: Associative recall is performed using following procedure: the input vector x 0 is a starting point of the iterations: where f is a monotonic odd function such that

33
33 Attraction radius We will call the stable fixed point of this discrete-time dynamical system an attractor. The maximum Hamming distance between x 0 and a memorized pattern v k such that the examination procedure still converges to v k is called an attraction radius.

34
34 Problem statement

35
35 Generalized averaging on the manifold argmin

36
36 Computing generalized average on the Grassmann manifold Generalized averaging as an optimization problem Transforming objective function:

37
37 Statistical estimation

38
38 Statistical estimation

39
39 Experimental results: the simulated data n=256– for all experiments Nature of the data

40
40 Experimental results: simulated data

41
41 Experimental results: simulated data Frequencies of attractors of associative clustering network for different m, p=8

42
42 Experimental results: simulated data Frequencies of attractors of associative clustering network for different p, and m=p

43
43 Experimental results: simulated data Distinction coefficients of attractors of associative clustering network for different p, and m=p

44
44 The MNIST database: data description Gray-scale images 28 28 10 classes: digits from 0 to 9 Training sample: 60000 images Test sample:10000 images Before entering to the network images were tresholded to obtain 784- dimensional bipolar vectors

45
45 Experimental results: the MNIST database Example of handwritten digits from MNIST database

46
46 Experimental results: the MNIST database Generalized images of digits found by the network

47
47 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose

48
48 Kernel AM The main algorithm

49
49 Kernel AM The Basic Algorithm (Continued)

50
50 Algorithm Scheme

51
51 Experimental Results Gaussian Kernel

52
52 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose

53
53 Model of Signal

54
54 Signal Trajectories in the phase space

55
55 The Manifold

56
56

57
57 Example of Signal Processing

58
58 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose

59
59 Application for Real-Life Problem Electronic Nose: QCM Setup overview Variance Distribution between principal Components

60
60 Chemical images in space spanned by first 3 PCs

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google