Download presentation
Presentation is loading. Please wait.
Published byZachary Byrd Modified over 10 years ago
1
1 Geometric Methods for Learning and Memory A thesis presented by Dimitri Nowicki To Universite Paul Sabatier in partial fulfillment for the degree of Doctor es Science in the subject of Applied Mathematics
2
2 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose
3
3 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose
4
4 Models and Algorithms requiring Geometric Approach Kalman–like filters Blind Signal Separation Feed-Forward Neural Networks Independent Component Analysis
5
5 Introduction Riemannian spaces Lie groups and homogeneous spaces Metric spaces without any Riemannian structure Spaces emerging in learning problems
6
6 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose
7
7 Outline Some facts from Riemannian geometry Optimization algorithms –Smooth –Nonsmooth Implementation –The case of Submanifolds –Computing exponential maps –Computing Hessian etc.
8
8 Some concepts from Riemannian Geometry Geodesics
9
9 Exponential map
10
10 Parallel transport Computing parallel transport using an exponential map Where u such that
11
11 Newton Method for Geometric optimization The modified Newton operator
12
12 Wolfe condition for Riemannian manifolds
13
13 Global convergence of modified Newton method
14
14 Nonsmooth methods The subgradient:
15
15 The r -algorithm. Here
16
16 Problem of constrained optimization Equality constraints
17
17 Classical (extrinsic) methods The Lagrangian Newton-Lagrange method Sequential quadratic programming
18
18 Classical methods Penalty functions and the augmented Lagrangian
19
19 Advantages of Geometric methods Dimension of the manifold is n-m against n+m in the case of Lagrangian-based methods We may have convex function in the manifold even if the Lagrangian is non- convex Geometric Hessian may be positive- definite even if the classical one is not
20
20 Implementation: The case of Submanifolds
21
21 Hamilton Equations for the Geodesics The Lagrangian: The Hamiltonian:
22
22 Hamilton Equations for the Geodesics
23
23 Lagrange equation are also constrained Hamiltonian We can rewrite Lagrange equations in the form:
24
24 Symplectic Numerical Integration A transformation is called symplectic if it preserves following differential 2-form:
25
25 Implicit Runge-Kutta Integrators The IRK method is called symplectic if associated transformation preserves y=(x,p)
26
26 The Gauss method of order 4 i=1i=2 j=11/4 j=2 1/4 1/2
27
27 Backward error analysis
28
28 Covariant Derivative on the Submanifold
29
29 Computing the constrained Hessian Direct computation Mixed computation where
30
30 Example of geometric iterations
31
31 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose
32
32 Neural Associative memory Hopfield-type auto-associative memory. Memorized vectors are bipolar: vk {-1, 1} n, k=1…m. Suppose these vectors are columns of n m matrix V. Then synaptic matrix C of the memory is given by: Associative recall is performed using following procedure: the input vector x 0 is a starting point of the iterations: where f is a monotonic odd function such that
33
33 Attraction radius We will call the stable fixed point of this discrete-time dynamical system an attractor. The maximum Hamming distance between x 0 and a memorized pattern v k such that the examination procedure still converges to v k is called an attraction radius.
34
34 Problem statement
35
35 Generalized averaging on the manifold argmin
36
36 Computing generalized average on the Grassmann manifold Generalized averaging as an optimization problem Transforming objective function:
37
37 Statistical estimation
38
38 Statistical estimation
39
39 Experimental results: the simulated data n=256– for all experiments Nature of the data
40
40 Experimental results: simulated data
41
41 Experimental results: simulated data Frequencies of attractors of associative clustering network for different m, p=8
42
42 Experimental results: simulated data Frequencies of attractors of associative clustering network for different p, and m=p
43
43 Experimental results: simulated data Distinction coefficients of attractors of associative clustering network for different p, and m=p
44
44 The MNIST database: data description Gray-scale images 28 28 10 classes: digits from 0 to 9 Training sample: 60000 images Test sample:10000 images Before entering to the network images were tresholded to obtain 784- dimensional bipolar vectors
45
45 Experimental results: the MNIST database Example of handwritten digits from MNIST database
46
46 Experimental results: the MNIST database Generalized images of digits found by the network
47
47 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose
48
48 Kernel AM The main algorithm
49
49 Kernel AM The Basic Algorithm (Continued)
50
50 Algorithm Scheme
51
51 Experimental Results Gaussian Kernel
52
52 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose
53
53 Model of Signal
54
54 Signal Trajectories in the phase space
55
55 The Manifold
56
56
57
57 Example of Signal Processing
58
58 Outline Introduction Geodesics, Newton Method and Geometric Optimization Generalized averaging over RM and Associative memories Kernel Machines and AM Quotient spaces for Signal Processing Application: Electronic Nose
59
59 Application for Real-Life Problem Electronic Nose: QCM Setup overview Variance Distribution between principal Components
60
60 Chemical images in space spanned by first 3 PCs
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.