1. Associative Memory by Recurrent Neural Networks with Delay Elements Seiji MIYOSHI Hiro-Fumi YANAI Masato OKADA Kobe City College of Tech. Ibaraki Univ. RIKEN BSI, ERATO KDB JAPAN JAPAN JAPAN miyoshi@kobe-kosen.ac.jp www.kobe-kosen.ac.jp/ ~ miyoshi/

2. Background Synapses of real neural systems seem to have delays. It is very important to analyze associative memory model with delayed synapses. Computer simulation is powerful method. Theoretical and analytical approach is indispensable to research on delayed networks. There is a Limit on the number of neurons.However, Yanai-Kim theory by using Statistical Neurodynamics Good Agreement with computer simulation Computational Complexity is O(L 4 t) Simulating network with large delay steps is realistically impossible.

3. Objective To derive macroscopic steady state equations by using discrete Fourier transformation To discuss storage capacity quantitatively even for a large L limit (L: length of delay)

4. Recurrent Neural Network with Delay Elements Model

5. Overlap Model Discrete Synchronous Updating Rule Correlation Learning for Sequence Processing

6. Macrodynamical Equations by Statistical Neurodynamics Yanai & Kim(1995) Miyoshi, Yanai & Okada(2002)

7. Initial Condition of the Network One Step Set Initial Condition Only the states of neurons are set explicitly. The states of delay elements are set to be zero. All Steps Set Initial Condition The states of all neurons and all delay elements are set to be close to the stored pattern sequences. If they are set to be the stored pattern sequences themselves ≡ Optimum Initial Condition setzero set

8. Dynamical Behaviors of Recall Process All Steps Set Intial Condition Loading rateα=0.5 Length of delay L=3 Simulation （ N=2000 ） Theory

9. Dynamical Behaviors of Recall Process All Steps Set Intial Condition Loading rateα=0.5 Length of delay L=2 Simulation （ N=2000 ） Theory

10. Loading rates α - Steady State Overlaps m Simulation （ N=500 ） Theory

11. Length of delay L - Critical Loading Rate α C

12. Macrodynamical Equations by Statistical Neurodynamics Yanai & Kim(1995) Miyoshi, Yanai & Okada(2002) Good Agreement with Computer Simulation Computational Complexity is O(L 4 t)

13. Macroscopic Steady State Equations Accounting for Steady State Parallel Symmetry in terms of Time Steps Discrete Fourier Transformation

14. Loading rates α - Steady State Overlaps m

15. Simulation （ N=500 ） Theory

16. Loading rate α - Steady State Overlap

17. Storage Capacity of Delayed Network Storage Capacity = 0.195 L

18. Conclusions Yanai-Kim theory (macrodynamical equations for delayed network) is re-derived. Steady state equations are derived by using discrete Fourier transformation. Storage capacity is 0.195 L in a large L limit. → Computational Complexity is O(L 4 t) → Intractable to discuss macroscopic properties in a large L limit → Computational complexity does not formally depend on L → Phase transition points agree with those under the optimum initial conditions, that is, the Storage Capacities !