1. Neural Network with Memory and Cognitive Functions Janusz A. Starzyk, and Yue Li School of Electrical Engineering and Computer Science Ohio University, Athens, OH 45701, U.S.A. David D. Vogel Ross University School of Medicine, Commonwealth of Dominica

2. 2 Organization Motivation R-nets Statistical model of R-nets Simulation results of statistical model Simulation results of R-nets Conclusions

3. 3 Perception Capacity to learn and memorize useful things Spatio-temporal memories Motor skills in relation to sensing and anticipation Surviving in complex environment and adaptation Value driven behavior (formulation of goals and their implementation) Abstract thinking and action planning Ability to communicate Intuition and creativity Consciousness Motivation To design an intelligent machine capable of

4. 4 Problems of Classical AI Lack of robustness and generalization No real-time processing Central processing of information by a single processor No natural interface to environment No self-organization Need to write software

5. 5 Intelligent Behavior Emergent from interaction with environment Based on large number of sparsely connected neurons Asynchronous Interact with environment through sensory- motor system Value driven Adaptive Coordinated through agent’s –multiple sensory-motor modalities –constraints from morphology and materials –generation of correlations through physical process –basis for cross-modal associations From http://tokyolectures.org/

6. 6 Sparse Connectivity The brain is sparsely connected. (Unlike most neural nets.) A neuron in cortex may have on the order of 100,000 synapses. There are more than 10 10 neurons in the brain. Fractional connectivity is very low: 0.001%. Implications: Connections are expensive biologically since they take up space, use energy, and are hard to wire up correctly. Therefore, connections are valuable. The pattern of connection is under tight control. Short local connections are cheaper than long ones. Our model makes extensive use of local connections.

7. 7 Human Brain at Birth 6 Years Old 14 Years Old Neuron Structure and Self-Organizing Principles

8. 8 “ The basic unit of cortical operation is the minicolumn … It contains of the order of 80-100 neurons except in the primate striate cortex, where the number is more than doubled. The minicolumn measures of the order of 40-50  m in transverse diameter, separated from adjacent minicolumns by vertical, cell-sparse zones … “ (VB Mountcastle (2003). Introduction [to a special issue of Cerebral Cortex on columns]. Cerebral Cortex, 13, p. 2) Stain of cortex in planum temporale. Cortical Minicolumns

9. 9 R-nets R-nets are defined as randomly connected artificial neural networks with primary and secondary neurons They implement distributed memories able to recall input patterns R-net with 10 6 excitatory neurons and brain-like connectivity will store at least 2x10 8 bits of information R-nets have been used to construct networks with higher cognitive functions. A statistical model of R-nets is presented and results are compared with simulated R-nets.

10. 10 Properties of R-nets Biological plausibility Similar to Hebbian networks Sparsly connected Large storage capacities Use inhibition to prevent neurons not associated with a recalled pattern from firing Store sequential patterns and attributes From Marr, D. (1971). Simple memory: a theory for archicortex. Philosophical Transaction of the Royal Society of London, B. 262, 23 ‑ 81.

11. 11 R-nets During training, an input pattern is presented to the R-net by activating a selected cluster C of primary neurons. All links between active primary neurons are trained. During recall a subset of one of the stored patterns is presented to the input, activating corresponding primary neurons (initial recall set). links from initial primary neuron 1 to destination primary neurons 1 and 2 are untrained. So is the link from initial primary neuron 2 to destination primary neuron 1. The only trained link is from initial primary neuron 2 to destination primary neuron 2. Both secondary neurons are strongly activated.

12. 12 R-nets During recall, secondary neurons sum the weighted projections of active primary neurons a i,x represents the activity if the ith secondary neuron in the xth cycle; a e,x is the current activity of the eth primary neuron with possible values 0 or 1; W i,e is the weight of the projection of the eth primary neuron onto the ith secondary neuron with possible values of 1 (untrained) or 10 (trained).

13. 13 R-nets The primary neurons are synchronously updated. I e,x is the inhibition of the eth primary neuron on the xth cycle, I i,x-1 is a function of the ith secondary neuron, which is given by: and the projection of the ith secondary neuron is untrained and the projection of the ith secondary neuron is trained if

14. 14 Statistical Model of R-nets P-primary neurons S-secondary neurons K p -primary neurons’ outgoing sets K s -secondary neurons’ outgoing sets Set of primary neurons linked to ci

15. 15 Statistical Model of R-nets The expected value of the number of different primary neurons reaching to (or reached from) a secondary neuron is The expected value of the number of secondary neurons reached from (or reaching to) a primary neuron is The number of primary neurons linked to a given primary neuron is

16. 16 Spurious neurons Spurious neuron Is not a part of the original pattern but is activated during the recall process. The probability that a potentially spurious neuron cj, is inhibited depends on the probability of an inhibitory link from an active primary neuron. Alternatively, a neuron may be spurious if it is not linked to any activated neurons at all.

17. 17 Spurious neurons A neuron is spurious if it satisfies the following conditions: a) it has no projection from S wa b) all its projections from S sa are trained. S sa is the strongly activated set of secondary neurons and S wa is the weakly activated set of secondary neurons

18. 18 Eliminating spurious neurons The probability that a secondary node y belongs to S a is The probability that a node in S a is strongly activated is approximately The probability that a potentially spurious node, z, is not linked to any node in S wa is

19. 19 Eliminating spurious neurons The probability that a primary node z not linked to S wa is connected to a node in S sa is The probability that z is a spurious node is The probability that the recall set has no more than S max spurious neurons is

20. 20 Missing neurons A missing neuron is a neuron from C-Cr which is suppressed by an inhibitory projection to an activated primary neuron. Lemma Each missing neuron is suppressed by an inhibitory link to a spurious neuron connected through a secondary node w, where w is different from all nodes in activated set S a.

21. 21 Eliminating missing neurons The probability that a given primary node will be missing due to a single spurious node can be estimated to be less than The probability that a single primary neuron is missing is The probability that the recall set has no more than S max missing neurons is

22. 22 Results of the Statistical Model Storage capacities of R-nets with 4000 and 64,000 primary neurons The statistical model is in a good agreement with simulated R-nets and can be applied to estimate the computational performance of very large R-nets.

23. 23 Results of the Statistical Model Storage capacities of R-nets for 100 and 150 neurons patterns

24. 24 Results of the Statistical Model Storage capacity of R- nets with 10 6 up to 10 7 primary neurons 10 5 to 2x10 6 secondary neurons 20 to 160 neuron patterns Up to 1000 projections from each primary neuron to secondary neurons with Up to 5000 projections from each secondary neuron back to primary neurons.

25. 25 Simulations of R-nets storage capacity for 1 cycle of recall of an R-net with 4000 primary neurons

26. 26 Iterative operation of R-nets Quick suppression of the spurious neuronsSpurious neurons suppressed after 14 cycles

27. 27 Iterative operation of R-nets Oscillations in the recovery processSpurious neurons destroy the recovery process

28. 28 Conclusion When the training sets are small, errors are due to the firing of spurious neurons with few links to the recall sets When the training sets are large, errors are chiefly due to over-training Storage capacity grows faster than the number of primary neurons with the slope of growth close to 10/7 on the logarithmic scale R-net with 10 6 primary neurons can store more than 130,000 100-neuron sets When R-net size reaches 10 9 primary neurons (with average number of projections 10 4 - similar to interconnection density of human brain), the network can store over 10 9 patterns of 150 neurons each