We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published bySilvester Rich
Modified about 1 year ago
Template design only ©copyright 2008 Ohio UniversityMedia Production 740.597-2521 Spring Quarter A hierarchical neural network structure for text learning is obtained through self-organization Similar representation for text based semantic network was used  An input layer takes in characters, then learns and activates words stored in memory Direct activation of words requires large computational cost for large dictionaries Extension to phrases, sentences or paragraphs would render such a network impractical due to associated computational cost Computer memory required would also be tremendously large This leads to a sparse hierarchical structure. The higher layers represent more complex concepts. Basic nodes in this network are capable of differentiating input sequences. Sequence learning is prerequisite to building spatio-temporal memories. This is performed using laminar minicolumn  LTM cells (Fig.1) In such networks, the interconnection scheme is naturally obtained through sequence learning and structural self-organization. No prior assumption about locality of connections or structure sparsity is made. Machine learns only inputs useful to its objectives, a process that is regulated by reinforcement signals and self organization. Hierarchical Neural Network for Text Based Learning Janusz A Starzyk, Basawaraj Ohio University, Athens, OH Introduction References Hierarchical Network Traditional approach is to describe semantic network structure and/or probabilities of transition in associated Markov models Biological networks learn Different Neural Network structures, but common goal Simple and efficient to solve the given problem Sparsity is essential Size of the network and time to train important for large data sets Hierarchical structure of identical processing units was proposed  Layered organization and sparse structure is biologically inspired Neurons on different layers interact through trained links Mountcastle, V. B., et. al, Response Properties of Neurons of Cat’s Somatic Sensory Cortex to Peripheral Stimuli, J. Neurophysiology, vol. 20, 1957, pp. 374-407. Rogers, T. T., McClelland, J. L., Semantic Cognition text: A parallel Distributed Processing Approach, 2004, MIT Press. Grossberg, S., How does the cerebral cortex work? Learning, attention and grouping by the laminar circuits of visual cortex. Spatial Vision, 12, 163-186,1999. Starzyk J.A., Liu Y., Hierarchical spatio-temporal memory for machine learning based on laminar minicolumn structure, 11 th ICCNS, Boston, 2007. Network Simplification Proposed approach uses intermediate neurons to lower the computational cost Intermediate neurons decrease number of activations associated with higher level neurons This concept can be extended to associations of words Small number of rules for concurrent processing are used We can arrive at local optimum of network structure / performance The network topology is self-organizing through addition and removal of neurons and redirecting of neuron connections Neurons are described by their sets of input and output neurons Local optimization criteria are checked by searching the set SL A before the structure is updated when creating or merging the neurons. where IL X is the input list of neuron X and OL A is the output list of neuron A Fig. 3 If create a new node C. B A B Fig. 4 Neuron “A” with single output is merged with Neuron “B”, and “A” is removed. B A A Fig. 5 Neuron “B” with single input is merged with Neuron “A”, and “B” is removed. Batch Mode: All words used for training are available at initiation. Network simplification & optimization is done by processing - all the words in the training set. Total number of neurons is 23% higher than the reference (6000) Dynamic Mode: Words used for training are increased incrementally, - one word at a time Simplification & optimization is done by processing - one word at a time. Total number of neurons is 68% higher than the reference (6000) Implementation Results and Conclusion Tests were run with dictionary up to 6000 words The percent reduction in number of interconnections increases (by up to 65 – 70%) as the number of words increase. The time required to process network activation for all the words used decreases as the number of words increases (reduction by a factor of 55, in batch mode; and 35, in dynamic mode; for 6000 words). Dynamic implementation takes longer compared to the batch implementation, mainly due to the additional overhead required for bookkeeping. The savings (connections and activations) obtained in case of dynamic implementation are less compared to the batch implementation Combination of both methods is advisable for continuous learning and self-organization. A B X4X4 X1X1 X2X2 X3X3 AB X4X4 C X1X1 X2X2 X3X3 Rules for Self-Organization Fig. 2 If create a new node C. X1X1 B X3X3 X4X4 X2X2 A B A C X4X4 X3X3 X2X2 X1X1 Few simple rules used for self organization Rules Contd.
.Edward G. Jones, Microcolumns in the Cerebral Cortex, Proc. of National Academy of Science of United States of America, vol. 97(10), 2000, pp
 B. Fritzke, “A Growing Neural Gas Network Learns Topologies,” Advances in Neural Information Processing Systems 7, G. Tesauro, D.S. Toretzky, and T.K.
Hybrid Pipeline Structure for Self-Organizing Learning Array Yinyin Liu 1, Ding Mingwei 2, Janusz A. Starzyk 1, 1 School of Electrical Engineering & Computer.
Sparse Coding in Sparse Winner networks Janusz A. Starzyk 1, Yinyin Liu 1, David Vogel 2 1 School of Electrical Engineering & Computer Science Ohio University,
Associative Learning in Hierarchical Self Organizing Learning Arrays Janusz A. Starzyk, Zhen Zhu, and Yue Li School of Electrical Engineering and Computer.
EE141 1 Broca’s area Pars opercularis Motor cortexSomatosensory cortex Sensory associative cortex Primary Auditory cortex Wernicke’s area Visual associative.
Neural Network with Memory and Cognitive Functions Janusz A. Starzyk, and Yue Li School of Electrical Engineering and Computer Science Ohio University,
What is a neural network? Collection of interconnected neurons that compute and generate impulses. Components of a neural network include neurons, synapses,
1 OPTIMIZED INTERCONNECTIONS IN PROBABILISTIC SELF-ORGANIZING LEARNING Janusz Starzyk, Mingwei Ding, Haibo He School of EECS Ohio University, Athens, OH.
A Hybrid Self-Organizing Neural Gas Network James Graham and Janusz Starzyk School of EECS, Ohio University Stocker Center, Athens, OH USA IEEE World.
EE141 1 Design of Self-Organizing Learning Array for Intelligent Machines Janusz Starzyk School of Electrical Engineering and Computer Science Heidi Meeting.
“Principles of Soft Computing, 2 nd Edition” by S.N. Sivanandam & SN Deepa Copyright 2011 Wiley India Pvt. Ltd. All rights reserved. CHAPTER 2 ARTIFICIAL.
Hierarchical Temporal Memory as a Means for Image Recognition by Wesley Bruning CHEM/CSE 597D Final Project Presentation December 10, 2008.
B.Macukow 1 Lecture 3 Neural Networks. B.Macukow 2 Principles to which the nervous system works.
Why Can't A Computer Be More Like A Brain?. Outline Introduction Turning Test HTM ◦ A. Theory ◦ B. Applications & Limits Conclusion.
Chapter Seven The Network Approach: Mind as a Web.
A Scalable Self-organizing Map Algorithm for Textual Classification: A Neural Network Approach to Thesaurus Generation Dmitri G. Roussinov Department of.
Advances in Modeling Neocortex and its impact on machine intelligence Jeff Hawkins Numenta Inc. VS265 Neural Computation December 2, 2010 Documentation.
Self organizing maps 1 iCSC2014, Juan López González, University of Oviedo Self organizing maps A visualization technique with data dimension reduction.
Fourth International Symposium on Neural Networks (ISNN) June 3-7, 2007, Nanjing, China Online Dynamic Value System for Machine Learning Haibo He, Stevens.
CS 478 – Tools for Machine Learning and Data Mining Perceptron.
Chapter 16. Basal Ganglia Models for Autonomous Behavior Learning in Creating Brain-Like Intelligence, Sendhoff et al. Course: Robots Learning from Humans.
Mental Development and Representation Building through Motivated Learning Janusz A. Starzyk, Ohio University, USA, Pawel Raif, Silesian University of Technology,
Analog Circuits for Self-organizing Neural Networks Based on Mutual Information Janusz Starzyk and Jing Liang School of Electrical Engineering and Computer.
Brain-like design of sensory-motor programs for robots G. Palm, Uni-Ulm.
Biologically Inspired Robotics Group,EPFL Associative memory using coupled non-linear oscillators Semester project Final Presentation Vlad TRIFA.
2 2 Background Vision in Human Brain Efficient Coding Theory Motivation Natural Pictures Methodology Statistical Characteristics Models.
COGNITIVE NEUROSCIENCE. Note Please read book to review major brain structures and their functions Please read book to review brain imaging techniques.
MACHINE LEARNING 12. Multilayer Perceptrons. Neural Networks Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)
METU Informatics Institute Min720 Pattern Classification with Bio-Medical Applications Part 8: Neural Networks.
Symbolic Reasoning in Spiking Neurons: A Model of the Cortex/Basal Ganglia/Thalamus Loop Terrence C. Stewart Xuan Choo Chris Eliasmith Centre for Theoretical.
1 Lecture 5 Neural Control. 2 History Early stages 1943 McCulloch-Pitts: neuron, origins 1948 Wiener: cybernatics 1949 Hebb: learning rule 1958.
Neural Networks Lab 5. What Is Neural Networks? Neural networks are composed of simple elements( Neurons) operating in parallel. Neural networks are composed.
Self Organizing Feature Map CS570 인공지능 이대성 Computer Science KAIST.
Efficient Pairwise Key Establishment Scheme Based on Random Pre-Distribution Keys in Wireless Sensor Networks Source: Lecture Notes in Computer Science,
Hybrid BDD and All-SAT Method for Model Checking Orna Grumberg Joint work with Assaf Schuster and Avi Yadgar Technion – Israel Institute of Technology.
Un Supervised Learning & Self Organizing Maps. Un Supervised Competitive Learning In Hebbian networks, all neurons can fire at the same time Competitive.
Learning in Neural and Belief Networks - Feed Forward Neural Network 2001 년 3 월 28 일 안순길.
IJCNN, International Joint Conference on Neural Networks, San Jose 2011 Pawel Raif Silesian University of Technology, Poland, Janusz A. Starzyk Ohio University,
PRIN WOMEN PROJECT Research Unit: University of Naples Federico II G. Ferraiuolo
Brunning – Chapter 2 Sensory, Short Term and Working Memory.
Soft Computing Lecture 12 Self-organizing maps of Kohonen RBF-networks.
Fourth International Symposium on Neural Networks (ISNN) June 3-7, 2007, Nanjing, China A Hierarchical Self-organizing Associative Memory for Machine Learning.
Presented By Dr. Paul Cottrell Company: Reykjavik.
Information Processing Approach Define cognition and differentiate among the stage, levels-of-processing, parallel distributed processing, and connectionist.
Spatio-Temporal Sequence Learning of Visual Place Cells for Robotic Navigation presented by Nguyen Vu Anh date: 20 th July, 2010 Nguyen Vu Anh, Alex Leng-Phuan.
Bayesian Brain - Chapter 11 Neural Models of Bayesian Belief Propagation Rajesh P.N. Rao Summary by B.-H. Kim Biointelligence Lab School of.
Chapter 8 NEURAL NETWORKS FOR DATA MINING. Learning Objectives Understand the concept and different types of artificial neural networks (ANN) Learn the.
Chapter 3 Neural Network Xiu-jun GONG (Ph. D) School of Computer Science and Technology, Tianjin University
© 2017 SlidePlayer.com Inc. All rights reserved.