Dialog Processing with Unsupervised Artificial Neural Networks Andrew Richardson Thomas Jefferson High School for Science and Technology Computer Systems Laboratory

Dialog Processing with Unsupervised Neural Networks Contents: What I did... Background (unsupervised neural networks) Program Mechanics Attributes of Nodes Attributes of Connections Lessons from Neurobiology Algorithms Further Research

What I Did Interest in Neural Networks (Unsupervised) Most researchers use Supervised NN's (Boring) Theory's really complicated Learning from brains... I found a new Field! (Cognitive Science) Too complicated for now Program a failure

Background: Neural Networks Outside of research, the neural networks used today are supervised, such that output for an input is matched against the right answer, and connections that produce the right answer are reinforced. The idea is that connections which have been right in the past will be right in the future.

Background: Unsupervised Neural Networks, or a Connectionist Model However, I think that unsupervised neural networks have more promise for complex tasks. This is more analogous to the neurons within the brain. Instead of affecting the network in a series of supervised tests, the network is systematically modified as a series of inputs, such as words, are read in. In an attempt to mimic the brain, my network reinforces connections between nodes that often fire one after the other. In this case, each word is represented by a node.

Program Mechanisms: Nodes However, it's not as simple as that. If the brain only noted connections between words, it wouldn't note connections to emotions or abstract ideas. In order to mimic these attributes of the brain, the ones that really think, nodes are added to the network that do not represent words. These take on meaning as they build connections to words and to each other. In time, they may let the network form complex ideas represented by nodes that have been influenced by the input text.

Program Mechanisms: Attributes of Nodes Like neurons in the human brain, nodes in my program vary in a variety of ways. Plasticity: A measure of how easy it is to modify the connections to and from this node Metaplasticity: A measure of how much more difficult it becomes to modify connections. This is important because it allows connections within the brain to become fix and finalized after having been changed, resisting further change. Of course, nodes can become less rigid as time goes on, or else the network would become unusable. The ease with which nodes do this also varies. This is important in the human brain in facilitating short term memory, wherein connections remain constant after having been established, but then become plastic again.

Program Mechanics: Attributes of Nodes Number of Connections: Some nodes have the capacity to connect to more nodes than others. This is theoretically more important when metasystems get more advanced than those in my current project. Threshold: Some nodes require more stimulation in order to fire than others. Base Values for Connections: Most connections between nodes are only the basic connections that do not yet reflect changes from the environment. The nodes remember what these values are for their connections. Type of Node: This is a reflection of something the brain does. I'm not sure why, but I put it in for good measure, because it seems important in the brain.

Attributes of Connections The links between the nodes are where the nodes actually remember past actions, so these attributes are particularly important. Strength of Connection: This is the power a connection has to activate the end node. This also stores whether the connection is excitatory or inhibitory. This is affected by attributes of the connected nodes.

Lessons From Neurobiology In designing my project, I tried to copy neurobiology, because designing from scratch is difficult Hebbian Learning Excitory/Inhibitory Neurotransmitter types/receivers Cognitive Science Network structures Plasticity Metaplasticity

Difficulties in Modeling and the Need for Algorithms In the human brain, which can also be thought of as an unsupervised neural network, neurons each have thousands of connections, and there are billions of neurons in the brain. We cannot expect a computer to handle all this without the mechanisms being simplified and optimized a bit.

Program Mechanics: Algorithms An unsupervised neural network can be thought of as a collection of nodes which form connections to each other. In the beginning, the network is set up having different types of nodes, with different types of characteristics and connections. In the beginning, these attributes and connections are all cookie-cutter; they do not encode meaningful information. Only after the network has changed in response to stimuli will the connections and attributes be important. Furthermore, only those connections that have changed to reflect the stimuli have important changes, and then only before they have been changed back to being non-descript.

Program Mechanisms: Algorithms So, my program attempts to conserve computational resources by taking advantage of the fact that most nodes aren't important. It keeps track of which nodes encode meaningful information, and keeps statistical information on those nodes that do not. Whenever new information needs to be assimilated, the existence of nodes is predicted using statistical information which are then brought into reality in order to hold useful information. In this way, the program processes no more than is actually needed, while at the same time reducing informational artifacts of the program from becoming too large.

Theory Computational Complexity Number of important connections proportionate to information to be stored How much does it need to know? Processing kept to a minimum Cognitive Science

Further Research: Representations As it currently stands, the program represents information by storing the connections between nodes as well as storing which nodes are important. It would be better if information were stored in a more intuitive and less spacious manner. Representational standards should be developed based on symbolic cognitive science.

Bibliography - Explanation of the computationalist approach to cognitive science, the approach used in the theory of this program. - explanation of how neurons need to be in phase to communicate. - Explanation of Bayesian math, which I'm attempting to use to model this program. psych.stanford.edu/~andreas/Research/Papers/TextCategorization/Wiener.Pedersen.Wei gend_SDAIR95.pshttp://www- psych.stanford.edu/~andreas/Research/Papers/TextCategorization/Wiener.Pedersen.Wei gend_SDAIR95.ps - Neural net used for topic spotting.

Bibliography - The ambiguous nature of words described in this article supports the use of neural networks for processing rather than more rigid rule- based approaches. - This article is particularly loquacious in describing the difference between supervised and unsupervised networks. The real power of neural networks is that they can learn, and it is important that I get sufficient learning material for my network. This will include dictionaries (which I am having trouble obtaining), and conversational transcripts. - This article talks about how networks can "memorize" data. Tbat is to say that they avoid learning the rules about the data, but instead learn only to respond to the input data used so far. It is also important to consider the topology of networks, because that is an additional level of complexity within the brain, or a neural network.

Bibliography hysics.brown.edu/users/faculty/intrator/papers/face- j.ps.gz+unsupervised+neural+networkhttp://scholar.google.com/scholar?hl=en&lr=&q=cache:fYGM13j1fhUJ: hysics.brown.edu/users/faculty/intrator/papers/face- j.ps.gz+unsupervised+neural+network - Face recognition is generally done with more rigid algorithms, but this presents a way to use neural networks to achieve the desired recognition.