1. Brain, Mind, and Computation Part I: Computational Brain Brain, Mind, and Computation Part I: Computational Brain Brain-Mind-Behavior Seminar May 18, 2011 Byoung-Tak Zhang Biointelligence Laboratory Computer Science and Engineering & Brain Science, Cognitive Science, and Bioinformatics Programs & Brain-Mind-Behavior Concentration Program Seoul National University http://bi.snu.ac.kr/

2. (c) 2009 SNU Biointelligence Laboratory, http://bi.snu.ac.kr/ 2 Lecture Overview Part I: Computational Brain  How the brain encodes and processes information? Part II: Brain-Inspired Computation  How to build intelligent machines inspired by brain processes? Part III: Cognitive Brain Networks  How the brain networks perform cognitive processing?

3. © 2009, SNU CSE BioIntelligence Lab, http://bi.snu.ac.kr/ 3 Human Brain: Functional Architecture Brodmann’s areas & functions

4. © 2009, SNU CSE BioIntelligence Lab, http://bi.snu.ac.kr/ 4 Cortex: Perception, Action, and Cognition Fig 3-18 Primary sensory and motor cortex & association cortex

5. 5 (c) 2000-2007 SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ Mind, Brain, Cell, Molecule Brain Cell Molecule Mind 10 11 cells 10 10 mol. memory

6. Computational Neuroscience

7. (C) 2009, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 7 The Structure of Neurons

8. (C) 2006, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 8 Information Transmission between Neurons Overview of signaling between neurons  Synaptic inputs  Synaptic inputs make postsynaptic current.  Passive depolarizing currents  Action potential: depolarize the membrane, and trigger another action potential.  The inward current conducted down the axon.  This leads to depolarization of adjacent regions of membrane

9. (C) 2006, SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ 9 Voltage-gated channel in the neuronal membrane. Mechanisms of neurotransmitter receptor molecules.

10. 10

11. Hodgkin-Huxley Model Hodgkin-Huxley model  C: capacitance  I(t): external current Three ionic currents 11 Fig. 2.7

12. 12 (c) 2000-2007 SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ Molecular Basis of Learning and Memory in the Brain

13. Neuronal Connectivity 13

14. Associative Networks 14 Associative node and network architecture. (A) A simplified neuron that receives a large number of inputs r i in. The synaptic efficiency is denoted by w i. the output of the neuron, r out depends on the particular input stimulus. (B) A network of associative nodes. Each component of the input vector, r i in, is distributed to each neuron in the network. However, the effect of the input can be different for each neuron as each individual synapse can have different efficiency values w ij, where j labels the neuron in the network. Auto-associative node and network architecture. (A) Schematic illustration of an auto-associative node that is distinguished from the associative node as illustrated in Fig. 7.1A in that it has, in addition, a recurrent feedback connection. (B) An auto-associative network that consist of associative nodes that not only receive external input from other neural layers but, in addition, have many recurrent collateral connections between the nodes in the neural layer.

15. Principles of Brain Processing

16. Memory, Learning, and the Brain 기억과 학습은 뇌의 사고, 행동, 인지의 기반 메카니즘 McGaugh, J. L. Memory & Emotion: The Making of Lasting Memories, 2003. © 2009, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 16 It is our memory that enables us to value everything else we possess. Lacking memory, we would have no ability to be concerned about our hearts, achievements, loved ones, and incomes. Our brain has an amazing capacity to integrate the combined effects of our past experiences together with our present experiences in creating our thought and actions. This is all possible by the memory and the memories are formed by the learning process.

17. 17 Memory Systems in the Brain Source: Gazzaniga et al., Cognitive Neuroscience: The Biology of the Mind, 2002.

18. Summary: Principles of Cognitive Learning © 2009, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 18 Continuity. Learning is a continuous, lifelong process. “The experiences of each immediately past moment are memories that merge with current momentary experiences to create the impression of seamless continuity in our lives” [McGaugh, 2003] Glocality. “Perception is dependent on context” and it is important to maintain both global and local, i.e. glocal, representations [Peterson and Rhodes, 2003] Compositionality. “The brain activates existing metaphorical structures to form a conceptual blend, consisting of all the metaphors linked together” [Feldman, 2006]. “Mental chemistry” [J. S. Mill] [Zhang, IEEE Computational Intelligence Magazine, 2008]

19. 2. Multiple Levels of Representation Source: J. W. Rudy, The Neurobiology of Learning and Memory, 2008.

20. 3. Creation of New Memory Source: J. W. Rudy, The Neurobiology of Learning and Memory, 2008.

21. 21 (c) 2000-2007 SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ What is the information processing principle underlying human intelligence?

22. 22 (c) 2000-2007 SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ Von Neumann’s The Computer and the Brain (1958) John von Neumann (1903-1957)

23. (c) 2008 SNU Biointelligence Laboratory, http://bi.snu.ac.kr/ 23 Some Facts about the Brain Volume and mass: 1.35 liter & 1.35 kg Processors: 10 11 neurons Communication: 10 14 synapses Speed: 10 -3 sec  Computer: 1 GHz = 10 -9 sec Memory: 2.8 x 10 21 bits  = 14 bits/sec x 10 11 neurons x (2 x 10 9 ) sec (2 x 10 9 sec = 60 years of life time)  Computer disk: tera bits = 10 12 bits Reliability: 10 4 neurons dying everyday Plasticity: biochemical learning

24. 24 (c) 2000-2007 SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/ Principles of Information Processing in the Brain The Principle of Uncertainty  Precision vs. prediction The Principle of Nonseparability “UN-IBM”  Processor vs. memory The Principle of Infinity  Limited matter vs. unbounded memory The Principle of “Big Numbers Count”  Hyperinteraction of 10 11 neurons (or > 10 17 molecules) The Principle of “Matter Matters”  Material basis of “consciousness” [Zhang, 2005]

25. Neural Computers

26. Learning to extract the orientation of a face patch (Salakhutdinov & Hinton, NIPS 2007)

27. The training and test sets for predicting face orientation 11,000 unlabeled cases100, 500, or 1000 labeled cases face patches from new people

28. Deep Autoencoders (Hinton & Salakhutdinov, 2006) They always looked like a really nice way to do non-linear dimensionality reduction: –But it is very difficult to optimize deep autoencoders using backpropagation. We now have a much better way to optimize them: –First train a stack of 4 RBM’s –Then “unroll” them. –Then fine-tune with backprop. 1000 neurons 500 neurons 250 neurons 30 1000 neurons 28x28 linear units

29. A comparison of methods for compressing digit images to 30 real numbers. real data 30-D deep auto 30-D logistic PCA 30-D PCA

30. Retrieving documents that are similar to a query document We can use an autoencoder to find low- dimensional codes for documents that allow fast and accurate retrieval of similar documents from a large set. We start by converting each document into a “bag of words”. This a 2000 dimensional vector that contains the counts for each of the 2000 commonest words.

31. Proportion of retrieved documents in same class as query Number of documents retrieved

32. First compress all documents to 2 numbers using a type of PCA Then use different colors for different document categories

33. First compress all documents to 2 numbers. Then use different colors for different document categories